scsi: libfc: Revert "[SCSI] libfc: use offload EM instance again instead jumping...

scsi: libfc: Revert "[SCSI] libfc: use offload EM instance again instead jumping to next EM"

This reverts commit df7ec820f26f30c3c8389f0f5b9c9ce6987e3d67.

This revert came about because of efforts by Ewan Milne, Curtis Taylor
and I.  In researching this issue, significant performance issues were
seen on large CPU count systems using the software FCOE stack.  Hannes
also weighed in.

The same was not apparent on much smaller low count CPU systems.  The
behavior introduced by commit df7ec820f26f30c3c8389f0f5b9c9ce6987e3d67
lands sup with large count CPU systems seeing continual
blk_requeue_request() calls due to ML_QUEUE_HOST_BUSY.

fc_exch_alloc() used to try all the available exchange managers in the
list for an available exchange id, but this was changed in 2010 so that
if the first matched exchange manager couldn't allocate one, it fails
and we end up returning host busy.  This was due to commit:

Setting the ddp_min module parameter to fcoe to 128MB prevents the
->match function from permitting the use of the offload exchange manager
for the frame, and we no longer see the problem with host busy status,
since it uses the larger non-offloaded pool.

Reverting commit df7ec820f26f30c3c8389f0f5b9c9ce6987e3d67 was tested to
also prevent the host busy issue due to failing allocations.

Suggested-by: Ewan Milne <emilne@redhat.com>
Suggested-by: Curtis Taylor <cjt@us.ibm.com>
Tested-by: Laurence Oberman <loberman@redhat.com>
Signed-off-by: Laurence Oberman <loberman@redhat.com>