nvme-pci: Hold cq_poll_lock while completing CQEs

Completions need to consumed in the same order the controller submitted
them, otherwise future completion entries may overwrite ones we haven't
handled yet. Hold the nvme queue's poll lock while completing new CQEs to
prevent another thread from freeing command tags for reuse out-of-order.

Fixes: dabcefab45d3 ("nvme: provide optimized poll function for separate poll queues")
Signed-off-by: Bijan Mottahedeh <bijan.mottahedeh@oracle.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Keith Busch <kbusch@kernel.org>

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index ace4dd9e953c1681b0aeaa80196289f626a6b042..d3f23d6254e47a4d1a2cfd53323a327aca17c013 100644 (file)
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1078,9 +1078,9 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx)
 
        spin_lock(&nvmeq->cq_poll_lock);
        found = nvme_process_cq(nvmeq, &start, &end, -1);
+       nvme_complete_cqes(nvmeq, start, end);
        spin_unlock(&nvmeq->cq_poll_lock);
 
-       nvme_complete_cqes(nvmeq, start, end);
        return found;
 }