io_uring: fix CQ overflow condition

This is a leftover from when the rings initially were not free flowing,
and hence a test for tail + 1 == head would indicate full. Since we now
let them wrap instead of mask them with the size, we need to check if
they drift more than the ring size from each other.

This fixes a case where we'd overwrite CQ ring entries, if the user
failed to reap completions. Both cases would ultimately result in lost
completions as the application violated the depth it asked for. The only
difference is that before this fix we'd return invalid entries for the
overflowed completions, instead of properly flagging it in the
cq_ring->overflow variable.

Reported-by: Stefan Bühler <source@stbuehler.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/fs/io_uring.c b/fs/io_uring.c
index b35300e4c9a7924cae714bde72f66c9cb6b482bd..f65f85d892174f252cdd2ae48c4d9eb3902ef479 100644 (file)
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -338,7 +338,7 @@ static struct io_uring_cqe *io_get_cqring(struct io_ring_ctx *ctx)
        tail = ctx->cached_cq_tail;
        /* See comment at the top of the file */
        smp_rmb();
-       if (tail + 1 == READ_ONCE(ring->r.head))
+       if (tail - READ_ONCE(ring->r.head) == ring->ring_entries)
                return NULL;
 
        ctx->cached_cq_tail++;