]> git.baikalelectronics.ru Git - kernel.git/commit
rculist: Make list_entry_rcu() use lockless_dereference()
authorPatrick Marlier <patrick.marlier@gmail.com>
Fri, 11 Sep 2015 22:50:35 +0000 (15:50 -0700)
committerPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Tue, 6 Oct 2015 18:16:18 +0000 (11:16 -0700)
commit6e071d274842a99d94969e4b2fb4cbe5b540f0eb
treea51cb391fd45ca2a640742343906181822890c7d
parent1457930eefde64b36ae3672375a0640aceeaa958
rculist: Make list_entry_rcu() use lockless_dereference()

The current list_entry_rcu() implementation copies the pointer to a stack
variable, then invokes rcu_dereference_raw() on it.  This results in an
additional store-load pair.  Now, most compilers will emit normal store
and load instructions, which might seem to be of negligible overhead,
but this results in a load-hit-store situation that can cause surprisingly
long pipeline stalls, even on modern microprocessors.  The problem is
that it takes time for the store to get the store buffer updated, which
can delay the subsequent load, which immediately follows.

This commit therefore switches to the lockless_dereference() primitive,
which does not expect the __rcu annotations (that are anyway not present
in the list_head structure) and which, like rcu_dereference_raw(),
does not check for an enclosing RCU read-side critical section.
Most importantly, it does not copy the pointer, thus avoiding the
load-hit-store overhead.

Signed-off-by: Patrick Marlier <patrick.marlier@gmail.com>
[ paulmck: Switched to lockless_dereference() to suppress sparse warnings. ]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
include/linux/rculist.h