• Paul E. McKenney's avatar
    rcu: Reduce overhead of cond_resched() checks for RCU · 4a81e832
    Paul E. McKenney authored
    Commit ac1bea85 (Make cond_resched() report RCU quiescent states)
    fixed a problem where a CPU looping in the kernel with but one runnable
    task would give RCU CPU stall warnings, even if the in-kernel loop
    contained cond_resched() calls.  Unfortunately, in so doing, it introduced
    performance regressions in Anton Blanchard's will-it-scale "open1" test.
    The problem appears to be not so much the increased cond_resched() path
    length as an increase in the rate at which grace periods complete, which
    increased per-update grace-period overhead.
    
    This commit takes a different approach to fixing this bug, mainly by
    moving the RCU-visible quiescent state from cond_resched() to
    rcu_note_context_switch(), and by further reducing the check to a
    simple non-zero test of a single per-CPU variable.  However, this
    approach requires that the force-quiescent-state processing send
    resched IPIs to the offending CPUs.  These will be sent only once
    the grace period has reached an age specified by the boot/sysfs
    parameter rcutree.jiffies_till_sched_qs, or once the grace period
    reaches an age halfway to the point at which RCU CPU stall warnings
    will be emitted, whichever comes first.
    Reported-by: default avatarDave Hansen <dave.hansen@intel.com>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    Cc: Andi Kleen <ak@linux.intel.com>
    Cc: Christoph Lameter <cl@gentwo.org>
    Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
    Cc: Eric Dumazet <eric.dumazet@gmail.com>
    Reviewed-by: default avatarJosh Triplett <josh@joshtriplett.org>
    [ paulmck: Made rcu_momentary_dyntick_idle() as suggested by the
      ktest build robot.  Also fixed smp_mb() comment as noted by
      Oleg Nesterov. ]
    
    Merge with e552592e (Reduce overhead of cond_resched() checks for RCU)
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    4a81e832
tree.c 116 KB