• Paul E. McKenney's avatar
    rcu: Make hierarchical RCU less IPI-happy · ef631b0c
    Paul E. McKenney authored
    This patch fixes a hierarchical-RCU performance bug located by Anton
    Blanchard.  The problem stems from a misguided attempt to provide a
    work-around for jiffies-counter failure.  This work-around uses a per-CPU
    n_rcu_pending counter, which is incremented on each call to rcu_pending(),
    which in turn is called from each scheduling-clock interrupt.  Each CPU
    then treats this counter as a surrogate for the jiffies counter, so
    that if the jiffies counter fails to advance, the per-CPU n_rcu_pending
    counter will cause RCU to invoke force_quiescent_state(), which in turn
    will (among other things) send resched IPIs to CPUs that have thus far
    failed to pass through an RCU quiescent state.
    
    Unfortunately, each CPU resets only its own counter after sending a
    batch of IPIs.  This means that the other CPUs will also (needlessly)
    send -another- round of IPIs, for a full N-squared set of IPIs in the
    worst case every three scheduler-clock ticks until the grace period
    finally ends.  It is not reasonable for a given CPU to reset each and
    every n_rcu_pending for all the other CPUs, so this patch instead simply
    disables the jiffies-counter "training wheels", thus eliminating the
    excessive IPIs.
    
    Note that the jiffies-counter IPIs do not have this problem due to
    the fact that the jiffies counter is global, so that the CPU sending
    the IPIs can easily reset things, thus preventing the other CPUs from
    sending redundant IPIs.
    
    Note also that the n_rcu_pending counter remains, as it will continue to
    be used for tracing.  It may also see use to update the jiffies counter,
    should an appropriate kick-the-jiffies-counter API appear.
    Located-by: default avatarAnton Blanchard <anton@au1.ibm.com>
    Tested-by: default avatarAnton Blanchard <anton@au1.ibm.com>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    Cc: anton@samba.org
    Cc: akpm@linux-foundation.org
    Cc: dipankar@in.ibm.com
    Cc: manfred@colorfullife.com
    Cc: cl@linux-foundation.org
    Cc: josht@linux.vnet.ibm.com
    Cc: schamp@sgi.com
    Cc: niv@us.ibm.com
    Cc: dvhltc@us.ibm.com
    Cc: ego@in.ibm.com
    Cc: laijs@cn.fujitsu.com
    Cc: rostedt@goodmis.org
    Cc: peterz@infradead.org
    Cc: penberg@cs.helsinki.fi
    Cc: andi@firstfloor.org
    Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
    LKML-Reference: <12396834793575-git-send-email->
    Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
    ef631b0c
rcutree.c 44.2 KB