Commit 0bf3924b authored by Thomas Gleixner's avatar Thomas Gleixner

x86/entry: Force rcu_irq_enter() when in idle task

The idea of conditionally calling into rcu_irq_enter() only when RCU is
not watching turned out to be not completely thought through.

Paul noticed occasional premature end of grace periods in RCU torture
testing. Bisection led to the commit which made the invocation of
rcu_irq_enter() conditional on !rcu_is_watching().

It turned out that this conditional breaks RCU assumptions about the idle
task when the scheduler tick happens to be a nested interrupt. Nested
interrupts can happen when the first interrupt invokes softirq processing
on return which enables interrupts.

If that nested tick interrupt does not invoke rcu_irq_enter() then the
RCU's irq-nesting checks will believe that this interrupt came directly
from idle, which will cause RCU to report a quiescent state.  Because this
interrupt instead came from a softirq handler which might have been
executing an RCU read-side critical section, this can cause the grace
period to end prematurely.

Change the condition from !rcu_is_watching() to is_idle_task(current) which
enforces that interrupts in the idle task unconditionally invoke
rcu_irq_enter() independent of the RCU state.

This is also correct vs. user mode entries in NOHZ full scenarios because
user mode entries bring RCU out of EQS and force the RCU irq nesting state
accounting to nested. As only the first interrupt can enter from user mode
a nested tick interrupt will enter from kernel mode and as the nesting
state accounting is forced to nesting it will not do anything stupid even
if rcu_irq_enter() has not been invoked.

Fixes: 3eeec385 ("x86/entry: Provide idtentry_entry/exit_cond_rcu()")
Reported-by: default avatar"Paul E. McKenney" <paulmck@kernel.org>
Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
Tested-by: default avatar"Paul E. McKenney" <paulmck@kernel.org>
Reviewed-by: default avatar"Paul E. McKenney" <paulmck@kernel.org>
Acked-by: default avatarAndy Lutomirski <luto@kernel.org>
Acked-by: default avatarFrederic Weisbecker <frederic@kernel.org>
Link: https://lkml.kernel.org/r/87wo4cxubv.fsf@nanos.tec.linutronix.de
parent 71ed49d8
...@@ -557,14 +557,34 @@ bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs) ...@@ -557,14 +557,34 @@ bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs)
return false; return false;
} }
if (!__rcu_is_watching()) { /*
* If this entry hit the idle task invoke rcu_irq_enter() whether
* RCU is watching or not.
*
* Interupts can nest when the first interrupt invokes softirq
* processing on return which enables interrupts.
*
* Scheduler ticks in the idle task can mark quiescent state and
* terminate a grace period, if and only if the timer interrupt is
* not nested into another interrupt.
*
* Checking for __rcu_is_watching() here would prevent the nesting
* interrupt to invoke rcu_irq_enter(). If that nested interrupt is
* the tick then rcu_flavor_sched_clock_irq() would wrongfully
* assume that it is the first interupt and eventually claim
* quiescient state and end grace periods prematurely.
*
* Unconditionally invoke rcu_irq_enter() so RCU state stays
* consistent.
*
* TINY_RCU does not support EQS, so let the compiler eliminate
* this part when enabled.
*/
if (!IS_ENABLED(CONFIG_TINY_RCU) && is_idle_task(current)) {
/* /*
* If RCU is not watching then the same careful * If RCU is not watching then the same careful
* sequence vs. lockdep and tracing is required * sequence vs. lockdep and tracing is required
* as in enter_from_user_mode(). * as in enter_from_user_mode().
*
* This only happens for IRQs that hit the idle
* loop, i.e. if idle is not using MWAIT.
*/ */
lockdep_hardirqs_off(CALLER_ADDR0); lockdep_hardirqs_off(CALLER_ADDR0);
rcu_irq_enter(); rcu_irq_enter();
...@@ -576,9 +596,10 @@ bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs) ...@@ -576,9 +596,10 @@ bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs)
} }
/* /*
* If RCU is watching then RCU only wants to check * If RCU is watching then RCU only wants to check whether it needs
* whether it needs to restart the tick in NOHZ * to restart the tick in NOHZ mode. rcu_irq_enter_check_tick()
* mode. * already contains a warning when RCU is not watching, so no point
* in having another one here.
*/ */
instrumentation_begin(); instrumentation_begin();
rcu_irq_enter_check_tick(); rcu_irq_enter_check_tick();
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment