• Peter Zijlstra's avatar
    softirq,rcu: Inform RCU of irq_exit() activity · ec433f0c
    Peter Zijlstra authored
    The rcu_read_unlock_special() function relies on in_irq() to exclude
    scheduler activity from interrupt level.  This fails because exit_irq()
    can invoke the scheduler after clearing the preempt_count() bits that
    in_irq() uses to determine that it is at interrupt level.  This situation
    can result in failures as follows:
    
     $task			IRQ		SoftIRQ
    
     rcu_read_lock()
    
     /* do stuff */
    
     <preempt> |= UNLOCK_BLOCKED
    
     rcu_read_unlock()
       --t->rcu_read_lock_nesting
    
    			irq_enter();
    			/* do stuff, don't use RCU */
    			irq_exit();
    			  sub_preempt_count(IRQ_EXIT_OFFSET);
    			  invoke_softirq()
    
    					ttwu();
    					  spin_lock_irq(&pi->lock)
    					  rcu_read_lock();
    					  /* do stuff */
    					  rcu_read_unlock();
    					    rcu_read_unlock_special()
    					      rcu_report_exp_rnp()
    					        ttwu()
    					          spin_lock_irq(&pi->lock) /* deadlock */
    
       rcu_read_unlock_special(t);
    
    Ed can simply trigger this 'easy' because invoke_softirq() immediately
    does a ttwu() of ksoftirqd/# instead of doing the in-place softirq stuff
    first, but even without that the above happens.
    
    Cure this by also excluding softirqs from the
    rcu_read_unlock_special() handler and ensuring the force_irqthreads
    ksoftirqd/# wakeup is done from full softirq context.
    
    [ Alternatively, delaying the ->rcu_read_lock_nesting decrement
      until after the special handling would make the thing more robust
      in the face of interrupts as well.  And there is a separate patch
      for that. ]
    
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Reported-and-tested-by: default avatarEd Tomlinson <edt@aei.ca>
    Signed-off-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    ec433f0c
rcutree_plugin.h 57.9 KB