• Frederic Weisbecker's avatar
    rcu/nocb: Use the rcuog CPU's ->nocb_timer · d76e0926
    Frederic Weisbecker authored
    Currently each CPU has its own ->nocb_timer queued when the nocb_gp
    wakeup must be deferred.  This approach has many drawbacks, compared to
    a solution based on a single timer per NOCB group:
    
    * There are a lot of timers to maintain.
    
    * The per-rdp ->nocb_lock must be held to queue and cancel the timer
      and this lock can already be heavily contended.
    
    * One timer firing doesn't cancel the other timers in the same group:
      - These other timers can thus cause spurious wakeups
      - Each rdp that queued a timer must lock both ->nocb_lock and then
        ->nocb_gp_lock upon exit from the kernel to idle/user/guest mode.
    
    * We can't cancel all of them if we detect an unflushed bypass in
      nocb_gp_wait(). In fact currently we only ever cancel the ->nocb_timer
      of the leader group.
    
    * The leader group's nocb_timer is cancelled without locking ->nocb_lock
      in nocb_gp_wait().  This currently appears to be safe but is an
      accident waiting to happen.
    
    * Since the timer acquires ->nocb_lock, it requires extra care in the
      NOCB (de-)offloading process, requiring that it be either enabled or
      disabled and then flushed.
    
    This commit instead uses the rcuog kthread's CPU's ->nocb_timer instead.
    It is protected by nocb_gp_lock, which is _way_ less contended and
    remains so even after this change.  As a matter of fact, the nocb_timer
    almost never fires and the deferred wakeup is mostly carried out upon
    idle/user/guest entry.  Now the early check performed at this point in
    do_nocb_deferred_wakeup() is done on rdp_gp->nocb_defer_wakeup, which
    is of course racy.  However, this raciness is harmless because we only
    need the guarantee that the timer is queued if we were the last one to
    queue it.  Any other situation (another CPU has queued it and we either
    see it or not) is fine.
    
    This solves all the issues listed above.
    Signed-off-by: default avatarFrederic Weisbecker <frederic@kernel.org>
    Cc: Josh Triplett <josh@joshtriplett.org>
    Cc: Lai Jiangshan <jiangshanlai@gmail.com>
    Cc: Joel Fernandes <joel@joelfernandes.org>
    Cc: Neeraj Upadhyay <neeraju@codeaurora.org>
    Cc: Boqun Feng <boqun.feng@gmail.com>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
    d76e0926
tree.h 19.6 KB