• Davidlohr Bueso's avatar
    futexes: Avoid taking the hb->lock if there's nothing to wake up · b0c29f79
    Davidlohr Bueso authored
    In futex_wake() there is clearly no point in taking the hb->lock
    if we know beforehand that there are no tasks to be woken. While
    the hash bucket's plist head is a cheap way of knowing this, we
    cannot rely 100% on it as there is a racy window between the
    futex_wait call and when the task is actually added to the
    plist. To this end, we couple it with the spinlock check as
    tasks trying to enter the critical region are most likely
    potential waiters that will be added to the plist, thus
    preventing tasks sleeping forever if wakers don't acknowledge
    all possible waiters.
    
    Furthermore, the futex ordering guarantees are preserved,
    ensuring that waiters either observe the changed user space
    value before blocking or is woken by a concurrent waker. For
    wakers, this is done by relying on the barriers in
    get_futex_key_refs() -- for archs that do not have implicit mb
    in atomic_inc(), we explicitly add them through a new
    futex_get_mm function. For waiters we rely on the fact that
    spin_lock calls already update the head counter, so spinners
    are visible even if the lock hasn't been acquired yet.
    
    For more details please refer to the updated comments in the
    code and related discussion:
    
      https://lkml.org/lkml/2013/11/26/556
    
    Special thanks to tglx for careful review and feedback.
    Suggested-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    Reviewed-by: default avatarDarren Hart <dvhart@linux.intel.com>
    Reviewed-by: default avatarThomas Gleixner <tglx@linutronix.de>
    Reviewed-by: default avatarPeter Zijlstra <peterz@infradead.org>
    Signed-off-by: default avatarDavidlohr Bueso <davidlohr@hp.com>
    Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
    Cc: Mike Galbraith <efault@gmx.de>
    Cc: Jeff Mahoney <jeffm@suse.com>
    Cc: Scott Norton <scott.norton@hp.com>
    Cc: Tom Vaden <tom.vaden@hp.com>
    Cc: Aswin Chandramouleeswaran <aswin@hp.com>
    Cc: Waiman Long <Waiman.Long@hp.com>
    Cc: Jason Low <jason.low2@hp.com>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Link: http://lkml.kernel.org/r/1389569486-25487-5-git-send-email-davidlohr@hp.comSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
    b0c29f79
futex.c 75.7 KB