• Peter Zijlstra's avatar
    sched: Implement lockless wake-queues · 76751049
    Peter Zijlstra authored
    This is useful for locking primitives that can effect multiple
    wakeups per operation and want to avoid lock internal lock contention
    by delaying the wakeups until we've released the lock internal locks.
    
    Alternatively it can be used to avoid issuing multiple wakeups, and
    thus save a few cycles, in packet processing. Queue all target tasks
    and wakeup once you've processed all packets. That way you avoid
    waking the target task multiple times if there were multiple packets
    for the same task.
    
    Properties of a wake_q are:
    - Lockless, as queue head must reside on the stack.
    - Being a queue, maintains wakeup order passed by the callers. This can
      be important for otherwise, in scenarios where highly contended locks
      could affect any reliance on lock fairness.
    - A queued task cannot be added again until it is woken up.
    
    This patch adds the needed infrastructure into the scheduler code
    and uses the new wake_list to delay the futex wakeups until
    after we've released the hash bucket locks.
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    [tweaks, adjustments, comments, etc.]
    Signed-off-by: default avatarDavidlohr Bueso <dbueso@suse.de>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Acked-by: default avatarThomas Gleixner <tglx@linutronix.de>
    Cc: Borislav Petkov <bp@alien8.de>
    Cc: Chris Mason <clm@fb.com>
    Cc: Davidlohr Bueso <dave@stgolabs.net>
    Cc: George Spelvin <linux@horizon.com>
    Cc: H. Peter Anvin <hpa@zytor.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Manfred Spraul <manfred@colorfullife.com>
    Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
    Cc: Steven Rostedt <rostedt@goodmis.org>
    Link: http://lkml.kernel.org/r/1430494072-30283-2-git-send-email-dave@stgolabs.netSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
    76751049
core.c 201 KB