• Waiman Long's avatar
    locking/pvqspinlock, x86: Optimize the PV unlock code path · d7804530
    Waiman Long authored
    The unlock function in queued spinlocks was optimized for better
    performance on bare metal systems at the expense of virtualized guests.
    
    For x86-64 systems, the unlock call needs to go through a
    PV_CALLEE_SAVE_REGS_THUNK() which saves and restores 8 64-bit
    registers before calling the real __pv_queued_spin_unlock()
    function. The thunk code may also be in a separate cacheline from
    __pv_queued_spin_unlock().
    
    This patch optimizes the PV unlock code path by:
    
     1) Moving the unlock slowpath code from the fastpath into a separate
        __pv_queued_spin_unlock_slowpath() function to make the fastpath
        as simple as possible..
    
     2) For x86-64, hand-coded an assembly function to combine the register
        saving thunk code with the fastpath code. Only registers that
        are used in the fastpath will be saved and restored. If the
        fastpath fails, the slowpath function will be called via another
        PV_CALLEE_SAVE_REGS_THUNK(). For 32-bit, it falls back to the C
        __pv_queued_spin_unlock() code as the thunk saves and restores
        only one 32-bit register.
    
    With a microbenchmark of 5M lock-unlock loop, the table below shows
    the execution times before and after the patch with different number
    of threads in a VM running on a 32-core Westmere-EX box with x86-64
    4.2-rc1 based kernels:
    
      Threads	Before patch	After patch	% Change
      -------	------------	-----------	--------
         1		   134.1 ms	  119.3 ms	  -11%
         2		   1286  ms	   953  ms	  -26%
         3		   3715  ms	  3480  ms	  -6.3%
         4		   4092  ms	  3764  ms	  -8.0%
    Signed-off-by: default avatarWaiman Long <Waiman.Long@hpe.com>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Davidlohr Bueso <dave@stgolabs.net>
    Cc: Douglas Hatch <doug.hatch@hpe.com>
    Cc: H. Peter Anvin <hpa@zytor.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Scott J Norton <scott.norton@hpe.com>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Link: http://lkml.kernel.org/r/1447114167-47185-5-git-send-email-Waiman.Long@hpe.comSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
    d7804530
qspinlock_paravirt.h 1.79 KB