• Uros Bizjak's avatar
    x86/percpu: Do not clobber %rsi in percpu_{try_,}cmpxchg{64,128}_op · 7c097ca5
    Uros Bizjak authored
    The fallback alternative uses %rsi register to manually load pointer
    to the percpu variable before the call to the emulation function.
    This is unoptimal, because the load is hidden from the compiler.
    
    Move the load of %rsi outside inline asm, so the compiler can
    reuse the value. The code in slub.o improves from:
    
        55ac:	49 8b 3c 24          	mov    (%r12),%rdi
        55b0:	48 8d 4a 40          	lea    0x40(%rdx),%rcx
        55b4:	49 8b 1c 07          	mov    (%r15,%rax,1),%rbx
        55b8:	4c 89 f8             	mov    %r15,%rax
        55bb:	48 8d 37             	lea    (%rdi),%rsi
        55be:	e8 00 00 00 00       	callq  55c3 <...>
    			55bf: R_X86_64_PLT32	this_cpu_cmpxchg16b_emu-0x4
        55c3:	75 a3                	jne    5568 <...>
        55c5:	...
    
     0000000000000000 <.altinstr_replacement>:
       5:	65 48 0f c7 0f       	cmpxchg16b %gs:(%rdi)
    
    to:
    
        55ac:	49 8b 34 24          	mov    (%r12),%rsi
        55b0:	48 8d 4a 40          	lea    0x40(%rdx),%rcx
        55b4:	49 8b 1c 07          	mov    (%r15,%rax,1),%rbx
        55b8:	4c 89 f8             	mov    %r15,%rax
        55bb:	e8 00 00 00 00       	callq  55c0 <...>
    			55bc: R_X86_64_PLT32	this_cpu_cmpxchg16b_emu-0x4
        55c0:	75 a6                	jne    5568 <...>
        55c2:	...
    
    Where the alternative replacement instruction now uses %rsi:
    
     0000000000000000 <.altinstr_replacement>:
       5:	65 48 0f c7 0e       	cmpxchg16b %gs:(%rsi)
    
    The instruction (effectively a reg-reg move) at 55bb: in the original
    assembly is removed. Also, both the CALL and replacement CMPXCHG16B
    are 5 bytes long, removing the need for NOPs in the asm code.
    Suggested-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    Signed-off-by: default avatarUros Bizjak <ubizjak@gmail.com>
    Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
    Link: https://lore.kernel.org/r/20230918151452.62344-1-ubizjak@gmail.com
    7c097ca5
percpu.h 20.6 KB