• Maciej Fijalkowski's avatar
    bpf, x64: Drop "pop %rcx" instruction on BPF JIT epilogue · d207929d
    Maciej Fijalkowski authored
    Back when all of the callee-saved registers where always pushed to stack
    in x64 JIT prologue, tail call counter was placed at the bottom of the
    BPF program's stack frame that had a following layout:
    
    +-------------+
    |  ret addr   |
    +-------------+
    |     rbp     | <- rbp
    +-------------+
    |             |
    | free space  |
    | from:       |
    | sub $x,%rsp |
    |             |
    +-------------+
    |     rbx     |
    +-------------+
    |     r13     |
    +-------------+
    |     r14     |
    +-------------+
    |     r15     |
    +-------------+
    |  tail call  | <- rsp
    |   counter   |
    +-------------+
    
    In order to restore the callee saved registers, epilogue needed to
    explicitly toss away the tail call counter via "pop %rbx" insn, so that
    %rsp would be back at the place where %r15 was stored.
    
    Currently, the tail call counter is placed on stack *before* the callee
    saved registers (brackets on rbx through r15 mean that they are now
    pushed to stack only if they are used):
    
    +-------------+
    |  ret addr   |
    +-------------+
    |     rbp     | <- rbp
    +-------------+
    |             |
    | free space  |
    | from:       |
    | sub $x,%rsp |
    |             |
    +-------------+
    |  tail call  |
    |   counter   |
    +-------------+
    (     rbx     )
    +-------------+
    (     r13     )
    +-------------+
    (     r14     )
    +-------------+
    (     r15     ) <- rsp
    +-------------+
    
    For the record, the epilogue insns consist of (assuming all of the
    callee saved registers are used by program):
    pop    %r15
    pop    %r14
    pop    %r13
    pop    %rbx
    pop    %rcx
    leaveq
    retq
    
    "pop %rbx" for getting rid of tail call counter was not an option
    anymore as it would overwrite the restored value of %rbx register, so it
    was changed to use the %rcx register.
    
    Since epilogue can start popping the callee saved registers right away
    without any additional work, the "pop %rcx" could be dropped altogether
    as "leave" insn will simply move the %rbp to %rsp. IOW, tail call
    counter does not need the explicit handling.
    
    Having in mind the explanation above and the actual reason for that,
    let's piggy back on "leave" insn for discarding the tail call counter
    from stack and remove the "pop %rcx" from epilogue.
    Signed-off-by: default avatarMaciej Fijalkowski <maciej.fijalkowski@intel.com>
    Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
    Link: https://lore.kernel.org/bpf/20200929204653.4325-2-maciej.fijalkowski@intel.com
    d207929d
bpf_jit_comp.c 55.6 KB