• Andrii Nakryiko's avatar
    objpool: enable inlining objpool_push() and objpool_pop() operations · a3b00f10
    Andrii Nakryiko authored
    objpool_push() and objpool_pop() are very performance-critical functions
    and can be called very frequently in kretprobe triggering path.
    
    As such, it makes sense to allow compiler to inline them completely to
    eliminate function calls overhead. Luckily, their logic is quite well
    isolated and doesn't have any sprawling dependencies.
    
    This patch moves both objpool_push() and objpool_pop() into
    include/linux/objpool.h and marks them as static inline functions,
    enabling inlining. To avoid anyone using internal helpers
    (objpool_try_get_slot, objpool_try_add_slot), rename them to use leading
    underscores.
    
    We used kretprobe microbenchmark from BPF selftests (bench trig-kprobe
    and trig-kprobe-multi benchmarks) running no-op BPF kretprobe/kretprobe.multi
    programs in a tight loop to evaluate the effect. BPF own overhead in
    this case is minimal and it mostly stresses the rest of in-kernel
    kretprobe infrastructure overhead. Results are in millions of calls per
    second. This is not super scientific, but shows the trend nevertheless.
    
    BEFORE
    ======
    kretprobe      :    9.794 ± 0.086M/s
    kretprobe-multi:   10.219 ± 0.032M/s
    
    AFTER
    =====
    kretprobe      :    9.937 ± 0.174M/s (+1.5%)
    kretprobe-multi:   10.440 ± 0.108M/s (+2.2%)
    
    Link: https://lore.kernel.org/all/20240424215214.3956041-2-andrii@kernel.org/
    
    Cc: Matt (Qiang) Wu <wuqiang.matt@bytedance.com>
    Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
    Signed-off-by: default avatarMasami Hiramatsu (Google) <mhiramat@kernel.org>
    a3b00f10
objpool.c 4.8 KB