• Yonghong Song's avatar
    bpf, x86: Use kvmalloc_array instead kmalloc_array in bpf_jit_comp · de920fc6
    Yonghong Song authored
    x86 bpf_jit_comp.c used kmalloc_array to store jited addresses
    for each bpf insn. With a large bpf program, we have see the
    following allocation failures in our production server:
    
        page allocation failure: order:5, mode:0x40cc0(GFP_KERNEL|__GFP_COMP),
                                 nodemask=(null),cpuset=/,mems_allowed=0"
        Call Trace:
        dump_stack+0x50/0x70
        warn_alloc.cold.120+0x72/0xd2
        ? __alloc_pages_direct_compact+0x157/0x160
        __alloc_pages_slowpath+0xcdb/0xd00
        ? get_page_from_freelist+0xe44/0x1600
        ? vunmap_page_range+0x1ba/0x340
        __alloc_pages_nodemask+0x2c9/0x320
        kmalloc_order+0x18/0x80
        kmalloc_order_trace+0x1d/0xa0
        bpf_int_jit_compile+0x1e2/0x484
        ? kmalloc_order_trace+0x1d/0xa0
        bpf_prog_select_runtime+0xc3/0x150
        bpf_prog_load+0x480/0x720
        ? __mod_memcg_lruvec_state+0x21/0x100
        __do_sys_bpf+0xc31/0x2040
        ? close_pdeo+0x86/0xe0
        do_syscall_64+0x42/0x110
        entry_SYSCALL_64_after_hwframe+0x44/0xa9
        RIP: 0033:0x7f2f300f7fa9
        Code: Bad RIP value.
    
    Dumped assembly:
    
        ffffffff810b6d70 <bpf_int_jit_compile>:
        ; {
        ffffffff810b6d70: e8 eb a5 b4 00        callq   0xffffffff81c01360 <__fentry__>
        ffffffff810b6d75: 41 57                 pushq   %r15
        ...
        ffffffff810b6f39: e9 72 fe ff ff        jmp     0xffffffff810b6db0 <bpf_int_jit_compile+0x40>
        ;       addrs = kmalloc_array(prog->len + 1, sizeof(*addrs), GFP_KERNEL);
        ffffffff810b6f3e: 8b 45 0c              movl    12(%rbp), %eax
        ;       return __kmalloc(bytes, flags);
        ffffffff810b6f41: be c0 0c 00 00        movl    $3264, %esi
        ;       addrs = kmalloc_array(prog->len + 1, sizeof(*addrs), GFP_KERNEL);
        ffffffff810b6f46: 8d 78 01              leal    1(%rax), %edi
        ;       if (unlikely(check_mul_overflow(n, size, &bytes)))
        ffffffff810b6f49: 48 c1 e7 02           shlq    $2, %rdi
        ;       return __kmalloc(bytes, flags);
        ffffffff810b6f4d: e8 8e 0c 1d 00        callq   0xffffffff81287be0 <__kmalloc>
        ;       if (!addrs) {
        ffffffff810b6f52: 48 85 c0              testq   %rax, %rax
    
    Change kmalloc_array() to kvmalloc_array() to avoid potential
    allocation error for big bpf programs.
    Signed-off-by: default avatarYonghong Song <yhs@fb.com>
    Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
    Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
    Link: https://lore.kernel.org/bpf/20210309015647.3657852-1-yhs@fb.com
    de920fc6
bpf_jit_comp.c 61.5 KB