- 22 Jan, 2020 8 commits
-
-
Daniel Borkmann authored
Alexei Starovoitov says: ==================== The last few month BPF community has been discussing an approach to call chaining, since exiting bpt_tail_call() mechanism used in production XDP programs has plenty of downsides. The outcome of these discussion was a conclusion to implement dynamic re-linking of BPF programs. Where rootlet XDP program attached to a netdevice can programmatically define a policy of execution of other XDP programs. Such rootlet would be compiled as normal XDP program and provide a number of placeholder global functions which later can be replaced with future XDP programs. BPF trampoline, function by function verification were building blocks towards that goal. The patch 1 is a final building block. It introduces dynamic program extensions. A number of improvements like more flexible function by function verification and better libbpf api will be implemented in future patches. v1->v2: - addressed Andrii's comments - rebase ==================== Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
-
Alexei Starovoitov authored
Add program extension tests that build on top of fexit_bpf2bpf tests. Replace three global functions in previously loaded test_pkt_access.c program with three new implementations: int get_skb_len(struct __sk_buff *skb); int get_constant(long val); int get_skb_ifindex(int val, struct __sk_buff *skb, int var); New function return the same results as original only if arguments match. new_get_skb_ifindex() demonstrates that 'skb' argument doesn't have to be first and only argument of BPF program. All normal skb based accesses are available. Signed-off-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: John Fastabend <john.fastabend@gmail.com> Acked-by: Andrii Nakryiko <andriin@fb.com> Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://lore.kernel.org/bpf/20200121005348.2769920-4-ast@kernel.org
-
Alexei Starovoitov authored
Add minimal support for program extensions. bpf_object_open_opts() needs to be called with attach_prog_fd = target_prog_fd and BPF program extension needs to have in .c file section definition like SEC("freplace/func_to_be_replaced"). libbpf will search for "func_to_be_replaced" in the target_prog_fd's BTF and will pass it in attach_btf_id to the kernel. This approach works for tests, but more compex use case may need to request function name (and attach_btf_id that kernel sees) to be more dynamic. Such API will be added in future patches. Signed-off-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: John Fastabend <john.fastabend@gmail.com> Acked-by: Andrii Nakryiko <andriin@fb.com> Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://lore.kernel.org/bpf/20200121005348.2769920-3-ast@kernel.org
-
Alexei Starovoitov authored
Introduce dynamic program extensions. The users can load additional BPF functions and replace global functions in previously loaded BPF programs while these programs are executing. Global functions are verified individually by the verifier based on their types only. Hence the global function in the new program which types match older function can safely replace that corresponding function. This new function/program is called 'an extension' of old program. At load time the verifier uses (attach_prog_fd, attach_btf_id) pair to identify the function to be replaced. The BPF program type is derived from the target program into extension program. Technically bpf_verifier_ops is copied from target program. The BPF_PROG_TYPE_EXT program type is a placeholder. It has empty verifier_ops. The extension program can call the same bpf helper functions as target program. Single BPF_PROG_TYPE_EXT type is used to extend XDP, SKB and all other program types. The verifier allows only one level of replacement. Meaning that the extension program cannot recursively extend an extension. That also means that the maximum stack size is increasing from 512 to 1024 bytes and maximum function nesting level from 8 to 16. The programs don't always consume that much. The stack usage is determined by the number of on-stack variables used by the program. The verifier could have enforced 512 limit for combined original plus extension program, but it makes for difficult user experience. The main use case for extensions is to provide generic mechanism to plug external programs into policy program or function call chaining. BPF trampoline is used to track both fentry/fexit and program extensions because both are using the same nop slot at the beginning of every BPF function. Attaching fentry/fexit to a function that was replaced is not allowed. The opposite is true as well. Replacing a function that currently being analyzed with fentry/fexit is not allowed. The executable page allocated by BPF trampoline is not used by program extensions. This inefficiency will be optimized in future patches. Function by function verification of global function supports scalars and pointer to context only. Hence program extensions are supported for such class of global functions only. In the future the verifier will be extended with support to pointers to structures, arrays with sizes, etc. Signed-off-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: John Fastabend <john.fastabend@gmail.com> Acked-by: Andrii Nakryiko <andriin@fb.com> Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://lore.kernel.org/bpf/20200121005348.2769920-2-ast@kernel.org
-
Chris Down authored
When trying to compile with CONFIG_DEBUG_INFO_BTF enabled, I got this error: % make -s Failed to generate BTF for vmlinux Try to disable CONFIG_DEBUG_INFO_BTF make[3]: *** [vmlinux] Error 1 Compiling again without -s shows the true error (that pahole is missing), but since this is fatal, we should show the error unconditionally on stderr as well, not silence it using the `info` function. With this patch: % make -s BTF: .tmp_vmlinux.btf: pahole (pahole) is not available Failed to generate BTF for vmlinux Try to disable CONFIG_DEBUG_INFO_BTF make[3]: *** [vmlinux] Error 1 Signed-off-by: Chris Down <chris@chrisdown.name> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/20200122000110.GA310073@chrisdown.name
-
Daniel Díaz authored
During cross-compilation, it was discovered that LDFLAGS and LDLIBS were not being used while building binaries, leading to defaults which were not necessarily correct. OpenEmbedded reported this kind of problem: ERROR: QA Issue: No GNU_HASH in the ELF binary [...], didn't pass LDFLAGS? Signed-off-by: Daniel Díaz <daniel.diaz@linaro.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Andrii Nakryiko <andriin@fb.com> Acked-by: John Fastabend <john.fastabend@gmail.com>
-
Alexei Starovoitov authored
Restore the 'if (env->cur_state)' check that was incorrectly removed during code move. Under memory pressure env->cur_state can be freed and zeroed inside do_check(). Hence the check is necessary. Fixes: 51c39bb1 ("bpf: Introduce function-by-function verification") Reported-by: syzbot+b296579ba5015704d9fa@syzkaller.appspotmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Song Liu <songliubraving@fb.com> Link: https://lore.kernel.org/bpf/20200122024138.3385590-1-ast@kernel.org
-
Alexei Starovoitov authored
Though the second half of trampoline page is unused a task could be preempted in the middle of the first half of trampoline and two updates to trampoline would change the code from underneath the preempted task. Hence wait for tasks to voluntarily schedule or go to userspace. Add similar wait before freeing the trampoline. Fixes: fec56f58 ("bpf: Introduce BPF trampoline") Reported-by: Jann Horn <jannh@google.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Paul E. McKenney <paulmck@kernel.org> Link: https://lore.kernel.org/bpf/20200121032231.3292185-1-ast@kernel.org
-
- 21 Jan, 2020 14 commits
-
-
Björn Töpel authored
XDP sockets use the default implementation of struct sock's sk_data_ready callback, which is sock_def_readable(). This function is called in the XDP socket fast-path, and involves a retpoline. By letting sock_def_readable() have external linkage, and being called directly, the retpoline can be avoided. Signed-off-by: Björn Töpel <bjorn.topel@intel.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20200120092917.13949-1-bjorn.topel@gmail.com
-
Al Viro authored
kernel/bpf/inode.c misuses kern_path...() - it's much simpler (and more efficient, on top of that) to use user_path...() counterparts rather than bothering with doing getname() manually. Signed-off-by: Al Viro <viro@zeniv.linux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20200120232858.GF8904@ZenIV.linux.org.uk
-
Alexei Starovoitov authored
Toke Høiland-Jørgensen says: ==================== We are currently being somewhat inconsistent with the libbpf include paths, which makes it difficult to move files from the kernel into an external libbpf-using project without adjusting include paths. Having the bpf/ subdir of $INCLUDEDIR in the include path has never been a requirement for building against libbpf before, and indeed the libbpf pkg-config file doesn't include it. So let's make all libbpf includes across the kernel tree use the bpf/ prefix in their includes. Since bpftool skeleton generation emits code with a libbpf include, this also ensures that those can be used in existing external projects using the regular pkg-config include path. This turns out to be a somewhat invasive change in the number of files touched; however, the actual changes to files are fairly trivial (most of them are simply made with 'sed'). The series is split to make the change for one tool subdir at a time, while trying not to break the build along the way. It is structured like this: - Patch 1-3: Trivial fixes to Makefiles for issues I discovered while changing the include paths. - Patch 4-8: Change the include directives to use the bpf/ prefix, and updates Makefiles to make sure tools/lib/ is part of the include path, but without removing tools/lib/bpf - Patch 9-11: Remove tools/lib/bpf from include paths to make sure we don't inadvertently re-introduce includes without the bpf/ prefix. Changelog: v5: - Combine the libbpf build rules in selftests Makefile (using Andrii's suggestion for a make rule). - Re-use self-tests libbpf build for runqslower (new patch 10) - Formatting fixes v4: - Move runqslower error on missing BTF into make rule - Make sure we don't always force a rebuild selftests - Rebase on latest bpf-next (dropping patch 11) v3: - Don't add the kernel build dir to the runqslower Makefile, pass it in from selftests instead. - Use libbpf's 'make install_headers' in selftests instead of trying to generate bpf_helper_defs.h in-place (to also work on read-only filesystems). - Use a scratch builddir for both libbpf and bpftool when building in selftests. - Revert bpf_helpers.h to quoted include instead of angled include with a bpf/ prefix. - Fix a few style nits from Andrii v2: - Do a full cleanup of libbpf includes instead of just changing the bpf_helper_defs.h include. ==================== Acked-by: Andrii Nakryiko <andriin@fb.com> Tested-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
-
Toke Høiland-Jørgensen authored
To make sure no new files are introduced that doesn't include the bpf/ prefix in its #include, remove tools/lib/bpf from the include path entirely. Instead, we introduce a new header files directory under the scratch tools/ dir, and add a rule to run the 'install_headers' rule from libbpf to have a full set of consistent libbpf headers in $(OUTPUT)/tools/include/bpf, and then use $(OUTPUT)/tools/include as the include path for selftests. For consistency we also make sure we put all the scratch build files from other bpftool and libbpf into tools/build/, so everything stays within selftests/. Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/157952561246.1683545.2762245552022369203.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
This adds support for specifying the libbpf include and object paths as arguments to the runqslower Makefile, to support reusing the libbpf version built as part of the selftests. Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/157952561135.1683545.5660339645093141381.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
Since we are now consistently using the bpf/ prefix on #include directives, we don't need to include tools/lib/bpf in the include path. Remove it to make sure we don't inadvertently introduce new includes without the prefix. Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/157952561027.1683545.1976265477926794138.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
Fix all files in samples/bpf to include libbpf header files with the bpf/ prefix, to be consistent with external users of the library. Also ensure that all includes of exported libbpf header files (those that are exported on 'make install' of the library) use bracketed includes instead of quoted. To make sure no new files are introduced that doesn't include the bpf/ prefix in its include, remove tools/lib/bpf from the include path entirely, and use tools/lib instead. Fixes: 6910d7d3 ("selftests/bpf: Ensure bpf_helper_defs.h are taken from selftests dir") Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/157952560911.1683545.8795966751309534150.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
Fix perf to include libbpf header files with the bpf/ prefix, to be consistent with external users of the library. Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/157952560797.1683545.7685921032671386301.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
Fix bpftool to include libbpf header files with the bpf/ prefix, to be consistent with external users of the library. Also ensure that all includes of exported libbpf header files (those that are exported on 'make install' of the library) use bracketed includes instead of quoted. To make sure no new files are introduced that doesn't include the bpf/ prefix in its include, remove tools/lib/bpf from the include path entirely, and use tools/lib instead. Fixes: 6910d7d3 ("selftests/bpf: Ensure bpf_helper_defs.h are taken from selftests dir") Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/157952560684.1683545.4765181397974997027.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
Fix all selftests to include libbpf header files with the bpf/ prefix, to be consistent with external users of the library. Also ensure that all includes of exported libbpf header files (those that are exported on 'make install' of the library) use bracketed includes instead of quoted. To not break the build, keep the old include path until everything has been changed to the new one; a subsequent patch will remove that. Fixes: 6910d7d3 ("selftests/bpf: Ensure bpf_helper_defs.h are taken from selftests dir") Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/157952560568.1683545.9649335788846513446.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
Fix the runqslower tool to include libbpf header files with the bpf/ prefix, to be consistent with external users of the library. Also ensure that all includes of exported libbpf header files (those that are exported on 'make install' of the library) use bracketed includes instead of quoted. To not break the build, keep the old include path until everything has been changed to the new one; a subsequent patch will remove that. Fixes: 6910d7d3 ("selftests/bpf: Ensure bpf_helper_defs.h are taken from selftests dir") Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/157952560457.1683545.9913736511685743625.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
Add a VMLINUX_BTF variable with the locally-built path when calling the runqslower Makefile from selftests. This makes sure a simple 'make' invocation in the selftests dir works even when there is no BTF information for the running kernel. Do a wildcard expansion and include the same paths for BTF for the running kernel as in the runqslower Makefile, to make it possible to build selftests without having a vmlinux in the local tree. Also fix the make invocation to use $(OUTPUT)/tools as the destination directory instead of $(CURDIR)/tools. Fixes: 3a0d3092 ("selftests/bpf: Build runqslower from selftests") Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/157952560344.1683545.2723631988771664417.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
The runqslower tool refuses to build without a file to read vmlinux BTF from. The build fails with an error message to override the location by setting the VMLINUX_BTF variable if autodetection fails. However, the Makefile doesn't actually work with that override - the error message is still emitted. Fix this by including the value of VMLINUX_BTF in the expansion, and only emitting the error message if the *result* is empty. Also permit running 'make clean' even though no VMLINUX_BTF is set. Fixes: 9c01546d ("tools/bpf: Add runqslower tool to tools/bpf") Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/157952560237.1683545.17771785178857224877.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
The 'clean' rule in the samples/bpf Makefile tries to remove backup files (ending in ~). However, if no such files exist, it will instead try to remove the user's home directory. While the attempt is mostly harmless, it does lead to a somewhat scary warning like this: rm: cannot remove '~': Is a directory Fix this by using find instead of shell expansion to locate any actual backup files that need to be removed. Fixes: b62a796c ("samples/bpf: allow make to be run from samples/bpf/ directory") Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> Link: https://lore.kernel.org/bpf/157952560126.1683545.7273054725976032511.stgit@toke.dk
-
- 20 Jan, 2020 3 commits
-
-
Hangbin Liu authored
The same with commit 4e59afbb ("selftests/bpf: skip nmi test when perf hw events are disabled"), it would make more sense to skip the test_stacktrace_build_id_nmi test if the setup (e.g. virtual machines) has disabled hardware perf events. Fixes: 13790d1c ("bpf: add selftest for stackmap with build_id in NMI context") Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: John Fastabend <john.fastabend@gmail.com> Link: https://lore.kernel.org/bpf/20200117100656.10359-1-liuhangbin@gmail.com
-
Stanislav Fomichev authored
After commit 0d13bfce ("libbpf: Don't require root for bpf_object__open()") we no longer load BTF during bpf_object__open(), so let's remove the expectation from test_btf that the fd is not -1. The test currently fails. Before: BTF libbpf test[1] (test_btf_haskv.o): do_test_file:4152:FAIL bpf_object__btf_fd: -1 BTF libbpf test[2] (test_btf_newkv.o): do_test_file:4152:FAIL bpf_object__btf_fd: -1 BTF libbpf test[3] (test_btf_nokv.o): do_test_file:4152:FAIL bpf_object__btf_fd: -1 After: BTF libbpf test[1] (test_btf_haskv.o): OK BTF libbpf test[2] (test_btf_newkv.o): OK BTF libbpf test[3] (test_btf_nokv.o): OK Fixes: 0d13bfce ("libbpf: Don't require root for bpf_object__open()") Signed-off-by: Stanislav Fomichev <sdf@google.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: John Fastabend <john.fastabend@gmail.com> Link: https://lore.kernel.org/bpf/20200118010546.74279-1-sdf@google.com
-
Brian Vazquez authored
Generic update/delete batch ops functions were using __bpf_copy_key without properly freeing the memory. Handle the memory allocation and copy_from_user separately. Fixes: aa2e93b8 ("bpf: Add generic support for update and delete batch ops") Reported-by: Dan Carpenter <dan.carpenter@oracle.com> Signed-off-by: Brian Vazquez <brianvv@google.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Yonghong Song <yhs@fb.com> Link: https://lore.kernel.org/bpf/20200119194040.128369-1-brianvv@google.com
-
- 17 Jan, 2020 10 commits
-
-
KP Singh authored
As more programs (TRACING, STRUCT_OPS, and upcoming LSM) use vmlinux BTF information, loading the BTF vmlinux information for every program in an object is sub-optimal. The fix was originally proposed in: https://lore.kernel.org/bpf/CAEf4BzZodr3LKJuM7QwD38BiEH02Cc1UbtnGpVkCJ00Mf+V_Qg@mail.gmail.com/ The btf_vmlinux is populated in the object if any of the programs in the object requires it just before the programs are loaded and freed after the programs finish loading. Reported-by: Andrii Nakryiko <andrii.nakryiko@gmail.com> Signed-off-by: KP Singh <kpsingh@google.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Reviewed-by: Brendan Jackman <jackmanb@chromium.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/20200117212825.11755-1-kpsingh@chromium.org
-
Andrii Nakryiko authored
Prevent potential overflow performed in 32-bit integers, before assigning result to size_t. Reported by LGTM static analysis. Fixes: eba9c5f4 ("libbpf: Refactor global data map initialization") Signed-off-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/20200117060801.1311525-4-andriin@fb.com
-
Andrii Nakryiko authored
Current implementation of bpf_object's BTF initialization is very convoluted and thus prone to errors. It doesn't have to be like that. This patch simplifies it significantly. This code also triggered static analysis issues over logically dead code due to redundant error checks. This simplification should fix that as well. Signed-off-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/20200117060801.1311525-3-andriin@fb.com
-
Andrii Nakryiko authored
Fix missing jump to error handling in btf_dump__new, found by Coverity static code analysis. Fixes: 9f81654e ("libbpf: Expose BTF-to-C type declaration emitting API") Signed-off-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/20200117060801.1311525-2-andriin@fb.com
-
YueHaibing authored
kernel/bpf/syscall.c: In function generic_map_lookup_batch: kernel/bpf/syscall.c:1339:7: warning: variable first_key set but not used [-Wunused-but-set-variable] It is never used, so remove it. Reported-by: Hulk Robot <hulkci@huawei.com> Signed-off-by: YueHaibing <yuehaibing@huawei.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Brian Vazquez <brianvv@google.com> Link: https://lore.kernel.org/bpf/20200116145300.59056-1-yuehaibing@huawei.com
-
Alexei Starovoitov authored
Toke Høiland-Jørgensen says: ==================== Since commit 96360004 ("xdp: Make devmap flush_list common for all map instances"), devmap flushing is a global operation instead of tied to a particular map. This means that with a bit of refactoring, we can finally fix the performance delta between the bpf_redirect_map() and bpf_redirect() helper functions, by introducing bulking for the latter as well. This series makes this change by moving the data structure used for the bulking into struct net_device itself, so we can access it even when there is not devmap. Once this is done, moving the bpf_redirect() helper to use the bulking mechanism becomes quite trivial, and brings bpf_redirect() up to the same as bpf_redirect_map(): Before: After: 1 CPU: bpf_redirect_map: 8.4 Mpps 8.4 Mpps (no change) bpf_redirect: 5.0 Mpps 8.4 Mpps (+68%) 2 CPUs: bpf_redirect_map: 15.9 Mpps 16.1 Mpps (+1% or ~no change) bpf_redirect: 9.5 Mpps 15.9 Mpps (+67%) After this patch series, the only semantics different between the two variants of the bpf() helper (apart from the absence of a map argument, obviously) is that the _map() variant will return an error if passed an invalid map index, whereas the bpf_redirect() helper will succeed, but drop packets on xdp_do_redirect(). This is because the helper has no reference to the calling netdev, so unfortunately we can't do the ifindex lookup directly in the helper. Changelog: v3: - Switch two more fields to avoid a list_head spanning two cache lines - Include Jesper's tracepoint patch - Also rename xdp_do_flush_map() - Fix a few nits from Maciej v2: - Consolidate code paths and tracepoints for map and non-map redirect variants (Björn) - Add performance data for 2-CPU test (Jesper) - Move fields to avoid shifting cache lines in struct net_device (Eric) ==================== Signed-off-by: Alexei Starovoitov <ast@kernel.org>
-
Jesper Dangaard Brouer authored
Now that we don't have a reference to a devmap when flushing the device bulk queue, let's change the the devmap_xmit tracepoint to remote the map_id and map_index fields entirely. Rearrange the fields so 'drops' and 'sent' stay in the same position in the tracepoint struct, to make it possible for the xdp_monitor utility to read both the old and the new format. Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/157918768613.1458396.9165902403373826572.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
Since the bulk queue used by XDP_REDIRECT now lives in struct net_device, we can re-use the bulking for the non-map version of the bpf_redirect() helper. This is a simple matter of having xdp_do_redirect_slow() queue the frame on the bulk queue instead of sending it out with __bpf_tx_xdp(). Unfortunately we can't make the bpf_redirect() helper return an error if the ifindex doesn't exit (as bpf_redirect_map() does), because we don't have a reference to the network namespace of the ingress device at the time the helper is called. So we have to leave it as-is and keep the device lookup in xdp_do_redirect_slow(). Since this leaves less reason to have the non-map redirect code in a separate function, so we get rid of the xdp_do_redirect_slow() function entirely. This does lose us the tracepoint disambiguation, but fortunately the xdp_redirect and xdp_redirect_map tracepoints use the same tracepoint entry structures. This means both can contain a map index, so we can just amend the tracepoint definitions so we always emit the xdp_redirect(_err) tracepoints, but with the map ID only populated if a map is present. This means we retire the xdp_redirect_map(_err) tracepoints entirely, but keep the definitions around in case someone is still listening for them. With this change, the performance of the xdp_redirect sample program goes from 5Mpps to 8.4Mpps (a 68% increase). Since the flush functions are no longer map-specific, rename the flush() functions to drop _map from their names. One of the renamed functions is the xdp_do_flush_map() callback used in all the xdp-enabled drivers. To keep from having to update all drivers, use a #define to keep the old name working, and only update the virtual drivers in this patch. Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: John Fastabend <john.fastabend@gmail.com> Link: https://lore.kernel.org/bpf/157918768505.1458396.17518057312953572912.stgit@toke.dk
-
Toke Høiland-Jørgensen authored
Commit 96360004 ("xdp: Make devmap flush_list common for all map instances"), changed devmap flushing to be a global operation instead of a per-map operation. However, the queue structure used for bulking was still allocated as part of the containing map. This patch moves the devmap bulk queue into struct net_device. The motivation for this is reusing it for the non-map variant of XDP_REDIRECT, which will be changed in a subsequent commit. To avoid other fields of struct net_device moving to different cache lines, we also move a couple of other members around. We defer the actual allocation of the bulk queue structure until the NETDEV_REGISTER notification devmap.c. This makes it possible to check for ndo_xdp_xmit support before allocating the structure, which is not possible at the time struct net_device is allocated. However, we keep the freeing in free_netdev() to avoid adding another RCU callback on NETDEV_UNREGISTER. Because of this change, we lose the reference back to the map that originated the redirect, so change the tracepoint to always return 0 as the map ID and index. Otherwise no functional change is intended with this patch. After this patch, the relevant part of struct net_device looks like this, according to pahole: /* --- cacheline 14 boundary (896 bytes) --- */ struct netdev_queue * _tx __attribute__((__aligned__(64))); /* 896 8 */ unsigned int num_tx_queues; /* 904 4 */ unsigned int real_num_tx_queues; /* 908 4 */ struct Qdisc * qdisc; /* 912 8 */ unsigned int tx_queue_len; /* 920 4 */ spinlock_t tx_global_lock; /* 924 4 */ struct xdp_dev_bulk_queue * xdp_bulkq; /* 928 8 */ struct xps_dev_maps * xps_cpus_map; /* 936 8 */ struct xps_dev_maps * xps_rxqs_map; /* 944 8 */ struct mini_Qdisc * miniq_egress; /* 952 8 */ /* --- cacheline 15 boundary (960 bytes) --- */ struct hlist_head qdisc_hash[16]; /* 960 128 */ /* --- cacheline 17 boundary (1088 bytes) --- */ struct timer_list watchdog_timer; /* 1088 40 */ /* XXX last struct has 4 bytes of padding */ int watchdog_timeo; /* 1128 4 */ /* XXX 4 bytes hole, try to pack */ struct list_head todo_list; /* 1136 16 */ /* --- cacheline 18 boundary (1152 bytes) --- */ Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Björn Töpel <bjorn.topel@intel.com> Acked-by: John Fastabend <john.fastabend@gmail.com> Link: https://lore.kernel.org/bpf/157918768397.1458396.12673224324627072349.stgit@toke.dk
-
Andrii Nakryiko authored
Revert bpf_helpers.h's change to include auto-generated bpf_helper_defs.h through <> instead of "", which causes it to be searched in include path. This can break existing applications that don't have their include path pointing directly to where libbpf installs its headers. There is ongoing work to make all (not just bpf_helper_defs.h) includes more consistent across libbpf and its consumers, but this unbreaks user code as is right now without any regressions. Selftests still behave sub-optimally (taking bpf_helper_defs.h from libbpf's source directory, if it's present there), which will be fixed in subsequent patches. Fixes: 6910d7d3 ("selftests/bpf: Ensure bpf_helper_defs.h are taken from selftests dir") Reported-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/20200117004103.148068-1-andriin@fb.com
-
- 16 Jan, 2020 3 commits
-
-
Yonghong Song authored
Alexei observed that test_progs send_signal may fail if run with command line "./test_progs" and the tests will pass if just run "./test_progs -n 40". I observed similar issue with nmi subtest failure and added a delay 100 us in Commit ab8b7f0c ("tools/bpf: Add self tests for bpf_send_signal_thread()") and the problem is gone for me. But the issue still exists in Alexei's testing environment. The current code uses sample_freq = 50 (50 events/second), which may not be enough. But if the sample_freq value is larger than sysctl kernel/perf_event_max_sample_rate, the perf_event_open syscall will fail. This patch changed nmi perf testing to use sample_period = 1, which means trying to sampling every event. This seems fixing the issue. Fixes: ab8b7f0c ("tools/bpf: Add self tests for bpf_send_signal_thread()") Signed-off-by: Yonghong Song <yhs@fb.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/20200116174004.1522812-1-yhs@fb.com
-
Brian Vazquez authored
bpf_attr doesn't required to be declared with '= {}' as memset is used in the code. Fixes: 2ab3d86e ("libbpf: Add libbpf support to batch ops") Reported-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Brian Vazquez <brianvv@google.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20200116045918.75597-1-brianvv@google.com
-
Andrii Nakryiko authored
Add ability to specify a list of test name substrings for selecting which tests to run. So now -t is accepting a comma-separated list of strings, similarly to how -n accepts a comma-separated list of test numbers. Additionally, add ability to blacklist tests by name. Blacklist takes precedence over whitelist. Blacklisting is important for cases where it's known that some tests can't pass (e.g., due to perf hardware events that are not available within VM). This is going to be used for libbpf testing in Travis CI in its Github repo. Example runs with just whitelist and whitelist + blacklist: $ sudo ./test_progs -tattach,core/existence #1 attach_probe:OK #6 cgroup_attach_autodetach:OK #7 cgroup_attach_multi:OK #8 cgroup_attach_override:OK #9 core_extern:OK #10/44 existence:OK #10/45 existence___minimal:OK #10/46 existence__err_int_sz:OK #10/47 existence__err_int_type:OK #10/48 existence__err_int_kind:OK #10/49 existence__err_arr_kind:OK #10/50 existence__err_arr_value_type:OK #10/51 existence__err_struct_type:OK #10 core_reloc:OK #19 flow_dissector_reattach:OK #60 tp_attach_query:OK Summary: 8/8 PASSED, 0 SKIPPED, 0 FAILED $ sudo ./test_progs -tattach,core/existence -bcgroup,flow/arr #1 attach_probe:OK #9 core_extern:OK #10/44 existence:OK #10/45 existence___minimal:OK #10/46 existence__err_int_sz:OK #10/47 existence__err_int_type:OK #10/48 existence__err_int_kind:OK #10/51 existence__err_struct_type:OK #10 core_reloc:OK #60 tp_attach_query:OK Summary: 4/6 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Cc: Julia Kartseva <hex@fb.com> Link: https://lore.kernel.org/bpf/20200116005549.3644118-1-andriin@fb.com
-
- 15 Jan, 2020 2 commits
-
-
Alexei Starovoitov authored
Martin Lau says: ==================== When a map is storing a kernel's struct, its map_info->btf_vmlinux_value_type_id is set. The first map type supporting it is BPF_MAP_TYPE_STRUCT_OPS. This series adds support to dump this kind of map with BTF. The first two patches are bug fixes which are only applicable to bpf-next. Please see individual patches for details. v3: - Remove unnecessary #include "libbpf_internal.h" from patch 5 v2: - Expose bpf_find_kernel_btf() as a LIBBPF_API in patch 3 (Andrii) - Cache btf_vmlinux in bpftool/map.c (Andrii) ==================== Signed-off-by: Alexei Starovoitov <ast@kernel.org>
-
Martin KaFai Lau authored
This patch makes bpftool support dumping a map's value properly when the map's value type is a type of the running kernel's btf. (i.e. map_info.btf_vmlinux_value_type_id is set instead of map_info.btf_value_type_id). The first usecase is for the BPF_MAP_TYPE_STRUCT_OPS. Signed-off-by: Martin KaFai Lau <kafai@fb.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/20200115230044.1103008-1-kafai@fb.com
-