- 13 May, 2022 4 commits
-
-
Andrii Nakryiko authored
Convert a bunch of selftests to using newly added high-level BPF map APIs. This change exposed that map_kptr selftests allocated too big buffer, which is fixed in this patch as well. Signed-off-by:
Andrii Nakryiko <andrii@kernel.org> Signed-off-by:
Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20220512220713.2617964-2-andrii@kernel.org
-
Andrii Nakryiko authored
Add high-level API wrappers for most common and typical BPF map operations that works directly on instances of struct bpf_map * (so you don't have to call bpf_map__fd()) and validate key/value size expectations. These helpers require users to specify key (and value, where appropriate) sizes when performing lookup/update/delete/etc. This forces user to actually think and validate (for themselves) those. This is a good thing as user is expected by kernel to implicitly provide correct key/value buffer sizes and kernel will just read/write necessary amount of data. If it so happens that user doesn't set up buffers correctly (which bit people for per-CPU maps especially) kernel either randomly overwrites stack data or return -EFAULT, depending on user's luck and circumstances. These high-level APIs are meant to prevent such unpleasant and hard to debug bugs. This patch also adds bpf_map_delete_elem_flags() low-level API and requires passing flags to bpf_map__delete_elem() API for consistency across all similar APIs, even though currently kernel doesn't expect any extra flags for BPF_MAP_DELETE_ELEM operation. List of map operations that get these high-level APIs: - bpf_map_lookup_elem; - bpf_map_update_elem; - bpf_map_delete_elem; - bpf_map_lookup_and_delete_elem; - bpf_map_get_next_key. Signed-off-by:
-
Alexei Starovoitov authored
Check that ld_imm64 with src_reg=1 (aka BPF_PSEUDO_FUNC) works with jit_blinding. Signed-off-by:
Alexei Starovoitov <ast@kernel.org> Signed-off-by:
Martin KaFai Lau <kafai@fb.com> Link: https://lore.kernel.org/bpf/20220513011025.13344-2-alexei.starovoitov@gmail.com
-
Alexei Starovoitov authored
The combination of jit blinding and pointers to bpf subprogs causes: [ 36.989548] BUG: unable to handle page fault for address: 0000000100000001 [ 36.990342] #PF: supervisor instruction fetch in kernel mode [ 36.990968] #PF: error_code(0x0010) - not-present page [ 36.994859] RIP: 0010:0x100000001 [ 36.995209] Code: Unable to access opcode bytes at RIP 0xffffffd7. [ 37.004091] Call Trace: [ 37.004351] <TASK> [ 37.004576] ? bpf_loop+0x4d/0x70 [ 37.004932] ? bpf_prog_3899083f75e4c5de_F+0xe3/0x13b The jit blinding logic didn't recognize that ld_imm64 with an address of bpf subprogram is a special instruction and proceeded to randomize it. By itself it wouldn't have been an issue, but jit_subprogs() logic relies on two step process to JIT all subprogs and then JIT them again when addresses of all subprogs are known. Blinding process in the first JIT phase caused second JIT to miss adjustment of special ld_imm64. Fix this issue by ignoring special ld_imm64 instructions that don't have user controlled constants and shouldn't be blinded. Fixes: 69c087ba ("bpf: Add bpf_for_each_map_elem() helper") Reported-by:
-
- 12 May, 2022 5 commits
-
-
Yuntao Wang authored
The cnt value in the 'cnt >= BPF_MAX_TRAMP_PROGS' check does not include BPF_TRAMP_MODIFY_RETURN bpf programs, so the number of the attached BPF_TRAMP_MODIFY_RETURN bpf programs in a trampoline can exceed BPF_MAX_TRAMP_PROGS. When this happens, the assignment '*progs++ = aux->prog' in bpf_trampoline_get_progs() will cause progs array overflow as the progs field in the bpf_tramp_progs struct can only hold at most BPF_MAX_TRAMP_PROGS bpf programs. Fixes: 88fd9e53 ("bpf: Refactor trampoline update code") Signed-off-by:
Yuntao Wang <ytcoode@gmail.com> Link: https://lore.kernel.org/r/20220430130803.210624-1-ytcoode@gmail.comSigned-off-by:
-
Andrii Nakryiko authored
fexit_stress is attaching maximum allowed amount of fexit programs to bpf_fentry_test1 kernel function, which is used by a bunch of other parallel tests, thus pretty frequently interfering with their execution. Given the test assumes nothing else is attaching to bpf_fentry_test1, mark it serial. Suggested-by:
-
Alexei Starovoitov authored
Feng zhou says: ==================== From: Feng Zhou <zhoufeng.zf@bytedance.com> Trace some functions, such as enqueue_task_fair, need to access the corresponding cpu, not the current cpu, and bpf_map_lookup_elem percpu map cannot do it. So add bpf_map_lookup_percpu_elem to accomplish it for percpu_array_map, percpu_hash_map, lru_percpu_hash_map. v1->v2: Addressed comments from Alexei Starovoitov. - add a selftest for bpf_map_lookup_percpu_elem. ==================== Signed-off-by: -
Feng Zhou authored
test_progs: Tests new ebpf helpers bpf_map_lookup_percpu_elem. Signed-off-by:
Feng Zhou <zhoufeng.zf@bytedance.com> Link: https://lore.kernel.org/r/20220511093854.411-3-zhoufeng.zf@bytedance.comSigned-off-by:
-
Feng Zhou authored
Add new ebpf helpers bpf_map_lookup_percpu_elem. The implementation method is relatively simple, refer to the implementation method of map_lookup_elem of percpu map, increase the parameters of cpu, and obtain it according to the specified cpu. Signed-off-by:
-
- 11 May, 2022 25 commits
-
-
Alexei Starovoitov authored
Kumar Kartikeya Dwivedi says: ==================== Fix a build time warning, and address comments from Alexei on the merged version [0]. [0]: https://lore.kernel.org/bpf/20220424214901.2743946-1-memxor@gmail.com Changelog: ---------- v1 -> v2 v1: https://lore.kernel.org/bpf/20220510211727.575686-1-memxor@gmail.com * Add Fixes tag to patch 1 * Fix test_progs-noalu32 failure in CI due to different alloc_insn (Alexei) * Remove per-CPU struct, use global struct (Alexei) ==================== Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
Check at runtime how various operations for kptr_ref affect its refcount and verify against the actual count. Signed-off-by:
Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://lore.kernel.org/r/20220511194654.765705-5-memxor@gmail.comSigned-off-by:
-
Kumar Kartikeya Dwivedi authored
This uses the newly added SEC("?foo") naming to disable autoload of programs, and then loads them one by one for the object and verifies that loading fails and matches the returned error string from verifier. This is similar to already existing verifier tests but provides coverage for BPF C. Signed-off-by: -
Kumar Kartikeya Dwivedi authored
In an effort to actually test the refcounting logic at runtime, add a refcount_t member to prog_test_ref_kfunc and use it in selftests to verify and test the whole logic more exhaustively. The kfunc calls for prog_test_member do not require runtime refcounting, as they are only used for verifier selftests, not during runtime execution. Hence, their implementation now has a WARN_ON_ONCE as it is not meant to be reachable code at runtime. It is strictly used in tests triggering failure cases in the verifier. bpf_kfunc_call_memb_release is called from map free path, since prog_test_member is embedded in map value for some verifier tests, so we skip WARN_ON_ONCE for it. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
Kernel Test Robot complained about missing static storage class annotation for bpf_kptr_xchg_proto variable. sparse: symbol 'bpf_kptr_xchg_proto' was not declared. Should it be static? This caused by missing extern definition in the header. Add it to suppress the sparse warning. Fixes: c0a5a21c ("bpf: Allow storing referenced kptr in map") Reported-by:
kernel test robot <lkp@intel.com> Signed-off-by:
-
Yonghong Song authored
With latest clang, I got the following compilation errors: .../prog_tests/test_tunnel.c:291:6: error: variable 'local_ip_map_fd' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes-uninitialized] if (attach_tc_prog(&tc_hook, -1, set_dst_prog_fd)) ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .../bpf/prog_tests/test_tunnel.c:312:6: note: uninitialized use occurs here if (local_ip_map_fd >= 0) ^~~~~~~~~~~~~~~ ... .../prog_tests/kprobe_multi_test.c:346:6: error: variable 'err' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes-uninitialized] if (IS_ERR(map)) ^~~~~~~~~~~ .../prog_tests/kprobe_multi_test.c:388:6: note: uninitialized use occurs here if (err) { ^~~ This patch fixed the above compilation errors. Signed-off-by:Yonghong Song <yhs@fb.com> Acked-by:
David Vernet <void@manifault.com> Link: https://lore.kernel.org/r/20220511184735.3670214-1-yhs@fb.comSigned-off-by:
-
Daniel Müller authored
Some of the BPF selftests are failing when running with a rather bare bones configuration based on tools/testing/selftests/bpf/config. Specifically, we see a bunch of failures due to errno 95: > test_attach_api:PASS:fentry_raw_skel_load 0 nsec > libbpf: prog 'test_kprobe_manual': failed to attach: Operation not supported > test_attach_api:FAIL:bpf_program__attach_kprobe_multi_opts unexpected error: -95 > 79 /6 kprobe_multi_test/attach_api_syms:FAIL The cause of these is that CONFIG_FPROBE is missing. With this change we add this configuration value to the BPF selftests config. Signed-off-by:
Daniel Müller <deso@posteo.net> Acked-by:
-
Alexei Starovoitov authored
Magnus Karlsson says: ==================== This patch set adds busy-poll testing to the xsk selftests. It runs exactly the same tests as with regular softirq processing, but with busy-poll enabled. I have also included a number of fixes to the selftests that have been bugging me for a while or was discovered while implementing the busy-poll support. In summary these are: * Fix the error reporting of failed tests. Each failed test used to be reported as both failed and passed, messing up things. * Added a summary test printout at the end of the test suite so that users do not have to scroll up and look at the result of both the softirq run and the busy_poll run. * Added a timeout to the tests, so that if a test locks up, we report a fail and still get to run all the other tests. * Made the stats test just look and feel like all the other tests. Makes the code simpler and the test reporting more consistent. These are the 3 last commits. * Replaced zero length packets with packets of 64 byte length. This so that some of the tests will pass after commit 726e2c59 ("veth: Ensure eth header is in skb's linear part"). * Added clean-up of the veth pair when terminating the test run. * Some smaller clean-ups of unused stuff. Note, to pass the busy-poll tests commit 8de8b71b ("xsk: Fix l2fwd for copy mode + busy poll combo") need to be present. It is present in bpf but not yet in bpf-next. Thanks: Magnus ==================== Acked-by:
Björn Töpel <bjorn@kernel.org> Signed-off-by:
-
Magnus Karlsson authored
Convert the stats tests from spinning on the getsockopt to just check getsockopt once when the Rx thread has received all the packets. The actual completion of receiving the last packet forms a natural point in time when the receiver is ready to call the getsockopt to check the stats. In the previous version , we just span on the getsockopt until we received the right answer. This could be forever or just getting the "correct" answer by shear luck. The pacing_on variable can now be dropped since all test can now handle pacing properly. Signed-off-by:
Magnus Karlsson <magnus.karlsson@intel.com> Link: https://lore.kernel.org/r/20220510115604.8717-10-magnus.karlsson@gmail.comSigned-off-by:
-
Magnus Karlsson authored
Make the stats tests look and feel just like normal tests instead of bunched under the umbrella of TEST_STATS. This means we will always run each of them even if one fails. Also gets rid of some special case code. Signed-off-by:
-
Magnus Karlsson authored
Introduce validation functions that can be optionally called by the Rx and Tx threads. These are then used to replace the Rx and Tx stats dispatchers. This so that we in the next commit can make the stats tests proper normal tests and not be some special case, as today. Signed-off-by:
-
Magnus Karlsson authored
Remove the veth pair when the tests are aborted by pressing ctrl-c. Currently in this situation, the veth pair is left on the system polluting the netdev space. Signed-off-by:
-
Magnus Karlsson authored
Add a timeout to the tests so that if all packets have not been received within 3 seconds, fail the ongoing test. Hinders a test from dead-locking if there is something wrong. Signed-off-by:
-
Magnus Karlsson authored
Fix the reporting of failed tests as it was broken in several ways. First, a failed test was reported as both failed and passed messing up the count. Second, tests were not aborted after a failure and could generate more "failures" messing up the count even more. Third, the failure reporting from the application to the shell script was wrong. It always reported pass. And finally, the handling of the failures in the launch script was not correct. Correct all this by propagating the failure up through the function calls to a calling function that can abort the test. A receiver or sender thread will mark the new variable in the test spec called fail, if a test has failed. This is then picked up by the main thread when everyone else has exited and this is then marked and propagated up to the calling script. Also add a summary function in the calling script so that a user does not have to go through the sub tests to see if something has failed. Signed-off-by:
-
Magnus Karlsson authored
Execute all xsk selftests for busy-poll mode too. Currently they were only run for the standard interrupt driven softirq mode. Replace the unused option queue-id with the new option busy-poll. Signed-off-by:
-
Magnus Karlsson authored
Do not try to send packets of zero length since they are dropped by veth after commit 726e2c59 ("veth: Ensure eth header is in skb's linear part"). Replace these two packets with packets of length 60 so that they are not dropped. Also clean up the confusing naming. MIN_PKT_SIZE was really MIN_ETH_PKT_SIZE and PKT_SIZE was both MIN_ETH_SIZE and the default packet size called just PKT_SIZE. Make it consistent by using the right define in the right place. Signed-off-by:
-
Magnus Karlsson authored
Remove the spec-file that is not used any longer from the shell scripts. Also remove an unused option. Signed-off-by:
-
Jiri Olsa authored
Adding bpf_program__set_insns that allows to set new instructions for a BPF program. This is a very advanced libbpf API and users need to know what they are doing. This should be used from prog_prepare_load_fn callback only. We can have changed instructions after calling prog_prepare_load_fn callback, reloading them. One of the users of this new API will be perf's internal BPF prologue generation. Signed-off-by:
Jiri Olsa <jolsa@kernel.org> Signed-off-by:
-
Andrii Nakryiko authored
Drop unused iteration variable, move overflow prevention check into the for loop. Fixes: 0087a681 ("libbpf: Automatically fix up BPF_MAP_TYPE_RINGBUF size, if necessary") Reported-by:
Nathan Chancellor <nathan@kernel.org> Signed-off-by:
-
Andrii Nakryiko authored
Kui-Feng Lee says: ==================== Allow users to attach a 64-bits cookie to a bpf_link of fentry, fexit, or fmod_ret. This patchset includes several major changes. - Define struct bpf_tramp_links to replace bpf_tramp_prog. struct bpf_tramp_links collects bpf_links of a trampoline - Generate a trampoline to call bpf_progs of given bpf_links. - Trampolines always set/reset bpf_run_ctx before/after calling/leaving a tracing program. - Attach a cookie to a bpf_link of fentry/fexit/fmod_ret/lsm. The value will be available when running the associated bpf_prog. Th major differences from v6: - bpf_link_create() can create links of BPF_LSM_MAC attach type. - Add a test for lsm. - Add function proto of bpf_get_attach_cookie() for lsm. - Check BPF_LSM_MAC in bpf_prog_has_trampoline(). - Adapt to the changes of LINK_CREATE made by Andrii. The major differences from v7: - Change stack_size instead of pushing/popping run_ctx. - Move cookie to bpf_tramp_link from bpf_tracing_link.. v1: https://lore.kernel.org/all/20220126214809.3868787-1-kuifeng@fb.com/ v2: https://lore.kernel.org/bpf/20220316004231.1103318-1-kuifeng@fb.com/ v3: https://lore.kernel.org/bpf/20220407192552.2343076-1-kuifeng@fb.com/ v4: https://lore.kernel.org/bpf/20220411173429.4139609-1-kuifeng@fb.com/ v5: https://lore.kernel.org/bpf/20220412165555.4146407-1-kuifeng@fb.com/ v6: https://lore.kernel.org/bpf/20220416042940.656344-1-kuifeng@fb.com/ v7: https://lore.kernel.org/bpf/20220508032117.2783209-1-kuifeng@fb.com/ ==================== Signed-off-by:
-
Kui-Feng Lee authored
Make sure BPF cookies are correct for fentry/fexit/fmod_ret/lsm. Signed-off-by:
Kui-Feng Lee <kuifeng@fb.com> Signed-off-by:
-
Kui-Feng Lee authored
Add a cookie field to the attributes of bpf_link_create(). Add bpf_program__attach_trace_opts() to attach a cookie to a link. Signed-off-by:
-
Kui-Feng Lee authored
Pass a cookie along with BPF_LINK_CREATE requests. Add a bpf_cookie field to struct bpf_tracing_link to attach a cookie. The cookie of a bpf_tracing_link is available by calling bpf_get_attach_cookie when running the BPF program of the attached link. The value of a cookie will be set at bpf_tramp_run_ctx by the trampoline of the link. Signed-off-by:
-
Kui-Feng Lee authored
BPF trampolines will create a bpf_tramp_run_ctx, a bpf_run_ctx, on stacks and set/reset the current bpf_run_ctx before/after calling a bpf_prog. Signed-off-by:
-
Kui-Feng Lee authored
Replace struct bpf_tramp_progs with struct bpf_tramp_links to collect struct bpf_tramp_link(s) for a trampoline. struct bpf_tramp_link extends bpf_link to act as a linked list node. arch_prepare_bpf_trampoline() accepts a struct bpf_tramp_links to collects all bpf_tramp_link(s) that a trampoline should call. Change BPF trampoline and bpf_struct_ops to pass bpf_tramp_links instead of bpf_tramp_progs. Signed-off-by:
-
- 10 May, 2022 6 commits
-
-
Alexei Starovoitov authored
Jiri Olsa says: ==================== hi, sending additional fix for symbol resolving in kprobe multi link requested by Alexei and Andrii [1]. This speeds up bpftrace kprobe attachment, when using pure symbols (3344 symbols) to attach: Before: # perf stat -r 5 -e cycles ./src/bpftrace -e 'kprobe:x* { } i:ms:1 { exit(); }' ... 6.5681 +- 0.0225 seconds time elapsed ( +- 0.34% ) After: # perf stat -r 5 -e cycles ./src/bpftrace -e 'kprobe:x* { } i:ms:1 { exit(); }' ... 0.5661 +- 0.0275 seconds time elapsed ( +- 4.85% ) v6 changes: - rewrote patch 1 changelog and fixed the line length [Christoph] v5 changes: - added acks [Masami] - workaround in selftest for RCU warning by filtering out several functions to attach v4 changes: - fix compile issue [kernel test robot] - added acks [Andrii] v3 changes: - renamed kallsyms_lookup_names to ftrace_lookup_symbols and moved it to ftrace.c [Masami] - added ack [Andrii] - couple small test fixes [Andrii] v2 changes (first version [2]): - removed the 2 seconds check [Alexei] - moving/forcing symbols sorting out of kallsyms_lookup_names function [Alexei] - skipping one array allocation and copy_from_user [Andrii] - several small fixes [Masami,Andrii] - build fix [kernel test robot] thanks, jirka [1] https://lore.kernel.org/bpf/CAEf4BzZtQaiUxQ-sm_hH2qKPRaqGHyOfEsW96DxtBHRaKLoL3Q@mail.gmail.com/ [2] https://lore.kernel.org/bpf/20220407125224.310255-1-jolsa@kernel.org/ ==================== Signed-off-by: -
Jiri Olsa authored
Adding test that reads all functions from ftrace available_filter_functions file and attach them all through kprobe_multi API. It also prints stats info with -v option, like on my setup: test_bench_attach: found 48712 functions test_bench_attach: attached in 1.069s test_bench_attach: detached in 0.373s Acked-by:
-
Jiri Olsa authored
Using kallsyms_lookup_names function to speed up symbols lookup in kprobe multi link attachment and replacing with it the current kprobe_multi_resolve_syms function. This speeds up bpftrace kprobe attachment: # perf stat -r 5 -e cycles ./src/bpftrace -e 'kprobe:x* { } i:ms:1 { exit(); }' ... 6.5681 +- 0.0225 seconds time elapsed ( +- 0.34% ) After: # perf stat -r 5 -e cycles ./src/bpftrace -e 'kprobe:x* { } i:ms:1 { exit(); }' ... 0.5661 +- 0.0275 seconds time elapsed ( +- 4.85% ) Acked-by: -
Jiri Olsa authored
Using ftrace_lookup_symbols to speed up symbols lookup in register_fprobe_syms API. Acked-by:
Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by:
-
Jiri Olsa authored
Adding ftrace_lookup_symbols function that resolves array of symbols with single pass over kallsyms. The user provides array of string pointers with count and pointer to allocated array for resolved values. int ftrace_lookup_symbols(const char **sorted_syms, size_t cnt, unsigned long *addrs) It iterates all kallsyms symbols and tries to loop up each in provided symbols array with bsearch. The symbols array needs to be sorted by name for this reason. We also check each symbol to pass ftrace_location, because this API will be used for fprobe symbols resolving. Suggested-by: -
Jiri Olsa authored
Making kallsyms_on_each_symbol generally available, so it can be used outside CONFIG_LIVEPATCH option in following changes. Rather than adding another ifdef option let's make the function generally available (when CONFIG_KALLSYMS option is defined). Cc: Christoph Hellwig <hch@lst.de> Reviewed-by:
-
