Commit 93b8713d authored by Alexei Starovoitov's avatar Alexei Starovoitov

Merge branch 'bpf: Support multi-attach for freplace'

Toke Høiland-Jørgensen says:

====================
This series adds support attaching freplace BPF programs to multiple targets.
This is needed to support incremental attachment of multiple XDP programs using
the libxdp dispatcher model.

Patch 1 moves prog_aux->linked_prog and the trampoline to be embedded in
bpf_tracing_link on attach, and freed by the link release logic, and introduces
a mutex to protect the writing of the pointers in prog->aux.

Based on this refactoring (and previously applied patches), it becomes pretty
straight-forward to support multiple-attach for freplace programs (patch 2).
This is simply a matter of creating a second bpf_tracing_link if a target is
supplied. However, for API consistency with other types of link attach, this
option is added to the BPF_LINK_CREATE API instead of extending
bpf_raw_tracepoint_open().

Patch 3 is a port of Jiri Olsa's patch to support fentry/fexit on freplace
programs. His approach of getting the target type from the target program
reference no longer works after we've gotten rid of linked_prog (because the
bpf_tracing_link reference disappears on attach). Instead, we used the saved
reference to the target prog type that is also used to verify compatibility on
secondary freplace attachment.

Patch 4 is the accompanying libbpf update, and patches 5-7 are selftests: patch
5 tests for the multi-freplace functionality itself; patch 6 is Jiri's previous
selftest for the fentry-to-freplace fix; patch 7 is a test for the change
introduced in the previously-applied patches, blocking MODIFY_RETURN functions
from attaching to other BPF programs.

With this series, libxdp and xdp-tools can successfully attach multiple programs
one at a time. To play with this, use the 'freplace-multi-attach' branch of
xdp-tools:

$ git clone --recurse-submodules --branch freplace-multi-attach https://github.com/xdp-project/xdp-tools
$ cd xdp-tools/xdp-loader
$ make
$ sudo ./xdp-loader load veth0 ../lib/testing/xdp_drop.o
$ sudo ./xdp-loader load veth0 ../lib/testing/xdp_pass.o
$ sudo ./xdp-loader status

The series is also available here:
https://git.kernel.org/pub/scm/linux/kernel/git/toke/linux.git/log/?h=bpf-freplace-multi-attach-alt-10

Changelog:

v10:
  - Dial back the s/tgt_/dst_/ replacement a bit
  - Fix smatch warning (from ktest robot)
  - Rebase to bpf-next, drop already-applied patches

v9:
  - Clarify commit message of patch 3
  - Add new struct bpf_attach_target_info for returning from
    bpf_check_attach_target() and passing to bpf_trampoline_get()
  - Move trampoline key computation into a helper
  - Make sure we don't break bpffs debug umh
  - Add some comment blocks explaining the logic flow in
    bpf_tracing_prog_attach()
  - s/tgt_/dst_/ in prog->aux, and for local variables using those members
  - Always drop dst_trampoline and dst_prog from prog->aux on first attach
  - Don't remove syscall fmod_ret test from selftest benchmarks
  - Add saved_ prefix to dst_{prog,attach}_type members in prog_aux
  - Drop prog argument from check_attach_modify_return()
  - Add comment about possible NULL of tr_link->tgt_prog on link_release()

v8:
  - Add a separate error message when trying to attach FMOD_REPLACE to tgt_prog
  - Better error messages in bpf_program__attach_freplace()
  - Don't lock mutex when setting tgt_* pointers in prog create and verifier
  - Remove fmod_ret programs from benchmarks in selftests (new patch 11)
  - Fix a few other nits in selftests

v7:
  - Add back missing ptype == prog->type check in link_create()
  - Use tracing_bpf_link_attach() instead of separate freplace_bpf_link_attach()
  - Don't break attachment of bpf_iters in libbpf (by clobbering link_create.iter_info)

v6:
  - Rebase to latest bpf-next
  - Simplify logic in bpf_tracing_prog_attach()
  - Don't create a new attach_type for link_create(), disambiguate on prog->type
    instead
  - Use raw_tracepoint_open() in libbpf bpf_program__attach_ftrace() if called
    with NULL target
  - Switch bpf_program__attach_ftrace() to take function name as parameter
    instead of btf_id
  - Add a patch disallowing MODIFY_RETURN programs from attaching to other BPF
    programs, and an accompanying selftest (patches 1 and 10)

v5:
  - Fix typo in inline function definition of bpf_trampoline_get()
  - Don't put bpf_tracing_link in prog->aux, use a mutex to protect tgt_prog and
    trampoline instead, and move them to the link on attach.
  - Restore Jiri as author of the last selftest patch

v4:
  - Cleanup the refactored check_attach_btf_id() to make the logic easier to follow
  - Fix cleanup paths for bpf_tracing_link
  - Use xchg() for removing the bpf_tracing_link from prog->aux and restore on (some) failures
  - Use BPF_LINK_CREATE operation to create link with target instead of extending raw_tracepoint_open
  - Fold update of tools/ UAPI header into main patch
  - Update arg dereference patch to use skeletons and set_attach_target()

v3:
  - Get rid of prog_aux->linked_prog entirely in favour of a bpf_tracing_link
  - Incorporate Jiri's fix for attaching fentry to freplace programs

v2:
  - Drop the log arguments from bpf_raw_tracepoint_open
  - Fix kbot errors
  - Rebase to latest bpf-next
====================
Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
parents 85e3f318 bee4b7e6
......@@ -640,8 +640,8 @@ static __always_inline unsigned int bpf_dispatcher_nop_func(
return bpf_func(ctx, insnsi);
}
#ifdef CONFIG_BPF_JIT
int bpf_trampoline_link_prog(struct bpf_prog *prog);
int bpf_trampoline_unlink_prog(struct bpf_prog *prog);
int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr);
int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr);
struct bpf_trampoline *bpf_trampoline_get(u64 key,
struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
......@@ -688,11 +688,13 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym);
void bpf_ksym_add(struct bpf_ksym *ksym);
void bpf_ksym_del(struct bpf_ksym *ksym);
#else
static inline int bpf_trampoline_link_prog(struct bpf_prog *prog)
static inline int bpf_trampoline_link_prog(struct bpf_prog *prog,
struct bpf_trampoline *tr)
{
return -ENOTSUPP;
}
static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog)
static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog,
struct bpf_trampoline *tr)
{
return -ENOTSUPP;
}
......@@ -763,7 +765,11 @@ struct bpf_prog_aux {
u32 max_rdonly_access;
u32 max_rdwr_access;
const struct bpf_ctx_arg_aux *ctx_arg_info;
struct bpf_prog *linked_prog;
struct mutex dst_mutex; /* protects dst_* pointers below, *after* prog becomes visible */
struct bpf_prog *dst_prog;
struct bpf_trampoline *dst_trampoline;
enum bpf_prog_type saved_dst_prog_type;
enum bpf_attach_type saved_dst_attach_type;
bool verifier_zext; /* Zero extensions has been inserted by verifier. */
bool offload_requested;
bool attach_btf_trace; /* true if attaching to BTF-enabled raw tp */
......@@ -771,7 +777,6 @@ struct bpf_prog_aux {
bool sleepable;
bool tail_call_reachable;
enum bpf_tramp_prog_type trampoline_prog_type;
struct bpf_trampoline *trampoline;
struct hlist_node tramp_hlist;
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
......
......@@ -639,8 +639,13 @@ union bpf_attr {
};
__u32 attach_type; /* attach type */
__u32 flags; /* extra flags */
union {
__u32 target_btf_id; /* btf_id of target to attach to */
struct {
__aligned_u64 iter_info; /* extra bpf_iter_link_info */
__u32 iter_info_len; /* iter_info length */
};
};
} link_create;
struct { /* struct used by BPF_LINK_UPDATE command */
......
......@@ -4428,7 +4428,7 @@ struct btf *btf_parse_vmlinux(void)
struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog)
{
struct bpf_prog *tgt_prog = prog->aux->linked_prog;
struct bpf_prog *tgt_prog = prog->aux->dst_prog;
if (tgt_prog) {
return tgt_prog->aux->btf;
......@@ -4455,7 +4455,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
struct bpf_insn_access_aux *info)
{
const struct btf_type *t = prog->aux->attach_func_proto;
struct bpf_prog *tgt_prog = prog->aux->linked_prog;
struct bpf_prog *tgt_prog = prog->aux->dst_prog;
struct btf *btf = bpf_prog_get_target_btf(prog);
const char *tname = prog->aux->attach_func_name;
struct bpf_verifier_log *log = info->log;
......@@ -4582,7 +4582,14 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
info->reg_type = PTR_TO_BTF_ID;
if (tgt_prog) {
ret = btf_translate_to_vmlinux(log, btf, t, tgt_prog->type, arg);
enum bpf_prog_type tgt_type;
if (tgt_prog->type == BPF_PROG_TYPE_EXT)
tgt_type = tgt_prog->aux->saved_dst_prog_type;
else
tgt_type = tgt_prog->type;
ret = btf_translate_to_vmlinux(log, btf, t, tgt_type, arg);
if (ret > 0) {
info->btf_id = ret;
return true;
......@@ -5281,7 +5288,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog,
return -EFAULT;
}
if (prog_type == BPF_PROG_TYPE_EXT)
prog_type = prog->aux->linked_prog->type;
prog_type = prog->aux->dst_prog->type;
t = btf_type_by_id(btf, t->type);
if (!t || !btf_type_is_func_proto(t)) {
......
......@@ -99,6 +99,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag
INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode);
mutex_init(&fp->aux->used_maps_mutex);
mutex_init(&fp->aux->dst_mutex);
return fp;
}
......@@ -255,6 +256,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
{
if (fp->aux) {
mutex_destroy(&fp->aux->used_maps_mutex);
mutex_destroy(&fp->aux->dst_mutex);
free_percpu(fp->aux->stats);
kfree(fp->aux->poke_tab);
kfree(fp->aux);
......@@ -2138,7 +2140,8 @@ static void bpf_prog_free_deferred(struct work_struct *work)
if (aux->prog->has_callchain_buf)
put_callchain_buffers();
#endif
bpf_trampoline_put(aux->trampoline);
if (aux->dst_trampoline)
bpf_trampoline_put(aux->dst_trampoline);
for (i = 0; i < aux->func_cnt; i++)
bpf_jit_free(aux->func[i]);
if (aux->func_cnt) {
......@@ -2154,8 +2157,8 @@ void bpf_prog_free(struct bpf_prog *fp)
{
struct bpf_prog_aux *aux = fp->aux;
if (aux->linked_prog)
bpf_prog_put(aux->linked_prog);
if (aux->dst_prog)
bpf_prog_put(aux->dst_prog);
INIT_WORK(&aux->work, bpf_prog_free_deferred);
schedule_work(&aux->work);
}
......
......@@ -42,7 +42,7 @@ struct bpf_prog_aux {
__u32 id;
char name[16];
const char *attach_func_name;
struct bpf_prog *linked_prog;
struct bpf_prog *dst_prog;
struct bpf_func_info *func_info;
struct btf *btf;
};
......@@ -108,7 +108,7 @@ int dump_bpf_prog(struct bpf_iter__bpf_prog *ctx)
BPF_SEQ_PRINTF(seq, "%4u %-16s %s %s\n", aux->id,
get_name(aux->btf, aux->func_info[0].type_id, aux->name),
aux->attach_func_name, aux->linked_prog->aux->name);
aux->attach_func_name, aux->dst_prog->aux->name);
return 0;
}
char LICENSE[] SEC("license") = "GPL";
......@@ -4,6 +4,7 @@
#include <linux/bpf.h>
#include <linux/bpf_trace.h>
#include <linux/bpf_lirc.h>
#include <linux/bpf_verifier.h>
#include <linux/btf.h>
#include <linux/syscalls.h>
#include <linux/slab.h>
......@@ -2154,14 +2155,14 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr)
prog->expected_attach_type = attr->expected_attach_type;
prog->aux->attach_btf_id = attr->attach_btf_id;
if (attr->attach_prog_fd) {
struct bpf_prog *tgt_prog;
struct bpf_prog *dst_prog;
tgt_prog = bpf_prog_get(attr->attach_prog_fd);
if (IS_ERR(tgt_prog)) {
err = PTR_ERR(tgt_prog);
dst_prog = bpf_prog_get(attr->attach_prog_fd);
if (IS_ERR(dst_prog)) {
err = PTR_ERR(dst_prog);
goto free_prog_nouncharge;
}
prog->aux->linked_prog = tgt_prog;
prog->aux->dst_prog = dst_prog;
}
prog->aux->offload_requested = !!attr->prog_ifindex;
......@@ -2498,11 +2499,23 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd)
struct bpf_tracing_link {
struct bpf_link link;
enum bpf_attach_type attach_type;
struct bpf_trampoline *trampoline;
struct bpf_prog *tgt_prog;
};
static void bpf_tracing_link_release(struct bpf_link *link)
{
WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog));
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link);
WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog,
tr_link->trampoline));
bpf_trampoline_put(tr_link->trampoline);
/* tgt_prog is NULL if target is a kernel function */
if (tr_link->tgt_prog)
bpf_prog_put(tr_link->tgt_prog);
}
static void bpf_tracing_link_dealloc(struct bpf_link *link)
......@@ -2542,10 +2555,15 @@ static const struct bpf_link_ops bpf_tracing_link_lops = {
.fill_link_info = bpf_tracing_link_fill_link_info,
};
static int bpf_tracing_prog_attach(struct bpf_prog *prog)
static int bpf_tracing_prog_attach(struct bpf_prog *prog,
int tgt_prog_fd,
u32 btf_id)
{
struct bpf_link_primer link_primer;
struct bpf_prog *tgt_prog = NULL;
struct bpf_trampoline *tr = NULL;
struct bpf_tracing_link *link;
u64 key = 0;
int err;
switch (prog->type) {
......@@ -2574,6 +2592,28 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog)
goto out_put_prog;
}
if (!!tgt_prog_fd != !!btf_id) {
err = -EINVAL;
goto out_put_prog;
}
if (tgt_prog_fd) {
/* For now we only allow new targets for BPF_PROG_TYPE_EXT */
if (prog->type != BPF_PROG_TYPE_EXT) {
err = -EINVAL;
goto out_put_prog;
}
tgt_prog = bpf_prog_get(tgt_prog_fd);
if (IS_ERR(tgt_prog)) {
err = PTR_ERR(tgt_prog);
tgt_prog = NULL;
goto out_put_prog;
}
key = bpf_trampoline_compute_key(tgt_prog, btf_id);
}
link = kzalloc(sizeof(*link), GFP_USER);
if (!link) {
err = -ENOMEM;
......@@ -2583,20 +2623,100 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog)
&bpf_tracing_link_lops, prog);
link->attach_type = prog->expected_attach_type;
err = bpf_link_prime(&link->link, &link_primer);
if (err) {
kfree(link);
goto out_put_prog;
mutex_lock(&prog->aux->dst_mutex);
/* There are a few possible cases here:
*
* - if prog->aux->dst_trampoline is set, the program was just loaded
* and not yet attached to anything, so we can use the values stored
* in prog->aux
*
* - if prog->aux->dst_trampoline is NULL, the program has already been
* attached to a target and its initial target was cleared (below)
*
* - if tgt_prog != NULL, the caller specified tgt_prog_fd +
* target_btf_id using the link_create API.
*
* - if tgt_prog == NULL when this function was called using the old
* raw_tracepoint_open API, and we need a target from prog->aux
*
* The combination of no saved target in prog->aux, and no target
* specified on load is illegal, and we reject that here.
*/
if (!prog->aux->dst_trampoline && !tgt_prog) {
err = -ENOENT;
goto out_unlock;
}
if (!prog->aux->dst_trampoline ||
(key && key != prog->aux->dst_trampoline->key)) {
/* If there is no saved target, or the specified target is
* different from the destination specified at load time, we
* need a new trampoline and a check for compatibility
*/
struct bpf_attach_target_info tgt_info = {};
err = bpf_check_attach_target(NULL, prog, tgt_prog, btf_id,
&tgt_info);
if (err)
goto out_unlock;
tr = bpf_trampoline_get(key, &tgt_info);
if (!tr) {
err = -ENOMEM;
goto out_unlock;
}
} else {
/* The caller didn't specify a target, or the target was the
* same as the destination supplied during program load. This
* means we can reuse the trampoline and reference from program
* load time, and there is no need to allocate a new one. This
* can only happen once for any program, as the saved values in
* prog->aux are cleared below.
*/
tr = prog->aux->dst_trampoline;
tgt_prog = prog->aux->dst_prog;
}
err = bpf_trampoline_link_prog(prog);
err = bpf_link_prime(&link->link, &link_primer);
if (err)
goto out_unlock;
err = bpf_trampoline_link_prog(prog, tr);
if (err) {
bpf_link_cleanup(&link_primer);
goto out_put_prog;
link = NULL;
goto out_unlock;
}
link->tgt_prog = tgt_prog;
link->trampoline = tr;
/* Always clear the trampoline and target prog from prog->aux to make
* sure the original attach destination is not kept alive after a
* program is (re-)attached to another target.
*/
if (prog->aux->dst_prog &&
(tgt_prog_fd || tr != prog->aux->dst_trampoline))
/* got extra prog ref from syscall, or attaching to different prog */
bpf_prog_put(prog->aux->dst_prog);
if (prog->aux->dst_trampoline && tr != prog->aux->dst_trampoline)
/* we allocated a new trampoline, so free the old one */
bpf_trampoline_put(prog->aux->dst_trampoline);
prog->aux->dst_prog = NULL;
prog->aux->dst_trampoline = NULL;
mutex_unlock(&prog->aux->dst_mutex);
return bpf_link_settle(&link_primer);
out_unlock:
if (tr && tr != prog->aux->dst_trampoline)
bpf_trampoline_put(tr);
mutex_unlock(&prog->aux->dst_mutex);
kfree(link);
out_put_prog:
if (tgt_prog_fd && tgt_prog)
bpf_prog_put(tgt_prog);
bpf_prog_put(prog);
return err;
}
......@@ -2710,7 +2830,7 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr)
tp_name = prog->aux->attach_func_name;
break;
}
return bpf_tracing_prog_attach(prog);
return bpf_tracing_prog_attach(prog, 0, 0);
case BPF_PROG_TYPE_RAW_TRACEPOINT:
case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
if (strncpy_from_user(buf,
......@@ -3894,10 +4014,15 @@ static int bpf_map_do_batch(const union bpf_attr *attr,
static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
if (attr->link_create.attach_type == BPF_TRACE_ITER &&
prog->expected_attach_type == BPF_TRACE_ITER)
return bpf_iter_link_attach(attr, prog);
if (attr->link_create.attach_type != prog->expected_attach_type)
return -EINVAL;
if (prog->expected_attach_type == BPF_TRACE_ITER)
return bpf_iter_link_attach(attr, prog);
else if (prog->type == BPF_PROG_TYPE_EXT)
return bpf_tracing_prog_attach(prog,
attr->link_create.target_fd,
attr->link_create.target_btf_id);
return -EINVAL;
}
......@@ -3911,18 +4036,25 @@ static int link_create(union bpf_attr *attr)
if (CHECK_ATTR(BPF_LINK_CREATE))
return -EINVAL;
ptype = attach_type_to_prog_type(attr->link_create.attach_type);
if (ptype == BPF_PROG_TYPE_UNSPEC)
return -EINVAL;
prog = bpf_prog_get_type(attr->link_create.prog_fd, ptype);
prog = bpf_prog_get(attr->link_create.prog_fd);
if (IS_ERR(prog))
return PTR_ERR(prog);
ret = bpf_prog_attach_check_attach_type(prog,
attr->link_create.attach_type);
if (ret)
goto err_out;
goto out;
if (prog->type == BPF_PROG_TYPE_EXT) {
ret = tracing_bpf_link_attach(attr, prog);
goto out;
}
ptype = attach_type_to_prog_type(attr->link_create.attach_type);
if (ptype == BPF_PROG_TYPE_UNSPEC || ptype != prog->type) {
ret = -EINVAL;
goto out;
}
switch (ptype) {
case BPF_PROG_TYPE_CGROUP_SKB:
......@@ -3950,7 +4082,7 @@ static int link_create(union bpf_attr *attr)
ret = -EINVAL;
}
err_out:
out:
if (ret < 0)
bpf_prog_put(prog);
return ret;
......
......@@ -261,14 +261,12 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)
}
}
int bpf_trampoline_link_prog(struct bpf_prog *prog)
int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
{
enum bpf_tramp_prog_type kind;
struct bpf_trampoline *tr;
int err = 0;
int cnt;
tr = prog->aux->trampoline;
kind = bpf_attach_type_to_tramp(prog);
mutex_lock(&tr->mutex);
if (tr->extension_prog) {
......@@ -301,7 +299,7 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog)
}
hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]);
tr->progs_cnt[kind]++;
err = bpf_trampoline_update(prog->aux->trampoline);
err = bpf_trampoline_update(tr);
if (err) {
hlist_del(&prog->aux->tramp_hlist);
tr->progs_cnt[kind]--;
......@@ -312,13 +310,11 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog)
}
/* bpf_trampoline_unlink_prog() should never fail. */
int bpf_trampoline_unlink_prog(struct bpf_prog *prog)
int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
{
enum bpf_tramp_prog_type kind;
struct bpf_trampoline *tr;
int err;
tr = prog->aux->trampoline;
kind = bpf_attach_type_to_tramp(prog);
mutex_lock(&tr->mutex);
if (kind == BPF_TRAMP_REPLACE) {
......@@ -330,7 +326,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog)
}
hlist_del(&prog->aux->tramp_hlist);
tr->progs_cnt[kind]--;
err = bpf_trampoline_update(prog->aux->trampoline);
err = bpf_trampoline_update(tr);
out:
mutex_unlock(&tr->mutex);
return err;
......
......@@ -2648,8 +2648,7 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno,
static enum bpf_prog_type resolve_prog_type(struct bpf_prog *prog)
{
return prog->aux->linked_prog ? prog->aux->linked_prog->type
: prog->type;
return prog->aux->dst_prog ? prog->aux->dst_prog->type : prog->type;
}
static bool may_access_direct_pkt_data(struct bpf_verifier_env *env,
......@@ -11405,6 +11404,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
if (!btf_type_is_func_proto(t))
return -EINVAL;
if ((prog->aux->saved_dst_prog_type || prog->aux->saved_dst_attach_type) &&
(!tgt_prog || prog->aux->saved_dst_prog_type != tgt_prog->type ||
prog->aux->saved_dst_attach_type != tgt_prog->expected_attach_type))
return -EINVAL;
if (tgt_prog && conservative)
t = NULL;
......@@ -11475,7 +11479,7 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
static int check_attach_btf_id(struct bpf_verifier_env *env)
{
struct bpf_prog *prog = env->prog;
struct bpf_prog *tgt_prog = prog->aux->linked_prog;
struct bpf_prog *tgt_prog = prog->aux->dst_prog;
struct bpf_attach_target_info tgt_info = {};
u32 btf_id = prog->aux->attach_btf_id;
struct bpf_trampoline *tr;
......@@ -11501,6 +11505,10 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return ret;
if (tgt_prog && prog->type == BPF_PROG_TYPE_EXT) {
/* to make freplace equivalent to their targets, they need to
* inherit env->ops and expected_attach_type for the rest of the
* verification
*/
env->ops = bpf_verifier_ops[tgt_prog->type];
prog->expected_attach_type = tgt_prog->expected_attach_type;
}
......@@ -11509,6 +11517,11 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
prog->aux->attach_func_proto = tgt_info.tgt_type;
prog->aux->attach_func_name = tgt_info.tgt_name;
if (tgt_prog) {
prog->aux->saved_dst_prog_type = tgt_prog->type;
prog->aux->saved_dst_attach_type = tgt_prog->expected_attach_type;
}
if (prog->expected_attach_type == BPF_TRACE_RAW_TP) {
prog->aux->attach_btf_trace = true;
return 0;
......@@ -11529,7 +11542,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
if (!tr)
return -ENOMEM;
prog->aux->trampoline = tr;
prog->aux->dst_trampoline = tr;
return 0;
}
......
......@@ -639,8 +639,13 @@ union bpf_attr {
};
__u32 attach_type; /* attach type */
__u32 flags; /* extra flags */
union {
__u32 target_btf_id; /* btf_id of target to attach to */
struct {
__aligned_u64 iter_info; /* extra bpf_iter_link_info */
__u32 iter_info_len; /* iter_info length */
};
};
} link_create;
struct { /* struct used by BPF_LINK_UPDATE command */
......
......@@ -586,19 +586,31 @@ int bpf_link_create(int prog_fd, int target_fd,
enum bpf_attach_type attach_type,
const struct bpf_link_create_opts *opts)
{
__u32 target_btf_id, iter_info_len;
union bpf_attr attr;
if (!OPTS_VALID(opts, bpf_link_create_opts))
return -EINVAL;
iter_info_len = OPTS_GET(opts, iter_info_len, 0);
target_btf_id = OPTS_GET(opts, target_btf_id, 0);
if (iter_info_len && target_btf_id)
return -EINVAL;
memset(&attr, 0, sizeof(attr));
attr.link_create.prog_fd = prog_fd;
attr.link_create.target_fd = target_fd;
attr.link_create.attach_type = attach_type;
attr.link_create.flags = OPTS_GET(opts, flags, 0);
if (iter_info_len) {
attr.link_create.iter_info =
ptr_to_u64(OPTS_GET(opts, iter_info, (void *)0));
attr.link_create.iter_info_len = OPTS_GET(opts, iter_info_len, 0);
attr.link_create.iter_info_len = iter_info_len;
} else if (target_btf_id) {
attr.link_create.target_btf_id = target_btf_id;
}
return sys_bpf(BPF_LINK_CREATE, &attr, sizeof(attr));
}
......
......@@ -174,8 +174,9 @@ struct bpf_link_create_opts {
__u32 flags;
union bpf_iter_link_info *iter_info;
__u32 iter_info_len;
__u32 target_btf_id;
};
#define bpf_link_create_opts__last_field iter_info_len
#define bpf_link_create_opts__last_field target_btf_id
LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
enum bpf_attach_type attach_type,
......
......@@ -9390,9 +9390,11 @@ static struct bpf_link *attach_iter(const struct bpf_sec_def *sec,
}
static struct bpf_link *
bpf_program__attach_fd(struct bpf_program *prog, int target_fd,
bpf_program__attach_fd(struct bpf_program *prog, int target_fd, int btf_id,
const char *target_name)
{
DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts,
.target_btf_id = btf_id);
enum bpf_attach_type attach_type;
char errmsg[STRERR_BUFSIZE];
struct bpf_link *link;
......@@ -9410,7 +9412,7 @@ bpf_program__attach_fd(struct bpf_program *prog, int target_fd,
link->detach = &bpf_link__detach_fd;
attach_type = bpf_program__get_expected_attach_type(prog);
link_fd = bpf_link_create(prog_fd, target_fd, attach_type, NULL);
link_fd = bpf_link_create(prog_fd, target_fd, attach_type, &opts);
if (link_fd < 0) {
link_fd = -errno;
free(link);
......@@ -9426,19 +9428,51 @@ bpf_program__attach_fd(struct bpf_program *prog, int target_fd,
struct bpf_link *
bpf_program__attach_cgroup(struct bpf_program *prog, int cgroup_fd)
{
return bpf_program__attach_fd(prog, cgroup_fd, "cgroup");
return bpf_program__attach_fd(prog, cgroup_fd, 0, "cgroup");
}
struct bpf_link *
bpf_program__attach_netns(struct bpf_program *prog, int netns_fd)
{
return bpf_program__attach_fd(prog, netns_fd, "netns");
return bpf_program__attach_fd(prog, netns_fd, 0, "netns");
}
struct bpf_link *bpf_program__attach_xdp(struct bpf_program *prog, int ifindex)
{
/* target_fd/target_ifindex use the same field in LINK_CREATE */
return bpf_program__attach_fd(prog, ifindex, "xdp");
return bpf_program__attach_fd(prog, ifindex, 0, "xdp");
}
struct bpf_link *bpf_program__attach_freplace(struct bpf_program *prog,
int target_fd,
const char *attach_func_name)
{
int btf_id;
if (!!target_fd != !!attach_func_name) {
pr_warn("prog '%s': supply none or both of target_fd and attach_func_name\n",
prog->name);
return ERR_PTR(-EINVAL);
}
if (prog->type != BPF_PROG_TYPE_EXT) {
pr_warn("prog '%s': only BPF_PROG_TYPE_EXT can attach as freplace",
prog->name);
return ERR_PTR(-EINVAL);
}
if (target_fd) {
btf_id = libbpf_find_prog_btf_id(attach_func_name, target_fd);
if (btf_id < 0)
return ERR_PTR(btf_id);
return bpf_program__attach_fd(prog, target_fd, btf_id, "freplace");
} else {
/* no target, so use raw_tracepoint_open for compatibility
* with old kernels
*/
return bpf_program__attach_trace(prog);
}
}
struct bpf_link *
......
......@@ -261,6 +261,9 @@ LIBBPF_API struct bpf_link *
bpf_program__attach_netns(struct bpf_program *prog, int netns_fd);
LIBBPF_API struct bpf_link *
bpf_program__attach_xdp(struct bpf_program *prog, int ifindex);
LIBBPF_API struct bpf_link *
bpf_program__attach_freplace(struct bpf_program *prog,
int target_fd, const char *attach_func_name);
struct bpf_map;
......
......@@ -304,6 +304,7 @@ LIBBPF_0.2.0 {
global:
bpf_prog_bind_map;
bpf_prog_test_run_opts;
bpf_program__attach_freplace;
bpf_program__section_name;
btf__add_array;
btf__add_const;
......
......@@ -2,36 +2,79 @@
/* Copyright (c) 2019 Facebook */
#include <test_progs.h>
#include <network_helpers.h>
#include <bpf/btf.h>
typedef int (*test_cb)(struct bpf_object *obj);
static int check_data_map(struct bpf_object *obj, int prog_cnt, bool reset)
{
struct bpf_map *data_map = NULL, *map;
__u64 *result = NULL;
const int zero = 0;
__u32 duration = 0;
int ret = -1, i;
result = malloc((prog_cnt + 32 /* spare */) * sizeof(__u64));
if (CHECK(!result, "alloc_memory", "failed to alloc memory"))
return -ENOMEM;
bpf_object__for_each_map(map, obj)
if (bpf_map__is_internal(map)) {
data_map = map;
break;
}
if (CHECK(!data_map, "find_data_map", "data map not found\n"))
goto out;
ret = bpf_map_lookup_elem(bpf_map__fd(data_map), &zero, result);
if (CHECK(ret, "get_result",
"failed to get output data: %d\n", ret))
goto out;
for (i = 0; i < prog_cnt; i++) {
if (CHECK(result[i] != 1, "result",
"fexit_bpf2bpf result[%d] failed err %llu\n",
i, result[i]))
goto out;
result[i] = 0;
}
if (reset) {
ret = bpf_map_update_elem(bpf_map__fd(data_map), &zero, result, 0);
if (CHECK(ret, "reset_result", "failed to reset result\n"))
goto out;
}
ret = 0;
out:
free(result);
return ret;
}
static void test_fexit_bpf2bpf_common(const char *obj_file,
const char *target_obj_file,
int prog_cnt,
const char **prog_name,
bool run_prog)
bool run_prog,
test_cb cb)
{
struct bpf_object *obj = NULL, *pkt_obj;
int err, pkt_fd, i;
struct bpf_link **link = NULL;
struct bpf_object *obj = NULL, *tgt_obj;
struct bpf_program **prog = NULL;
struct bpf_link **link = NULL;
__u32 duration = 0, retval;
struct bpf_map *data_map;
const int zero = 0;
__u64 *result = NULL;
int err, tgt_fd, i;
err = bpf_prog_load(target_obj_file, BPF_PROG_TYPE_UNSPEC,
&pkt_obj, &pkt_fd);
&tgt_obj, &tgt_fd);
if (CHECK(err, "tgt_prog_load", "file %s err %d errno %d\n",
target_obj_file, err, errno))
return;
DECLARE_LIBBPF_OPTS(bpf_object_open_opts, opts,
.attach_prog_fd = pkt_fd,
.attach_prog_fd = tgt_fd,
);
link = calloc(sizeof(struct bpf_link *), prog_cnt);
prog = calloc(sizeof(struct bpf_program *), prog_cnt);
result = malloc((prog_cnt + 32 /* spare */) * sizeof(__u64));
if (CHECK(!link || !prog || !result, "alloc_memory",
"failed to alloc memory"))
if (CHECK(!link || !prog, "alloc_memory", "failed to alloc memory"))
goto close_prog;
obj = bpf_object__open_file(obj_file, &opts);
......@@ -53,27 +96,22 @@ static void test_fexit_bpf2bpf_common(const char *obj_file,
goto close_prog;
}
if (!run_prog)
if (cb) {
err = cb(obj);
if (err)
goto close_prog;
}
data_map = bpf_object__find_map_by_name(obj, "fexit_bp.bss");
if (CHECK(!data_map, "find_data_map", "data map not found\n"))
if (!run_prog)
goto close_prog;
err = bpf_prog_test_run(pkt_fd, 1, &pkt_v6, sizeof(pkt_v6),
err = bpf_prog_test_run(tgt_fd, 1, &pkt_v6, sizeof(pkt_v6),
NULL, NULL, &retval, &duration);
CHECK(err || retval, "ipv6",
"err %d errno %d retval %d duration %d\n",
err, errno, retval, duration);
err = bpf_map_lookup_elem(bpf_map__fd(data_map), &zero, result);
if (CHECK(err, "get_result",
"failed to get output data: %d\n", err))
goto close_prog;
for (i = 0; i < prog_cnt; i++)
if (CHECK(result[i] != 1, "result", "fexit_bpf2bpf failed err %llu\n",
result[i]))
if (check_data_map(obj, prog_cnt, false))
goto close_prog;
close_prog:
......@@ -82,10 +120,9 @@ static void test_fexit_bpf2bpf_common(const char *obj_file,
bpf_link__destroy(link[i]);
if (!IS_ERR_OR_NULL(obj))
bpf_object__close(obj);
bpf_object__close(pkt_obj);
bpf_object__close(tgt_obj);
free(link);
free(prog);
free(result);
}
static void test_target_no_callees(void)
......@@ -96,7 +133,7 @@ static void test_target_no_callees(void)
test_fexit_bpf2bpf_common("./fexit_bpf2bpf_simple.o",
"./test_pkt_md_access.o",
ARRAY_SIZE(prog_name),
prog_name, true);
prog_name, true, NULL);
}
static void test_target_yes_callees(void)
......@@ -110,7 +147,7 @@ static void test_target_yes_callees(void)
test_fexit_bpf2bpf_common("./fexit_bpf2bpf.o",
"./test_pkt_access.o",
ARRAY_SIZE(prog_name),
prog_name, true);
prog_name, true, NULL);
}
static void test_func_replace(void)
......@@ -128,7 +165,7 @@ static void test_func_replace(void)
test_fexit_bpf2bpf_common("./fexit_bpf2bpf.o",
"./test_pkt_access.o",
ARRAY_SIZE(prog_name),
prog_name, true);
prog_name, true, NULL);
}
static void test_func_replace_verify(void)
......@@ -139,9 +176,116 @@ static void test_func_replace_verify(void)
test_fexit_bpf2bpf_common("./freplace_connect4.o",
"./connect4_prog.o",
ARRAY_SIZE(prog_name),
prog_name, false);
prog_name, false, NULL);
}
static int test_second_attach(struct bpf_object *obj)
{
const char *prog_name = "freplace/get_constant";
const char *tgt_name = prog_name + 9; /* cut off freplace/ */
const char *tgt_obj_file = "./test_pkt_access.o";
struct bpf_program *prog = NULL;
struct bpf_object *tgt_obj;
__u32 duration = 0, retval;
struct bpf_link *link;
int err = 0, tgt_fd;
prog = bpf_object__find_program_by_title(obj, prog_name);
if (CHECK(!prog, "find_prog", "prog %s not found\n", prog_name))
return -ENOENT;
err = bpf_prog_load(tgt_obj_file, BPF_PROG_TYPE_UNSPEC,
&tgt_obj, &tgt_fd);
if (CHECK(err, "second_prog_load", "file %s err %d errno %d\n",
tgt_obj_file, err, errno))
return err;
link = bpf_program__attach_freplace(prog, tgt_fd, tgt_name);
if (CHECK(IS_ERR(link), "second_link", "failed to attach second link prog_fd %d tgt_fd %d\n", bpf_program__fd(prog), tgt_fd))
goto out;
err = bpf_prog_test_run(tgt_fd, 1, &pkt_v6, sizeof(pkt_v6),
NULL, NULL, &retval, &duration);
if (CHECK(err || retval, "ipv6",
"err %d errno %d retval %d duration %d\n",
err, errno, retval, duration))
goto out;
err = check_data_map(obj, 1, true);
if (err)
goto out;
out:
bpf_link__destroy(link);
bpf_object__close(tgt_obj);
return err;
}
static void test_func_replace_multi(void)
{
const char *prog_name[] = {
"freplace/get_constant",
};
test_fexit_bpf2bpf_common("./freplace_get_constant.o",
"./test_pkt_access.o",
ARRAY_SIZE(prog_name),
prog_name, true, test_second_attach);
}
static void test_fmod_ret_freplace(void)
{
struct bpf_object *freplace_obj = NULL, *pkt_obj, *fmod_obj = NULL;
const char *freplace_name = "./freplace_get_constant.o";
const char *fmod_ret_name = "./fmod_ret_freplace.o";
DECLARE_LIBBPF_OPTS(bpf_object_open_opts, opts);
const char *tgt_name = "./test_pkt_access.o";
struct bpf_link *freplace_link = NULL;
struct bpf_program *prog;
__u32 duration = 0;
int err, pkt_fd;
err = bpf_prog_load(tgt_name, BPF_PROG_TYPE_UNSPEC,
&pkt_obj, &pkt_fd);
/* the target prog should load fine */
if (CHECK(err, "tgt_prog_load", "file %s err %d errno %d\n",
tgt_name, err, errno))
return;
opts.attach_prog_fd = pkt_fd;
freplace_obj = bpf_object__open_file(freplace_name, &opts);
if (CHECK(IS_ERR_OR_NULL(freplace_obj), "freplace_obj_open",
"failed to open %s: %ld\n", freplace_name,
PTR_ERR(freplace_obj)))
goto out;
err = bpf_object__load(freplace_obj);
if (CHECK(err, "freplace_obj_load", "err %d\n", err))
goto out;
prog = bpf_program__next(NULL, freplace_obj);
freplace_link = bpf_program__attach_trace(prog);
if (CHECK(IS_ERR(freplace_link), "freplace_attach_trace", "failed to link\n"))
goto out;
opts.attach_prog_fd = bpf_program__fd(prog);
fmod_obj = bpf_object__open_file(fmod_ret_name, &opts);
if (CHECK(IS_ERR_OR_NULL(fmod_obj), "fmod_obj_open",
"failed to open %s: %ld\n", fmod_ret_name,
PTR_ERR(fmod_obj)))
goto out;
err = bpf_object__load(fmod_obj);
if (CHECK(!err, "fmod_obj_load", "loading fmod_ret should fail\n"))
goto out;
out:
bpf_link__destroy(freplace_link);
bpf_object__close(freplace_obj);
bpf_object__close(fmod_obj);
bpf_object__close(pkt_obj);
}
static void test_func_sockmap_update(void)
{
const char *prog_name[] = {
......@@ -150,7 +294,7 @@ static void test_func_sockmap_update(void)
test_fexit_bpf2bpf_common("./freplace_cls_redirect.o",
"./test_cls_redirect.o",
ARRAY_SIZE(prog_name),
prog_name, false);
prog_name, false, NULL);
}
static void test_obj_load_failure_common(const char *obj_file,
......@@ -222,4 +366,8 @@ void test_fexit_bpf2bpf(void)
test_func_replace_return_code();
if (test__start_subtest("func_map_prog_compatibility"))
test_func_map_prog_compatibility();
if (test__start_subtest("func_replace_multi"))
test_func_replace_multi();
if (test__start_subtest("fmod_ret_freplace"))
test_fmod_ret_freplace();
}
// SPDX-License-Identifier: GPL-2.0
#define _GNU_SOURCE
#include <test_progs.h>
#include <network_helpers.h>
#include <sys/stat.h>
#include <linux/sched.h>
#include <sys/syscall.h>
#include "test_pkt_md_access.skel.h"
#include "test_trace_ext.skel.h"
#include "test_trace_ext_tracing.skel.h"
static __u32 duration;
void test_trace_ext(void)
{
struct test_pkt_md_access *skel_pkt = NULL;
struct test_trace_ext_tracing *skel_trace = NULL;
struct test_trace_ext_tracing__bss *bss_trace;
struct test_trace_ext *skel_ext = NULL;
struct test_trace_ext__bss *bss_ext;
int err, pkt_fd, ext_fd;
struct bpf_program *prog;
char buf[100];
__u32 retval;
__u64 len;
/* open/load/attach test_pkt_md_access */
skel_pkt = test_pkt_md_access__open_and_load();
if (CHECK(!skel_pkt, "setup", "classifier/test_pkt_md_access open failed\n"))
goto cleanup;
err = test_pkt_md_access__attach(skel_pkt);
if (CHECK(err, "setup", "classifier/test_pkt_md_access attach failed: %d\n", err))
goto cleanup;
prog = skel_pkt->progs.test_pkt_md_access;
pkt_fd = bpf_program__fd(prog);
/* open extension */
skel_ext = test_trace_ext__open();
if (CHECK(!skel_ext, "setup", "freplace/test_pkt_md_access open failed\n"))
goto cleanup;
/* set extension's attach target - test_pkt_md_access */
prog = skel_ext->progs.test_pkt_md_access_new;
bpf_program__set_attach_target(prog, pkt_fd, "test_pkt_md_access");
/* load/attach extension */
err = test_trace_ext__load(skel_ext);
if (CHECK(err, "setup", "freplace/test_pkt_md_access load failed\n")) {
libbpf_strerror(err, buf, sizeof(buf));
fprintf(stderr, "%s\n", buf);
goto cleanup;
}
err = test_trace_ext__attach(skel_ext);
if (CHECK(err, "setup", "freplace/test_pkt_md_access attach failed: %d\n", err))
goto cleanup;
prog = skel_ext->progs.test_pkt_md_access_new;
ext_fd = bpf_program__fd(prog);
/* open tracing */
skel_trace = test_trace_ext_tracing__open();
if (CHECK(!skel_trace, "setup", "tracing/test_pkt_md_access_new open failed\n"))
goto cleanup;
/* set tracing's attach target - fentry */
prog = skel_trace->progs.fentry;
bpf_program__set_attach_target(prog, ext_fd, "test_pkt_md_access_new");
/* set tracing's attach target - fexit */
prog = skel_trace->progs.fexit;
bpf_program__set_attach_target(prog, ext_fd, "test_pkt_md_access_new");
/* load/attach tracing */
err = test_trace_ext_tracing__load(skel_trace);
if (CHECK(err, "setup", "tracing/test_pkt_md_access_new load failed\n")) {
libbpf_strerror(err, buf, sizeof(buf));
fprintf(stderr, "%s\n", buf);
goto cleanup;
}
err = test_trace_ext_tracing__attach(skel_trace);
if (CHECK(err, "setup", "tracing/test_pkt_md_access_new attach failed: %d\n", err))
goto cleanup;
/* trigger the test */
err = bpf_prog_test_run(pkt_fd, 1, &pkt_v4, sizeof(pkt_v4),
NULL, NULL, &retval, &duration);
CHECK(err || retval, "run", "err %d errno %d retval %d\n", err, errno, retval);
bss_ext = skel_ext->bss;
bss_trace = skel_trace->bss;
len = bss_ext->ext_called;
CHECK(bss_ext->ext_called == 0,
"check", "failed to trigger freplace/test_pkt_md_access\n");
CHECK(bss_trace->fentry_called != len,
"check", "failed to trigger fentry/test_pkt_md_access_new\n");
CHECK(bss_trace->fexit_called != len,
"check", "failed to trigger fexit/test_pkt_md_access_new\n");
cleanup:
test_trace_ext_tracing__destroy(skel_trace);
test_trace_ext__destroy(skel_ext);
test_pkt_md_access__destroy(skel_pkt);
}
// SPDX-License-Identifier: GPL-2.0
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
volatile __u64 test_fmod_ret = 0;
SEC("fmod_ret/security_new_get_constant")
int BPF_PROG(fmod_ret_test, long val, int ret)
{
test_fmod_ret = 1;
return 120;
}
char _license[] SEC("license") = "GPL";
// SPDX-License-Identifier: GPL-2.0
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>
volatile __u64 test_get_constant = 0;
SEC("freplace/get_constant")
int security_new_get_constant(long val)
{
if (val != 123)
return 0;
test_get_constant = 1;
return test_get_constant; /* original get_constant() returns val - 122 */
}
char _license[] SEC("license") = "GPL";
// SPDX-License-Identifier: GPL-2.0
// Copyright (c) 2019 Facebook
#include <linux/bpf.h>
#include <stdbool.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>
#include <bpf/bpf_tracing.h>
__u64 ext_called = 0;
SEC("freplace/test_pkt_md_access")
int test_pkt_md_access_new(struct __sk_buff *skb)
{
ext_called = skb->len;
return 0;
}
char _license[] SEC("license") = "GPL";
// SPDX-License-Identifier: GPL-2.0
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
__u64 fentry_called = 0;
SEC("fentry/test_pkt_md_access_new")
int BPF_PROG(fentry, struct sk_buff *skb)
{
fentry_called = skb->len;
return 0;
}
__u64 fexit_called = 0;
SEC("fexit/test_pkt_md_access_new")
int BPF_PROG(fexit, struct sk_buff *skb)
{
fexit_called = skb->len;
return 0;
}
char _license[] SEC("license") = "GPL";
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment