Intended for search (Ctrl-F) and reference. For tutorials, start with [tutorial.md](tutorial.md).
This guide is incomplete. If something feels missing, check the bcc and kernel source. And if you confirm we're missing something, please send a pull request to fix it, and help out everyone.
This section describes the C part of a bcc program.
## Events & Arguments
### 1. kprobes
Syntax: kprobe__*kernel_function_name*
```kprobe__``` is a special prefix that creates a kprobe (dynamic tracing of a kernel function call) for the kernel function name provided as the remainder. You can also use kprobes by declaring a normal C function, then using the Python ```BPF.attach_kprobe()``` (covered later) to associate it with a kernel function.
Arguments are specified on the function declaration: kprobe__*kernel_function_name*(struct pt_regs *ctx [, *argument1* ...])
For example:
```C
int kprobe__tcp_v4_connect(struct pt_regs *ctx, struct sock *sk)
[...]
}
```
This instruments the tcp_v4_connect() kernel function using a kprobe, with the following arguments:
- ```struct pt_regs *ctx```: Registers and BPF context.
- ```struct sock *sk```: First argument to tcp_v4_connect().
The first argument is always ```struct pt_regs *```, the remainder are the arguments to the function (they don't need to be specified, if you don't intend to use them).
<!--- I can't add search links here, since github currently cannot handle partial-word searches needed for "kprobe__" --->
### 2. kretprobes
Syntax: kretprobe__*kernel_function_name*
```kretprobe__``` is a special prefix that creates a kretprobe (dynamic tracing of a kernel function return) for the kernel function name provided as the remainder. You can also use kretprobes by declaring a normal C function, then using the Python ```BPF.attach_kretprobe()``` (covered later) to associate it with a kernel function.
Return value is available as ```PT_REGS_RC(ctx)```, given a function declaration of: kretprobe__*kernel_function_name*(struct pt_regs *ctx)
For example:
```C
int kretprobe__tcp_v4_connect(struct pt_regs *ctx)
{
int ret = PT_REGS_RC(ctx);
[...]
}
```
This instruments the return of the tcp_v4_connect() kernel function using a kretprobe, and stores the return value in ```ret```.
This is a macro that instruments the tracepoint defined by *category*:*event*.
Arguments are available in an ```args``` struct, which are the tracepoint arguments. One way to list these is to cat the relevant format file under /sys/kernel/debug/tracing/events/*category*/*event*/format.
For example:
```C
TRACEPOINT_PROBE(random, urandom_read) {
// args is from /sys/kernel/debug/tracing/events/random/urandom_read/format
bpf_trace_printk("%d\\n", args->got_bits);
return 0;
}
```
This instruments the random:urandom_read tracepoint, and prints the tracepoint argument ```got_bits```.
These are instrumented by declaring a normal funciton in C, then associating it as a uprobe probe in Python via ```BPF.attach_uprobe()``` (covered later).
Arguments can be examined using ```PT_REGS_PARM``` macros.
These are instrumented by declaring a normal funciton in C, then associating it as a uretprobe probe in Python via ```BPF.attach_uretprobe()``` (covered later).
Return value is available as ```PT_REGS_RC(ctx)```, given a function declaration of: *function_name*(struct pt_regs *ctx)
For example:
```C
BPF_HISTOGRAM(dist);
int count(struct pt_regs *ctx) {
dist.increment(PT_REGS_RC(ctx));
return 0;
}
```
This increments the bucket in the ```dist``` histogram that is indexed by the return value.
Syntax: ```int bpf_probe_read(void *dst, int size, void *src)```
Return: 0 on success
This copies a memory location to the BPF stack, so that BPF can later operate on it. For safety, all memory reads must pass through bpf_probe_read(). This happens automatically in some cases, such as dereferencing kernel varibles, as bcc will rewrite the BPF program to include the necessary bpf_probe_reads().
Returns the process ID in the lower 32 bits (kernel's view of the PID, which in user space is usually presented as the thread ID), and the thread group ID in the upper 32 bits (what user space often thinks of as the PID). By directly setting this to a u32, we discard the upper 32 bits.
Syntax: ```bpf_get_current_comm(char *buf, int size_of_buf)```
Return: 0 on success
Populates the first argument address with the current process name. It should be a pointer to a char array of at least size TASK_COMM_LEN, which is defined in linux/sched.h. For example:
Syntax: ```int bpf_trace_printk(const char *fmt, int fmt_size, ...)```
Return: 0 on success
A simple kernel facility for printf() to the common trace_pipe (/sys/kernel/debug/tracing/trace_pipe). This is ok for some quick examples, but has limitations: 3 args max, 1 %s only, and trace_pipe is globally shared, so concurrent programs will have clashing output. A better interface is via BPF_PERF_OUTPUT().
Creates a BPF table for pushing out custom event data to user space via a perf ring buffer. This is the preferred method for pushing per-event data to user space.
A method of a BPF_PERF_OUTPUT table, for submitting custom event data to user space. See the BPF_PERF_OUTPUT entry. (This ultimately calls bpf_perf_event_output().)
This creates a hash named ```start``` where the key is a ```struct request *```, and the value defaults to u64. This hash is used by the disksnoop.py example for saving timestamps for each I/O request, where the key is the pointer to struct request, and the value is the timestamp.
Lookup the key in the map, and return a pointer to its value if it exists, else initialize the key's value to the second argument. This is often used to initialize values to zero.
Instruments the kernel function ```event()``` using kernel dynamic tracing of the function entry, and attaches our C defined function ```name()``` to be called when the kernel function is called.
Instruments the return of the kernel function ```event()``` using kernel dynamic tracing of the function return, and attaches our C defined function ```name()``` to be called when the kernel function returns.
Instruments the kernel tracepoint described by ```tracepoint```, and when hit, runs the BPF function ```name()```.
This is an explicit way to instrument tracepoints. The ```TRACEPOINT_PROBE``` syntax, covered in the earlier tracepoints section, is an alternate method with the advantage of auto-declaring an ```args``` struct containing the tracepoint arguments. With ```attach_tracepoint()```, the tracepoint arguments need to be declared in the BPF program.
For example:
```Python
# define BPF program
bpf_text = """
#include <uapi/linux/ptrace.h>
struct urandom_read_args {
// from /sys/kernel/debug/tracing/events/random/urandom_read/format
Instruments the user-level function ```symbol()``` from either the library or binary named by ```location``` using user-level dynamic tracing of the function entry, and attach our C defined function ```name()``` to be called whenever the user-level function is called.
Libraries can be given in the name argument without the lib prefix, or with the full path (/usr/lib/...). Binaries can be given only with the full path (/bin/sh).
This will instrument ```strlen()``` function from libc, and call our BPF function ```count()``` when it is called. Note how the "lib" in "libc" is not necessary to specify.
Instruments the return of the user-level function ```symbol()``` from either the library or binary named by ```location``` using user-level dynamic tracing of the function return, and attach our C defined function ```name()``` to be called whenever the user-level function returns.
This method continually reads the globally shared /sys/kernel/debug/tracing/trace_pipe file and prints its contents. This file can be written to via BPF and the bpf_trace_printk() function, however, that method has limitations, including a lack of concurrent tracing support. The BPF_PERF_OUTPUT mechanism, covered earlier, is preferred.
Arguments:
- ```fmt```: optional, and can contain a field formatting string. It defaults to ```None```.
This method reads one line from the globally shared /sys/kernel/debug/tracing/trace_pipe file and returns it as fields. This file can be written to via BPF and the bpf_trace_printk() function, however, that method has limitations, including a lack of concurrent tracing support. The BPF_PERF_OUTPUT mechanism, covered earlier, is preferred.
Arguments:
- ```nonblocking```: optional, defaults to ```False```. When set to ```True```, the program will not block waiting for input.
- per-event: using PERF_EVENT_OUTPUT, open_perf_buffer(), and kprobe_poll().
- map summary: using items(), or print_log2_hist(), covered in the Maps section.
### 1. kprobe_poll()
Syntax: ```BPF.kprobe_poll()```
This polls from the ring buffers for all of the open kprobes, calling the callback function that was given in the BPF constructor for each entry, usually via ```open_perf_buffer()```.
Maps are BPF data stores, and are used in bcc to implement a table, and then higher level objects on top of tables, including hashes and histograms.
### 1. get_table()
Syntax: ```BPF.get_table(name)```
Returns a table object. This is no longer used, as tables can now be read as items from BPF. Eg: ```BPF[name]```.
Examples:
```Python
counts = b.get_table("counts")
counts = b["counts"]
```
These are equivalent.
### 2. open_perf_buffer()
Syntax: ```table.open_perf_buffers(callback)```
This operates on a table as defined in BPF as BPF_PERF_OUTPUT(), and associates the callback Python function ```callback``` to be called when data is available in the perf ring buffer. This is part of the recommended mechanism for transferring per-event data from kernel to user space.
Example:
```Python
# process event
def print_event(cpu, data, size):
event = ct.cast(data, ct.POINTER(Data)).contents
[...]
# loop with callback to print_event
b["events"].open_perf_buffer(print_event)
while 1:
b.kprobe_poll()
```
Note that the data structure transferred will need to be declared in C in the BPF program, and in Python. For example:
```C
// define output data structure in C
struct data_t {
u32 pid;
u64 ts;
char comm[TASK_COMM_LEN];
};
```
```Python
# define output data structure in Python
TASK_COMM_LEN = 16 # linux/sched.h
class Data(ct.Structure):
_fields_ = [("pid", ct.c_ulonglong),
("ts", ct.c_ulonglong),
("comm", ct.c_char * TASK_COMM_LEN)]
```
Perhaps in a future bcc version, the Python data structure will be automatically generated from the C declaration.
Some helper methods provided by bcc. Note that since we're in Python, we can import any Python library and their methods, including, for example, the libraries: argparse, collections, ctypes, datetime, re, socket, struct, subprocess, sys, and time.
### 1. ksym()
Syntax: ```BPF.ksym(addr)```
Translate a kernel memory address into a kernel function name, which is returned.
Returns the number of open k[ret]probes. Can be useful for scenarios where event_re is used while attaching and detaching probes. Excludes perf_events readers.
See the "Understanding eBPF verifier messages" section in the kernel source under Documentation/networking/filter.txt.
## 1. Invalid mem access
This can be due to trying to read memory directly, instead of operating on memory on the BPF stack. All memory reads must be passed via bpf_probe_read() to copy memory into the BPF stack, which can be automatic by the bcc rewriter in some cases of simple dereferencing. bpf_probe_read() does all the required checks.
Example:
```
bpf: Permission denied
0: (bf) r6 = r1
1: (79) r7 = *(u64 *)(r6 +80)
2: (85) call 14
3: (bf) r8 = r0
[...]
23: (69) r1 = *(u16 *)(r7 +16)
R7 invalid mem access 'inv'
Traceback (most recent call last):
File "./tcpaccept", line 179, in <module>
b = BPF(text=bpf_text)
File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 172, in __init__
self._trace_autoload()
File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 612, in _trace_autoload
fn = self.load_func(func_name, BPF.KPROBE)
File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 212, in load_func
raise Exception("Failed to load BPF program %s" % func_name)
Exception: Failed to load BPF program kretprobe__inet_csk_accept
This tutorial is about developing [bcc](https://github.com/iovisor/bcc) tools and programs using the Python interface. There are two parts: observability then networking. Snippits are taken from various programs in bcc: see their files for licences.
There is also a lua interface for bcc, and a tutorial for end-users of tools: [tutorial.md](tutorial.md).
Also see the bcc developer's [reference_guide.md](reference_guide.md), and a tutorial for end-users of tools: [tutorial.md](tutorial.md). There is also a lua interface for bcc.
