Skip to content

L
linux
Commit 1f198e22 authored by Changbin Du's avatar Changbin Du Committed by Jonathan Corbet
Browse files
Options

trace doc: convert trace/ftrace.txt to rst format 

This converts the plain text documentation to reStructuredText format and
add it into Sphinx TOC tree. No essential content change.

Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: default avatarChangbin Du <changbin.du@intel.com>
Signed-off-by: default avatarJonathan Corbet <corbet@lwn.net>
parent 8fa4e720
Show whitespace changes
Inline Side-by-side
Showing with 3333 additions and 0 deletions
Documentation/trace/ftrace.txt → Documentation/trace/ftrace.rst
View file @ 1f198e22
ftrace - Function Tracer
========================
========================
ftrace - Function Tracer
========================
Copyright 2008 Red Hat Inc.
Author: Steven Rostedt <srostedt@redhat.com>
License: The GNU Free Documentation License, Version 1.2
:Author: Steven Rostedt <srostedt@redhat.com>
:License: The GNU Free Documentation License, Version 1.2
(dual licensed under the GPL v2)
Original Reviewers: Elias Oltmanns, Randy Dunlap, Andrew Morton,
:Original Reviewers: Elias Oltmanns, Randy Dunlap, Andrew Morton,
John Kacur, and David Teigland.
Written for: 2.6.28-rc2
Updated for: 3.10
Updated for: 4.13 - Copyright 2017 VMware Inc. Steven Rostedt
- Written for: 2.6.28-rc2
- Updated for: 3.10
- Updated for: 4.13 - Copyright 2017 VMware Inc. Steven Rostedt
- Converted to rst format - Changbin Du <changbin.du@intel.com>
Introduction
------------
......@@ -36,7 +40,7 @@ See events.txt for more information.
Implementation Details
----------------------
See ftrace-design.txt for details for arch porters and such.
See :doc:`ftrace-design` for details for arch porters and such.
The File System
......@@ -47,38 +51,38 @@ well as the files to display output.
When tracefs is configured into the kernel (which selecting any ftrace
option will do) the directory /sys/kernel/tracing will be created. To mount
this directory, you can add to your /etc/fstab file:
this directory, you can add to your /etc/fstab file::
tracefs /sys/kernel/tracing tracefs defaults 0 0
Or you can mount it at run time with:
Or you can mount it at run time with::
mount -t tracefs nodev /sys/kernel/tracing
For quicker access to that directory you may want to make a soft link to
it:
it::
ln -s /sys/kernel/tracing /tracing
*** NOTICE ***
.. attention::
Before 4.1, all ftrace tracing control files were within the debugfs
file system, which is typically located at /sys/kernel/debug/tracing.
For backward compatibility, when mounting the debugfs file system,
the tracefs file system will be automatically mounted at:
Before 4.1, all ftrace tracing control files were within the debugfs
file system, which is typically located at /sys/kernel/debug/tracing.
For backward compatibility, when mounting the debugfs file system,
the tracefs file system will be automatically mounted at:
/sys/kernel/debug/tracing
All files located in the tracefs file system will be located in that
debugfs file system directory as well.
All files located in the tracefs file system will be located in that
debugfs file system directory as well.
*** NOTICE ***
.. attention::
Any selected ftrace option will also create the tracefs file system.
The rest of the document will assume that you are in the ftrace directory
(cd /sys/kernel/tracing) and will only concentrate on the files within that
directory and not distract from the content with the extended
"/sys/kernel/tracing" path name.
Any selected ftrace option will also create the tracefs file system.
The rest of the document will assume that you are in the ftrace directory
(cd /sys/kernel/tracing) and will only concentrate on the files within that
directory and not distract from the content with the extended
"/sys/kernel/tracing" path name.
That's it! (assuming that you have ftrace configured into your kernel)
......@@ -407,41 +411,48 @@ of ftrace. Here is a list of some of the key files:
CPUs. In other words, the local clocks may not be in sync
with local clocks on other CPUs.
Usual clocks for tracing:
Usual clocks for tracing::
# cat trace_clock
[local] global counter x86-tsc
The clock with the square brackets around it is the one
in effect.
The clock with the square brackets around it is the one in effect.
local: Default clock, but may not be in sync across CPUs
local:
Default clock, but may not be in sync across CPUs
global: This clock is in sync with all CPUs but may
global:
This clock is in sync with all CPUs but may
be a bit slower than the local clock.
counter: This is not a clock at all, but literally an atomic
counter:
This is not a clock at all, but literally an atomic
counter. It counts up one by one, but is in sync
with all CPUs. This is useful when you need to
know exactly the order events occurred with respect to
each other on different CPUs.
uptime: This uses the jiffies counter and the time stamp
uptime:
This uses the jiffies counter and the time stamp
is relative to the time since boot up.
perf: This makes ftrace use the same clock that perf uses.
perf:
This makes ftrace use the same clock that perf uses.
Eventually perf will be able to read ftrace buffers
and this will help out in interleaving the data.
x86-tsc: Architectures may define their own clocks. For
x86-tsc:
Architectures may define their own clocks. For
example, x86 uses its own TSC cycle clock here.
ppc-tb: This uses the powerpc timebase register value.
ppc-tb:
This uses the powerpc timebase register value.
This is in sync across CPUs and can also be used
to correlate events across hypervisor/guest if
tb_offset is known.
mono: This uses the fast monotonic clock (CLOCK_MONOTONIC)
mono:
This uses the fast monotonic clock (CLOCK_MONOTONIC)
which is monotonic and is subject to NTP rate adjustments.
mono_raw:
......@@ -449,7 +460,8 @@ of ftrace. Here is a list of some of the key files:
which is montonic but is not subject to any rate adjustments
and ticks at the same rate as the hardware clocksource.
boot: This is the boot clock (CLOCK_BOOTTIME) and is based on the
boot:
This is the boot clock (CLOCK_BOOTTIME) and is based on the
fast monotonic clock, but also accounts for time spent in
suspend. Since the clock access is designed for use in
tracing in the suspend path, some side effects are possible
......@@ -461,7 +473,7 @@ of ftrace. Here is a list of some of the key files:
processing should be able to handle them. See comments in the
ktime_get_boot_fast_ns() function for more information.
To set a clock, simply echo the clock name into this file.
To set a clock, simply echo the clock name into this file::
echo global > trace_clock
......@@ -473,7 +485,7 @@ of ftrace. Here is a list of some of the key files:
It is useful in applications to open this file at the start
of the application and just reference the file descriptor
for the file.
for the file::
void trace_write(const char *fmt, ...)
{
......@@ -491,7 +503,7 @@ of ftrace. Here is a list of some of the key files:
write(trace_fd, buf, n);
}
start:
start::
trace_fd = open("trace_marker", WR_ONLY);
......@@ -597,25 +609,33 @@ of ftrace. Here is a list of some of the key files:
This displays certain stats about the ring buffer:
entries: The number of events that are still in the buffer.
entries:
The number of events that are still in the buffer.
overrun: The number of lost events due to overwriting when
overrun:
The number of lost events due to overwriting when
the buffer was full.
commit overrun: Should always be zero.
commit overrun:
Should always be zero.
This gets set if so many events happened within a nested
event (ring buffer is re-entrant), that it fills the
buffer and starts dropping events.
bytes: Bytes actually read (not overwritten).
bytes:
Bytes actually read (not overwritten).
oldest event ts: The oldest timestamp in the buffer
oldest event ts:
The oldest timestamp in the buffer
now ts: The current timestamp
now ts:
The current timestamp
dropped events: Events lost due to overwrite option being off.
dropped events:
Events lost due to overwrite option being off.
read events: The number of events read.
read events:
The number of events read.
The Tracers
-----------
......@@ -716,20 +736,19 @@ user-land utilities).
Output format:
--------------
Here is an example of the output format of the file "trace"
--------
# tracer: function
#
# entries-in-buffer/entries-written: 140080/250280 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
Here is an example of the output format of the file "trace"::
# tracer: function
#
# entries-in-buffer/entries-written: 140080/250280 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
bash-1977 [000] .... 17284.993652: sys_close <-system_call_fastpath
bash-1977 [000] .... 17284.993653: __close_fd <-sys_close
bash-1977 [000] .... 17284.993653: _raw_spin_lock <-__close_fd
......@@ -740,7 +759,7 @@ Here is an example of the output format of the file "trace"
bash-1977 [000] .... 17284.993657: filp_close <-__close_fd
bash-1977 [000] .... 17284.993657: dnotify_flush <-filp_close
sshd-1974 [003] .... 17284.993658: sys_select <-system_call_fastpath
--------
....
A header is printed with the tracer name that is represented by
the trace. In this case the tracer is "function". Then it shows the
......@@ -761,28 +780,28 @@ Latency trace format
When the latency-format option is enabled or when one of the latency
tracers is set, the trace file gives somewhat more information to see
why a latency happened. Here is a typical trace.
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 259 us, #4/4, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: ps-6143 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: __lock_task_sighand
# => ended at: _raw_spin_unlock_irqrestore
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
why a latency happened. Here is a typical trace::
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 259 us, #4/4, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: ps-6143 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: __lock_task_sighand
# => ended at: _raw_spin_unlock_irqrestore
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
ps-6143 2d... 0us!: trace_hardirqs_off <-__lock_task_sighand
ps-6143 2d..1 259us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore
ps-6143 2d..1 263us+: time_hardirqs_on <-_raw_spin_unlock_irqrestore
......@@ -813,8 +832,8 @@ occurred. (ps pid: 6143).
The start and stop (the functions in which the interrupts were
disabled and enabled respectively) that caused the latencies:
__lock_task_sighand is where the interrupts were disabled.
_raw_spin_unlock_irqrestore is where they were enabled again.
- __lock_task_sighand is where the interrupts were disabled.
- _raw_spin_unlock_irqrestore is where they were enabled again.
The next lines after the header are the trace itself. The header
explains which is which.
......@@ -826,44 +845,47 @@ explains which is which.
CPU#: The CPU which the process was running on.
irqs-off: 'd' interrupts are disabled. '.' otherwise.
Note: If the architecture does not support a way to
.. caution:: If the architecture does not support a way to
read the irq flags variable, an 'X' will always
be printed here.
need-resched:
'N' both TIF_NEED_RESCHED and PREEMPT_NEED_RESCHED is set,
'n' only TIF_NEED_RESCHED is set,
'p' only PREEMPT_NEED_RESCHED is set,
'.' otherwise.
- 'N' both TIF_NEED_RESCHED and PREEMPT_NEED_RESCHED is set,
- 'n' only TIF_NEED_RESCHED is set,
- 'p' only PREEMPT_NEED_RESCHED is set,
- '.' otherwise.
hardirq/softirq:
'Z' - NMI occurred inside a hardirq
'z' - NMI is running
'H' - hard irq occurred inside a softirq.
'h' - hard irq is running
's' - soft irq is running
'.' - normal context.
- 'Z' - NMI occurred inside a hardirq
- 'z' - NMI is running
- 'H' - hard irq occurred inside a softirq.
- 'h' - hard irq is running
- 's' - soft irq is running
- '.' - normal context.
preempt-depth: The level of preempt_disabled
The above is mostly meaningful for kernel developers.
time: When the latency-format option is enabled, the trace file
time:
When the latency-format option is enabled, the trace file
output includes a timestamp relative to the start of the
trace. This differs from the output when latency-format
is disabled, which includes an absolute timestamp.
delay: This is just to help catch your eye a bit better. And
delay:
This is just to help catch your eye a bit better. And
needs to be fixed to be only relative to the same CPU.
The marks are determined by the difference between this
current trace and the next trace.
'$' - greater than 1 second
'@' - greater than 100 milisecond
'*' - greater than 10 milisecond
'#' - greater than 1000 microsecond
'!' - greater than 100 microsecond
'+' - greater than 10 microsecond
' ' - less than or equal to 10 microsecond.
- '$' - greater than 1 second
- '@' - greater than 100 milisecond
- '*' - greater than 10 milisecond
- '#' - greater than 1000 microsecond
- '!' - greater than 100 microsecond
- '+' - greater than 10 microsecond
- ' ' - less than or equal to 10 microsecond.
The rest is the same as the 'trace' file.
......@@ -875,50 +897,52 @@ trace_options
The trace_options file (or the options directory) is used to control
what gets printed in the trace output, or manipulate the tracers.
To see what is available, simply cat the file:
To see what is available, simply cat the file::
cat trace_options
print-parent
nosym-offset
nosym-addr
noverbose
noraw
nohex
nobin
noblock
trace_printk
annotate
nouserstacktrace
nosym-userobj
noprintk-msg-only
context-info
nolatency-format
record-cmd
norecord-tgid
overwrite
nodisable_on_free
irq-info
markers
noevent-fork
function-trace
nofunction-fork
nodisplay-graph
nostacktrace
nobranch
print-parent
nosym-offset
nosym-addr
noverbose
noraw
nohex
nobin
noblock
trace_printk
annotate
nouserstacktrace
nosym-userobj
noprintk-msg-only
context-info
nolatency-format
record-cmd
norecord-tgid
overwrite
nodisable_on_free
irq-info
markers
noevent-fork
function-trace
nofunction-fork
nodisplay-graph
nostacktrace
nobranch
To disable one of the options, echo in the option prepended with
"no".
"no"::
echo noprint-parent > trace_options
To enable an option, leave off the "no".
To enable an option, leave off the "no"::
echo sym-offset > trace_options
Here are the available options:
print-parent - On function traces, display the calling (parent)
print-parent
On function traces, display the calling (parent)
function as well as the function being traced.
::
print-parent:
bash-4000 [01] 1477.606694: simple_strtoul <-kstrtoul
......@@ -927,61 +951,74 @@ Here are the available options:
bash-4000 [01] 1477.606694: simple_strtoul
sym-offset - Display not only the function name, but also the
sym-offset
Display not only the function name, but also the
offset in the function. For example, instead of
seeing just "ktime_get", you will see
"ktime_get+0xb/0x20".
::
sym-offset:
bash-4000 [01] 1477.606694: simple_strtoul+0x6/0xa0
sym-addr - this will also display the function address as well
sym-addr
This will also display the function address as well
as the function name.
::
sym-addr:
bash-4000 [01] 1477.606694: simple_strtoul <c0339346>
verbose - This deals with the trace file when the
verbose
This deals with the trace file when the
latency-format option is enabled.
::
bash 4000 1 0 00000000 00010a95 [58127d26] 1720.415ms \
(+0.000ms): simple_strtoul (kstrtoul)
raw - This will display raw numbers. This option is best for
raw
This will display raw numbers. This option is best for
use with user applications that can translate the raw
numbers better than having it done in the kernel.
hex - Similar to raw, but the numbers will be in a hexadecimal
format.
hex
Similar to raw, but the numbers will be in a hexadecimal format.
bin - This will print out the formats in raw binary.
bin
This will print out the formats in raw binary.
block - When set, reading trace_pipe will not block when polled.
block
When set, reading trace_pipe will not block when polled.
trace_printk - Can disable trace_printk() from writing into the buffer.
trace_printk
Can disable trace_printk() from writing into the buffer.
annotate - It is sometimes confusing when the CPU buffers are full
annotate
It is sometimes confusing when the CPU buffers are full
and one CPU buffer had a lot of events recently, thus
a shorter time frame, were another CPU may have only had
a few events, which lets it have older events. When
the trace is reported, it shows the oldest events first,
and it may look like only one CPU ran (the one with the
oldest events). When the annotate option is set, it will
display when a new CPU buffer started:
display when a new CPU buffer started::
<idle>-0 [001] dNs4 21169.031481: wake_up_idle_cpu <-add_timer_on
<idle>-0 [001] dNs4 21169.031482: _raw_spin_unlock_irqrestore <-add_timer_on
<idle>-0 [001] .Ns4 21169.031484: sub_preempt_count <-_raw_spin_unlock_irqrestore
##### CPU 2 buffer started ####
##### CPU 2 buffer started ####
<idle>-0 [002] .N.1 21169.031484: rcu_idle_exit <-cpu_idle
<idle>-0 [001] .Ns3 21169.031484: _raw_spin_unlock <-clocksource_watchdog
<idle>-0 [001] .Ns3 21169.031485: sub_preempt_count <-_raw_spin_unlock
userstacktrace - This option changes the trace. It records a
userstacktrace
This option changes the trace. It records a
stacktrace of the current user space thread after
each trace event.
sym-userobj - when user stacktrace are enabled, look up which
sym-userobj
when user stacktrace are enabled, look up which
object the address belongs to, and print a
relative address. This is especially useful when
ASLR is on, otherwise you don't get a chance to
......@@ -989,91 +1026,106 @@ Here are the available options:
the app is no longer running
The lookup is performed when you read
trace,trace_pipe. Example:
trace,trace_pipe. Example::
a.out-1623 [000] 40874.465068: /root/a.out[+0x480] <-/root/a.out[+0
x494] <- /root/a.out[+0x4a8] <- /lib/libc-2.7.so[+0x1e1a6]
x494] <- /root/a.out[+0x4a8] <- /lib/libc-2.7.so[+0x1e1a6]
printk-msg-only - When set, trace_printk()s will only show the format
printk-msg-only
When set, trace_printk()s will only show the format
and not their parameters (if trace_bprintk() or
trace_bputs() was used to save the trace_printk()).
context-info - Show only the event data. Hides the comm, PID,
context-info
Show only the event data. Hides the comm, PID,
timestamp, CPU, and other useful data.
latency-format - This option changes the trace output. When it is enabled,
latency-format
This option changes the trace output. When it is enabled,
the trace displays additional information about the
latency, as described in "Latency trace format".
record-cmd - When any event or tracer is enabled, a hook is enabled
record-cmd
When any event or tracer is enabled, a hook is enabled
in the sched_switch trace point to fill comm cache
with mapped pids and comms. But this may cause some
overhead, and if you only care about pids, and not the
name of the task, disabling this option can lower the
impact of tracing. See "saved_cmdlines".
record-tgid - When any event or tracer is enabled, a hook is enabled
record-tgid
When any event or tracer is enabled, a hook is enabled
in the sched_switch trace point to fill the cache of
mapped Thread Group IDs (TGID) mapping to pids. See
"saved_tgids".
overwrite - This controls what happens when the trace buffer is
overwrite
This controls what happens when the trace buffer is
full. If "1" (default), the oldest events are
discarded and overwritten. If "0", then the newest
events are discarded.
(see per_cpu/cpu0/stats for overrun and dropped)
disable_on_free - When the free_buffer is closed, tracing will
disable_on_free
When the free_buffer is closed, tracing will
stop (tracing_on set to 0).
irq-info - Shows the interrupt, preempt count, need resched data.
When disabled, the trace looks like:
irq-info
Shows the interrupt, preempt count, need resched data.
When disabled, the trace looks like::
# tracer: function
#
# entries-in-buffer/entries-written: 144405/9452052 #P:4
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
# tracer: function
#
# entries-in-buffer/entries-written: 144405/9452052 #P:4
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
<idle>-0 [002] 23636.756054: ttwu_do_activate.constprop.89 <-try_to_wake_up
<idle>-0 [002] 23636.756054: activate_task <-ttwu_do_activate.constprop.89
<idle>-0 [002] 23636.756055: enqueue_task <-activate_task
markers - When set, the trace_marker is writable (only by root).
markers
When set, the trace_marker is writable (only by root).
When disabled, the trace_marker will error with EINVAL
on write.
event-fork - When set, tasks with PIDs listed in set_event_pid will have
event-fork
When set, tasks with PIDs listed in set_event_pid will have
the PIDs of their children added to set_event_pid when those
tasks fork. Also, when tasks with PIDs in set_event_pid exit,
their PIDs will be removed from the file.
function-trace - The latency tracers will enable function tracing
function-trace
The latency tracers will enable function tracing
if this option is enabled (default it is). When
it is disabled, the latency tracers do not trace
functions. This keeps the overhead of the tracer down
when performing latency tests.
function-fork - When set, tasks with PIDs listed in set_ftrace_pid will
function-fork
When set, tasks with PIDs listed in set_ftrace_pid will
have the PIDs of their children added to set_ftrace_pid
when those tasks fork. Also, when tasks with PIDs in
set_ftrace_pid exit, their PIDs will be removed from the
file.
display-graph - When set, the latency tracers (irqsoff, wakeup, etc) will
display-graph
When set, the latency tracers (irqsoff, wakeup, etc) will
use function graph tracing instead of function tracing.
stacktrace - When set, a stack trace is recorded after any trace event
stacktrace
When set, a stack trace is recorded after any trace event
is recorded.
branch - Enable branch tracing with the tracer. This enables branch
branch
Enable branch tracing with the tracer. This enables branch
tracer along with the currently set tracer. Enabling this
with the "nop" tracer is the same as just enabling the
"branch" tracer.
Note: Some tracers have their own options. They only appear in this
.. tip:: Some tracers have their own options. They only appear in this
file when the tracer is active. They always appear in the
options directory.
......@@ -1082,7 +1134,8 @@ Here are the per tracer options:
Options for function tracer:
func_stack_trace - When set, a stack trace is recorded after every
func_stack_trace
When set, a stack trace is recorded after every
function that is recorded. NOTE! Limit the functions
that are recorded before enabling this, with
"set_ftrace_filter" otherwise the system performance
......@@ -1094,7 +1147,8 @@ Options for function_graph tracer:
Since the function_graph tracer has a slightly different output
it has its own options to control what is displayed.
funcgraph-overrun - When set, the "overrun" of the graph stack is
funcgraph-overrun
When set, the "overrun" of the graph stack is
displayed after each function traced. The
overrun, is when the stack depth of the calls
is greater than what is reserved for each task.
......@@ -1104,41 +1158,49 @@ Options for function_graph tracer:
The overrun is the number of functions missed
due to exceeding this array.
funcgraph-cpu - When set, the CPU number of the CPU where the trace
funcgraph-cpu
When set, the CPU number of the CPU where the trace
occurred is displayed.
funcgraph-overhead - When set, if the function takes longer than
funcgraph-overhead
When set, if the function takes longer than
A certain amount, then a delay marker is
displayed. See "delay" above, under the
header description.
funcgraph-proc - Unlike other tracers, the process' command line
funcgraph-proc
Unlike other tracers, the process' command line
is not displayed by default, but instead only
when a task is traced in and out during a context
switch. Enabling this options has the command
of each process displayed at every line.
funcgraph-duration - At the end of each function (the return)
funcgraph-duration
At the end of each function (the return)
the duration of the amount of time in the
function is displayed in microseconds.
funcgraph-abstime - When set, the timestamp is displayed at each
line.
funcgraph-abstime
When set, the timestamp is displayed at each line.
funcgraph-irqs - When disabled, functions that happen inside an
funcgraph-irqs
When disabled, functions that happen inside an
interrupt will not be traced.
funcgraph-tail - When set, the return event will include the function
funcgraph-tail
When set, the return event will include the function
that it represents. By default this is off, and
only a closing curly bracket "}" is displayed for
the return of a function.
sleep-time - When running function graph tracer, to include
sleep-time
When running function graph tracer, to include
the time a task schedules out in its function.
When enabled, it will account time the task has been
scheduled out as part of the function call.
graph-time - When running function profiler with function graph tracer,
graph-time
When running function profiler with function graph tracer,
to include the time to call nested functions. When this is
not set, the time reported for the function will only
include the time the function itself executed for, not the
......@@ -1146,7 +1208,8 @@ Options for function_graph tracer:
Options for blk tracer:
blk_classic - Shows a more minimalistic output.
blk_classic
Shows a more minimalistic output.
irqsoff
......@@ -1165,7 +1228,7 @@ new maximum is reached, the old saved trace is discarded and the
new trace is saved.
To reset the maximum, echo 0 into tracing_max_latency. Here is
an example:
an example::
# echo 0 > options/function-trace
# echo irqsoff > current_tracer
......@@ -1175,26 +1238,26 @@ an example:
[...]
# echo 0 > tracing_on
# cat trace
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 16 us, #4/4, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: swapper/0-0 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: run_timer_softirq
# => ended at: run_timer_softirq
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 16 us, #4/4, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: swapper/0-0 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: run_timer_softirq
# => ended at: run_timer_softirq
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
<idle>-0 0d.s2 0us+: _raw_spin_lock_irq <-run_timer_softirq
<idle>-0 0dNs3 17us : _raw_spin_unlock_irq <-run_timer_softirq
<idle>-0 0dNs3 17us+: trace_hardirqs_on <-run_timer_softirq
......@@ -1222,30 +1285,30 @@ between the time of recording the max latency and the time of
recording the function that had that latency.
Note the above example had function-trace not set. If we set
function-trace, we get a much larger output:
function-trace, we get a much larger output::
with echo 1 > options/function-trace
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 71 us, #168/168, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: bash-2042 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: ata_scsi_queuecmd
# => ended at: ata_scsi_queuecmd
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 71 us, #168/168, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: bash-2042 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: ata_scsi_queuecmd
# => ended at: ata_scsi_queuecmd
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
bash-2042 3d... 0us : _raw_spin_lock_irqsave <-ata_scsi_queuecmd
bash-2042 3d... 0us : add_preempt_count <-_raw_spin_lock_irqsave
bash-2042 3d..1 1us : ata_scsi_find_dev <-ata_scsi_queuecmd
......@@ -1255,7 +1318,7 @@ function-trace, we get a much larger output:
bash-2042 3d..1 3us : ata_sg_init <-__ata_scsi_queuecmd
bash-2042 3d..1 4us : ata_scsi_rw_xlat <-__ata_scsi_queuecmd
bash-2042 3d..1 4us : ata_build_rw_tf <-ata_scsi_rw_xlat
[...]
[...]
bash-2042 3d..1 67us : delay_tsc <-__delay
bash-2042 3d..1 67us : add_preempt_count <-delay_tsc
bash-2042 3d..2 67us : sub_preempt_count <-delay_tsc
......@@ -1312,6 +1375,7 @@ The preemptoff tracer traces the places that disable preemption.
Like the irqsoff tracer, it records the maximum latency for
which preemption was disabled. The control of preemptoff tracer
is much like the irqsoff tracer.
::
# echo 0 > options/function-trace
# echo preemptoff > current_tracer
......@@ -1321,26 +1385,26 @@ is much like the irqsoff tracer.
[...]
# echo 0 > tracing_on
# cat trace
# tracer: preemptoff
#
# preemptoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 46 us, #4/4, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: sshd-1991 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: do_IRQ
# => ended at: do_IRQ
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
# tracer: preemptoff
#
# preemptoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 46 us, #4/4, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: sshd-1991 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: do_IRQ
# => ended at: do_IRQ
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
sshd-1991 1d.h. 0us+: irq_enter <-do_IRQ
sshd-1991 1d..1 46us : irq_exit <-do_IRQ
sshd-1991 1d..1 47us+: trace_preempt_on <-do_IRQ
......@@ -1357,33 +1421,34 @@ But we also see that interrupts have been disabled when entering
the preempt off section and leaving it (the 'd'). We do not know if
interrupts were enabled in the mean time or shortly after this
was over.
::
# tracer: preemptoff
#
# preemptoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 83 us, #241/241, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: bash-1994 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: wake_up_new_task
# => ended at: task_rq_unlock
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
# tracer: preemptoff
#
# preemptoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 83 us, #241/241, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: bash-1994 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: wake_up_new_task
# => ended at: task_rq_unlock
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
bash-1994 1d..1 0us : _raw_spin_lock_irqsave <-wake_up_new_task
bash-1994 1d..1 0us : select_task_rq_fair <-select_task_rq
bash-1994 1d..1 1us : __rcu_read_lock <-select_task_rq_fair
bash-1994 1d..1 1us : source_load <-select_task_rq_fair
bash-1994 1d..1 1us : source_load <-select_task_rq_fair
[...]
[...]
bash-1994 1d..1 12us : irq_enter <-smp_apic_timer_interrupt
bash-1994 1d..1 12us : rcu_irq_enter <-irq_enter
bash-1994 1d..1 13us : add_preempt_count <-irq_enter
......@@ -1392,7 +1457,7 @@ was over.
bash-1994 1d.h1 13us : _raw_spin_lock <-hrtimer_interrupt
bash-1994 1d.h1 14us : add_preempt_count <-_raw_spin_lock
bash-1994 1d.h2 14us : ktime_get_update_offsets <-hrtimer_interrupt
[...]
[...]
bash-1994 1d.h1 35us : lapic_next_event <-clockevents_program_event
bash-1994 1d.h1 35us : irq_exit <-smp_apic_timer_interrupt
bash-1994 1d.h1 36us : sub_preempt_count <-irq_exit
......@@ -1403,7 +1468,7 @@ was over.
bash-1994 1d.s3 38us : _raw_spin_unlock <-run_timer_softirq
bash-1994 1d.s3 39us : sub_preempt_count <-_raw_spin_unlock
bash-1994 1d.s2 39us : call_timer_fn <-run_timer_softirq
[...]
[...]
bash-1994 1dNs2 81us : cpu_needs_another_gp <-rcu_process_callbacks
bash-1994 1dNs2 82us : __local_bh_enable <-__do_softirq
bash-1994 1dNs2 82us : sub_preempt_count <-__local_bh_enable
......@@ -1437,7 +1502,7 @@ preemption disabled for the longest times is helpful. But
sometimes we would like to know when either preemption and/or
interrupts are disabled.
Consider the following code:
Consider the following code::
local_irq_disable();
call_function_with_irqs_off();
......@@ -1462,6 +1527,7 @@ tracer.
Again, using this trace is much like the irqsoff and preemptoff
tracers.
::
# echo 0 > options/function-trace
# echo preemptirqsoff > current_tracer
......@@ -1471,26 +1537,26 @@ tracers.
[...]
# echo 0 > tracing_on
# cat trace
# tracer: preemptirqsoff
#
# preemptirqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 100 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: ls-2230 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: ata_scsi_queuecmd
# => ended at: ata_scsi_queuecmd
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
# tracer: preemptirqsoff
#
# preemptirqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 100 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: ls-2230 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: ata_scsi_queuecmd
# => ended at: ata_scsi_queuecmd
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
ls-2230 3d... 0us+: _raw_spin_lock_irqsave <-ata_scsi_queuecmd
ls-2230 3...1 100us : _raw_spin_unlock_irqrestore <-ata_scsi_queuecmd
ls-2230 3...1 101us+: trace_preempt_on <-ata_scsi_queuecmd
......@@ -1522,53 +1588,53 @@ function tracing, we do not know if interrupts were enabled
within the preemption points. We do see that it started with
preemption enabled.
Here is a trace with function-trace set:
# tracer: preemptirqsoff
#
# preemptirqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 161 us, #339/339, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: ls-2269 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: schedule
# => ended at: mutex_unlock
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
kworker/-59 3...1 0us : __schedule <-schedule
kworker/-59 3d..1 0us : rcu_preempt_qs <-rcu_note_context_switch
kworker/-59 3d..1 1us : add_preempt_count <-_raw_spin_lock_irq
kworker/-59 3d..2 1us : deactivate_task <-__schedule
kworker/-59 3d..2 1us : dequeue_task <-deactivate_task
kworker/-59 3d..2 2us : update_rq_clock <-dequeue_task
kworker/-59 3d..2 2us : dequeue_task_fair <-dequeue_task
kworker/-59 3d..2 2us : update_curr <-dequeue_task_fair
kworker/-59 3d..2 2us : update_min_vruntime <-update_curr
kworker/-59 3d..2 3us : cpuacct_charge <-update_curr
kworker/-59 3d..2 3us : __rcu_read_lock <-cpuacct_charge
kworker/-59 3d..2 3us : __rcu_read_unlock <-cpuacct_charge
kworker/-59 3d..2 3us : update_cfs_rq_blocked_load <-dequeue_task_fair
kworker/-59 3d..2 4us : clear_buddies <-dequeue_task_fair
kworker/-59 3d..2 4us : account_entity_dequeue <-dequeue_task_fair
kworker/-59 3d..2 4us : update_min_vruntime <-dequeue_task_fair
kworker/-59 3d..2 4us : update_cfs_shares <-dequeue_task_fair
kworker/-59 3d..2 5us : hrtick_update <-dequeue_task_fair
kworker/-59 3d..2 5us : wq_worker_sleeping <-__schedule
kworker/-59 3d..2 5us : kthread_data <-wq_worker_sleeping
kworker/-59 3d..2 5us : put_prev_task_fair <-__schedule
kworker/-59 3d..2 6us : pick_next_task_fair <-pick_next_task
kworker/-59 3d..2 6us : clear_buddies <-pick_next_task_fair
kworker/-59 3d..2 6us : set_next_entity <-pick_next_task_fair
kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity
Here is a trace with function-trace set::
# tracer: preemptirqsoff
#
# preemptirqsoff latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 161 us, #339/339, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: ls-2269 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: schedule
# => ended at: mutex_unlock
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
kworker/-59 3...1 0us : __schedule <-schedule
kworker/-59 3d..1 0us : rcu_preempt_qs <-rcu_note_context_switch
kworker/-59 3d..1 1us : add_preempt_count <-_raw_spin_lock_irq
kworker/-59 3d..2 1us : deactivate_task <-__schedule
kworker/-59 3d..2 1us : dequeue_task <-deactivate_task
kworker/-59 3d..2 2us : update_rq_clock <-dequeue_task
kworker/-59 3d..2 2us : dequeue_task_fair <-dequeue_task
kworker/-59 3d..2 2us : update_curr <-dequeue_task_fair
kworker/-59 3d..2 2us : update_min_vruntime <-update_curr
kworker/-59 3d..2 3us : cpuacct_charge <-update_curr
kworker/-59 3d..2 3us : __rcu_read_lock <-cpuacct_charge
kworker/-59 3d..2 3us : __rcu_read_unlock <-cpuacct_charge
kworker/-59 3d..2 3us : update_cfs_rq_blocked_load <-dequeue_task_fair
kworker/-59 3d..2 4us : clear_buddies <-dequeue_task_fair
kworker/-59 3d..2 4us : account_entity_dequeue <-dequeue_task_fair
kworker/-59 3d..2 4us : update_min_vruntime <-dequeue_task_fair
kworker/-59 3d..2 4us : update_cfs_shares <-dequeue_task_fair
kworker/-59 3d..2 5us : hrtick_update <-dequeue_task_fair
kworker/-59 3d..2 5us : wq_worker_sleeping <-__schedule
kworker/-59 3d..2 5us : kthread_data <-wq_worker_sleeping
kworker/-59 3d..2 5us : put_prev_task_fair <-__schedule
kworker/-59 3d..2 6us : pick_next_task_fair <-pick_next_task
kworker/-59 3d..2 6us : clear_buddies <-pick_next_task_fair
kworker/-59 3d..2 6us : set_next_entity <-pick_next_task_fair
kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity
ls-2269 3d..2 7us : finish_task_switch <-__schedule
ls-2269 3d..2 7us : _raw_spin_unlock_irq <-finish_task_switch
ls-2269 3d..2 8us : do_IRQ <-ret_from_intr
......@@ -1576,7 +1642,7 @@ kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity
ls-2269 3d..2 8us : rcu_irq_enter <-irq_enter
ls-2269 3d..2 9us : add_preempt_count <-irq_enter
ls-2269 3d.h2 9us : exit_idle <-do_IRQ
[...]
[...]
ls-2269 3d.h3 20us : sub_preempt_count <-_raw_spin_unlock
ls-2269 3d.h2 20us : irq_exit <-do_IRQ
ls-2269 3d.h2 21us : sub_preempt_count <-irq_exit
......@@ -1588,14 +1654,14 @@ kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity
ls-2269 3d.s5 31us : do_IRQ <-ret_from_intr
ls-2269 3d.s5 31us : irq_enter <-do_IRQ
ls-2269 3d.s5 31us : rcu_irq_enter <-irq_enter
[...]
[...]
ls-2269 3d.s5 31us : rcu_irq_enter <-irq_enter
ls-2269 3d.s5 32us : add_preempt_count <-irq_enter
ls-2269 3d.H5 32us : exit_idle <-do_IRQ
ls-2269 3d.H5 32us : handle_irq <-do_IRQ
ls-2269 3d.H5 32us : irq_to_desc <-handle_irq
ls-2269 3d.H5 33us : handle_fasteoi_irq <-handle_irq
[...]
[...]
ls-2269 3d.s5 158us : _raw_spin_unlock_irqrestore <-rtl8139_poll
ls-2269 3d.s3 158us : net_rps_action_and_irq_enable.isra.65 <-net_rx_action
ls-2269 3d.s3 159us : __local_bh_enable <-__do_softirq
......@@ -1631,7 +1697,7 @@ time it takes for a task that is woken to actually wake up.
Now for non Real-Time tasks, this can be arbitrary. But tracing
it none the less can be interesting.
Without function tracing:
Without function tracing::
# echo 0 > options/function-trace
# echo wakeup > current_tracer
......@@ -1640,23 +1706,23 @@ Without function tracing:
# chrt -f 5 sleep 1
# echo 0 > tracing_on
# cat trace
# tracer: wakeup
#
# wakeup latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 15 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: kworker/3:1H-312 (uid:0 nice:-20 policy:0 rt_prio:0)
# -----------------
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
# tracer: wakeup
#
# wakeup latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 15 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: kworker/3:1H-312 (uid:0 nice:-20 policy:0 rt_prio:0)
# -----------------
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
<idle>-0 3dNs7 0us : 0:120:R + [003] 312:100:R kworker/3:1H
<idle>-0 3dNs7 1us+: ttwu_do_activate.constprop.87 <-try_to_wake_up
<idle>-0 3d..3 15us : __schedule <-schedule
......@@ -1698,6 +1764,7 @@ Since this tracer only deals with RT tasks, we will run this
slightly differently than we did with the previous tracers.
Instead of performing an 'ls', we will run 'sleep 1' under
'chrt' which changes the priority of the task.
::
# echo 0 > options/function-trace
# echo wakeup_rt > current_tracer
......@@ -1706,25 +1773,25 @@ Instead of performing an 'ls', we will run 'sleep 1' under
# chrt -f 5 sleep 1
# echo 0 > tracing_on
# cat trace
# tracer: wakeup
#
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 5 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: sleep-2389 (uid:0 nice:0 policy:1 rt_prio:5)
# -----------------
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
# tracer: wakeup
#
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 5 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: sleep-2389 (uid:0 nice:0 policy:1 rt_prio:5)
# -----------------
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
<idle>-0 3d.h4 0us : 0:120:R + [003] 2389: 94:R sleep
<idle>-0 3d.h4 1us+: ttwu_do_activate.constprop.87 <-try_to_wake_up
<idle>-0 3d..3 5us : __schedule <-schedule
......@@ -1743,6 +1810,7 @@ and not the internal kernel priority. The policy is 1 for
SCHED_FIFO and 2 for SCHED_RR.
Note, that the trace data shows the internal priority (99 - rtprio).
::
<idle>-0 3d..3 5us : 0:120:R ==> [003] 2389: 94:R sleep
......@@ -1752,26 +1820,27 @@ and in the running state 'R'. The sleep task was scheduled in with
and it too is in the running state.
Doing the same with chrt -r 5 and function-trace set.
::
echo 1 > options/function-trace
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 29 us, #85/85, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: sleep-2448 (uid:0 nice:0 policy:1 rt_prio:5)
# -----------------
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 29 us, #85/85, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: sleep-2448 (uid:0 nice:0 policy:1 rt_prio:5)
# -----------------
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
<idle>-0 3d.h4 1us+: 0:120:R + [003] 2448: 94:R sleep
<idle>-0 3d.h4 2us : ttwu_do_activate.constprop.87 <-try_to_wake_up
<idle>-0 3d.h3 3us : check_preempt_curr <-ttwu_do_wakeup
......@@ -1871,6 +1940,7 @@ As function tracing can induce a much larger latency, but without
seeing what happens within the latency it is hard to know what
caused it. There is a middle ground, and that is with enabling
events.
::
# echo 0 > options/function-trace
# echo wakeup_rt > current_tracer
......@@ -1880,23 +1950,23 @@ events.
# chrt -f 5 sleep 1
# echo 0 > tracing_on
# cat trace
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 6 us, #12/12, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: sleep-5882 (uid:0 nice:0 policy:1 rt_prio:5)
# -----------------
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 3.8.0-test+
# --------------------------------------------------------------------
# latency: 6 us, #12/12, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: sleep-5882 (uid:0 nice:0 policy:1 rt_prio:5)
# -----------------
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
<idle>-0 2d.h4 0us : 0:120:R + [002] 5882: 94:R sleep
<idle>-0 2d.h4 0us : ttwu_do_activate.constprop.87 <-try_to_wake_up
<idle>-0 2d.h4 1us : sched_wakeup: comm=sleep pid=5882 prio=94 success=1 target_cpu=002
......@@ -1918,19 +1988,20 @@ The hardware latency detector is executed by enabling the "hwlat" tracer.
NOTE, this tracer will affect the performance of the system as it will
periodically make a CPU constantly busy with interrupts disabled.
::
# echo hwlat > current_tracer
# sleep 100
# cat trace
# tracer: hwlat
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
# tracer: hwlat
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<...>-3638 [001] d... 19452.055471: #1 inner/outer(us): 12/14 ts:1499801089.066141940
<...>-3638 [003] d... 19454.071354: #2 inner/outer(us): 11/9 ts:1499801091.082164365
<...>-3638 [002] dn.. 19461.126852: #3 inner/outer(us): 12/9 ts:1499801098.138150062
......@@ -1942,7 +2013,8 @@ periodically make a CPU constantly busy with interrupts disabled.
The above output is somewhat the same in the header. All events will have
interrupts disabled 'd'. Under the FUNCTION title there is:
#1 - This is the count of events recorded that were greater than the
#1
This is the count of events recorded that were greater than the
tracing_threshold (See below).
inner/outer(us): 12/14
......@@ -1968,7 +2040,8 @@ interrupts disabled 'd'. Under the FUNCTION title there is:
hwlat files:
tracing_threshold - This gets automatically set to "10" to represent 10
tracing_threshold
This gets automatically set to "10" to represent 10
microseconds. This is the threshold of latency that
needs to be detected before the trace will be recorded.
......@@ -1976,14 +2049,16 @@ hwlat files:
written into "current_tracer"), the original value for
tracing_threshold is placed back into this file.
hwlat_detector/width - The length of time the test runs with interrupts
disabled.
hwlat_detector/width
The length of time the test runs with interrupts disabled.
hwlat_detector/window - The length of time of the window which the test
hwlat_detector/window
The length of time of the window which the test
runs. That is, the test will run for "width"
microseconds per "window" microseconds
tracing_cpumask - When the test is started. A kernel thread is created that
tracing_cpumask
When the test is started. A kernel thread is created that
runs the test. This thread will alternate between CPUs
listed in the tracing_cpumask between each period
(one "window"). To limit the test to specific CPUs
......@@ -1997,6 +2072,7 @@ This tracer is the function tracer. Enabling the function tracer
can be done from the debug file system. Make sure the
ftrace_enabled is set; otherwise this tracer is a nop.
See the "ftrace_enabled" section below.
::
# sysctl kernel.ftrace_enabled=1
# echo function > current_tracer
......@@ -2004,17 +2080,17 @@ See the "ftrace_enabled" section below.
# usleep 1
# echo 0 > tracing_on
# cat trace
# tracer: function
#
# entries-in-buffer/entries-written: 24799/24799 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
# tracer: function
#
# entries-in-buffer/entries-written: 24799/24799 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
bash-1994 [002] .... 3082.063030: mutex_unlock <-rb_simple_write
bash-1994 [002] .... 3082.063031: __mutex_unlock_slowpath <-mutex_unlock
bash-1994 [002] .... 3082.063031: __fsnotify_parent <-fsnotify_modify
......@@ -2023,7 +2099,7 @@ See the "ftrace_enabled" section below.
bash-1994 [002] .... 3082.063032: add_preempt_count <-__srcu_read_lock
bash-1994 [002] ...1 3082.063032: sub_preempt_count <-__srcu_read_lock
bash-1994 [002] .... 3082.063033: __srcu_read_unlock <-fsnotify
[...]
[...]
Note: function tracer uses ring buffers to store the above
......@@ -2034,11 +2110,11 @@ record. For this reason, it is sometimes better to disable
tracing directly from a program. This allows you to stop the
tracing at the point that you hit the part that you are
interested in. To disable the tracing directly from a C program,
something like following code snippet can be used:
something like following code snippet can be used::
int trace_fd;
[...]
int main(int argc, char *argv[]) {
int trace_fd;
[...]
int main(int argc, char *argv[]) {
[...]
trace_fd = open(tracing_file("tracing_on"), O_WRONLY);
[...]
......@@ -2046,22 +2122,22 @@ int main(int argc, char *argv[]) {
write(trace_fd, "0", 1);
}
[...]
}
}
Single thread tracing
---------------------
By writing into set_ftrace_pid you can trace a
single thread. For example:
# cat set_ftrace_pid
no pid
# echo 3111 > set_ftrace_pid
# cat set_ftrace_pid
3111
# echo function > current_tracer
# cat trace | head
single thread. For example::
# cat set_ftrace_pid
no pid
# echo 3111 > set_ftrace_pid
# cat set_ftrace_pid
3111
# echo function > current_tracer
# cat trace | head
# tracer: function
#
# TASK-PID CPU# TIMESTAMP FUNCTION
......@@ -2072,8 +2148,8 @@ no pid
yum-updatesd-3111 [003] 1637.254683: lock_hrtimer_base <-hrtimer_try_to_cancel
yum-updatesd-3111 [003] 1637.254685: fget_light <-do_sys_poll
yum-updatesd-3111 [003] 1637.254686: pipe_poll <-do_sys_poll
# echo > set_ftrace_pid
# cat trace |head
# echo > set_ftrace_pid
# cat trace |head
# tracer: function
#
# TASK-PID CPU# TIMESTAMP FUNCTION
......@@ -2086,22 +2162,23 @@ no pid
yum-updatesd-3111 [003] 1701.957693: path_put <-audit_syscall_exit
If you want to trace a function when executing, you could use
something like this simple program:
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#define _STR(x) #x
#define STR(x) _STR(x)
#define MAX_PATH 256
const char *find_tracefs(void)
{
something like this simple program.
::
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#define _STR(x) #x
#define STR(x) _STR(x)
#define MAX_PATH 256
const char *find_tracefs(void)
{
static char tracefs[MAX_PATH+1];
static int tracefs_found;
char type[100];
......@@ -2133,17 +2210,17 @@ const char *find_tracefs(void)
tracefs_found = 1;
return tracefs;
}
}
const char *tracing_file(const char *file_name)
{
const char *tracing_file(const char *file_name)
{
static char trace_file[MAX_PATH+1];
snprintf(trace_file, MAX_PATH, "%s/%s", find_tracefs(), file_name);
return trace_file;
}
}
int main (int argc, char **argv)
{
int main (int argc, char **argv)
{
if (argc < 1)
exit(-1);
......@@ -2170,21 +2247,20 @@ int main (int argc, char **argv)
}
return 0;
}
}
Or this simple script!
::
------
#!/bin/bash
tracefs=`sed -ne 's/^tracefs \(.*\) tracefs.*/\1/p' /proc/mounts`
echo nop > $tracefs/tracing/current_tracer
echo 0 > $tracefs/tracing/tracing_on
echo $$ > $tracefs/tracing/set_ftrace_pid
echo function > $tracefs/tracing/current_tracer
echo 1 > $tracefs/tracing/tracing_on
exec "$@"
------
#!/bin/bash
tracefs=`sed -ne 's/^tracefs \(.*\) tracefs.*/\1/p' /proc/mounts`
echo nop > $tracefs/tracing/current_tracer
echo 0 > $tracefs/tracing/tracing_on
echo $$ > $tracefs/tracing/set_ftrace_pid
echo function > $tracefs/tracing/current_tracer
echo 1 > $tracefs/tracing/tracing_on
exec "$@"
function graph tracer
......@@ -2219,10 +2295,12 @@ This tracer is useful in several situations:
- you just want to peek inside a working kernel and want to see
what happens there.
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
::
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) | sys_open() {
0) | do_sys_open() {
......@@ -2251,25 +2329,25 @@ want, depending on your needs.
tracing_cpu_mask file) or you might sometimes see unordered
function calls while cpu tracing switch.
hide: echo nofuncgraph-cpu > trace_options
show: echo funcgraph-cpu > trace_options
- hide: echo nofuncgraph-cpu > trace_options
- show: echo funcgraph-cpu > trace_options
- The duration (function's time of execution) is displayed on
the closing bracket line of a function or on the same line
than the current function in case of a leaf one. It is default
enabled.
hide: echo nofuncgraph-duration > trace_options
show: echo funcgraph-duration > trace_options
- hide: echo nofuncgraph-duration > trace_options
- show: echo funcgraph-duration > trace_options
- The overhead field precedes the duration field in case of
reached duration thresholds.
hide: echo nofuncgraph-overhead > trace_options
show: echo funcgraph-overhead > trace_options
depends on: funcgraph-duration
- hide: echo nofuncgraph-overhead > trace_options
- show: echo funcgraph-overhead > trace_options
- depends on: funcgraph-duration
ie:
ie::
3) # 1837.709 us | } /* __switch_to */
3) | finish_task_switch() {
......@@ -2309,6 +2387,8 @@ want, depending on your needs.
1) + 18.542 us | }
2) $ 3594274 us | }
Flags::
+ means that the function exceeded 10 usecs.
! means that the function exceeded 100 usecs.
# means that the function exceeded 1000 usecs.
......@@ -2320,10 +2400,10 @@ want, depending on your needs.
- The task/pid field displays the thread cmdline and pid which
executed the function. It is default disabled.
hide: echo nofuncgraph-proc > trace_options
show: echo funcgraph-proc > trace_options
- hide: echo nofuncgraph-proc > trace_options
- show: echo funcgraph-proc > trace_options
ie:
ie::
# tracer: function_graph
#
......@@ -2344,10 +2424,10 @@ want, depending on your needs.
system clock since it started. A snapshot of this time is
given on each entry/exit of functions
hide: echo nofuncgraph-abstime > trace_options
show: echo funcgraph-abstime > trace_options
- hide: echo nofuncgraph-abstime > trace_options
- show: echo funcgraph-abstime > trace_options
ie:
ie::
#
# TIME CPU DURATION FUNCTION CALLS
......@@ -2376,17 +2456,19 @@ enabled for functions whose start is in the trace buffer,
allowing easier searching with grep for function durations.
It is default disabled.
hide: echo nofuncgraph-tail > trace_options
show: echo funcgraph-tail > trace_options
- hide: echo nofuncgraph-tail > trace_options
- show: echo funcgraph-tail > trace_options
Example with nofuncgraph-tail (default)::
Example with nofuncgraph-tail (default):
0) | putname() {
0) | kmem_cache_free() {
0) 0.518 us | __phys_addr();
0) 1.757 us | }
0) 2.861 us | }
Example with funcgraph-tail:
Example with funcgraph-tail::
0) | putname() {
0) | kmem_cache_free() {
0) 0.518 us | __phys_addr();
......@@ -2396,11 +2478,11 @@ It is default disabled.
You can put some comments on specific functions by using
trace_printk() For example, if you want to put a comment inside
the __might_sleep() function, you just have to include
<linux/ftrace.h> and call trace_printk() inside __might_sleep()
<linux/ftrace.h> and call trace_printk() inside __might_sleep()::
trace_printk("I'm a comment!\n")
trace_printk("I'm a comment!\n")
will produce:
will produce::
1) | __might_sleep() {
1) | /* I'm a comment! */
......@@ -2487,16 +2569,18 @@ listed in:
available_filter_functions
::
# cat available_filter_functions
put_prev_task_idle
kmem_cache_create
pick_next_task_rt
get_online_cpus
pick_next_task_fair
mutex_lock
[...]
put_prev_task_idle
kmem_cache_create
pick_next_task_rt
get_online_cpus
pick_next_task_fair
mutex_lock
[...]
If I am only interested in sys_nanosleep and hrtimer_interrupt:
If I am only interested in sys_nanosleep and hrtimer_interrupt::
# echo sys_nanosleep hrtimer_interrupt > set_ftrace_filter
# echo function > current_tracer
......@@ -2504,17 +2588,17 @@ If I am only interested in sys_nanosleep and hrtimer_interrupt:
# usleep 1
# echo 0 > tracing_on
# cat trace
# tracer: function
#
# entries-in-buffer/entries-written: 5/5 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
# tracer: function
#
# entries-in-buffer/entries-written: 5/5 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
usleep-2665 [001] .... 4186.475355: sys_nanosleep <-system_call_fastpath
<idle>-0 [001] d.h1 4186.475409: hrtimer_interrupt <-smp_apic_timer_interrupt
usleep-2665 [001] d.h1 4186.475426: hrtimer_interrupt <-smp_apic_timer_interrupt
......@@ -2522,39 +2606,46 @@ If I am only interested in sys_nanosleep and hrtimer_interrupt:
<idle>-0 [002] d.h1 4186.475427: hrtimer_interrupt <-smp_apic_timer_interrupt
To see which functions are being traced, you can cat the file:
::
# cat set_ftrace_filter
hrtimer_interrupt
sys_nanosleep
hrtimer_interrupt
sys_nanosleep
Perhaps this is not enough. The filters also allow glob(7) matching.
<match>* - will match functions that begin with <match>
*<match> - will match functions that end with <match>
*<match>* - will match functions that have <match> in it
<match1>*<match2> - will match functions that begin with
<match1> and end with <match2>
Note: It is better to use quotes to enclose the wild cards,
<match>*
will match functions that begin with <match>
*<match>
will match functions that end with <match>
*<match>*
will match functions that have <match> in it
<match1>*<match2>
will match functions that begin with <match1> and end with <match2>
.. note::
It is better to use quotes to enclose the wild cards,
otherwise the shell may expand the parameters into names
of files in the local directory.
::
# echo 'hrtimer_*' > set_ftrace_filter
Produces:
# tracer: function
#
# entries-in-buffer/entries-written: 897/897 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
Produces::
# tracer: function
#
# entries-in-buffer/entries-written: 897/897 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<idle>-0 [003] dN.1 4228.547803: hrtimer_cancel <-tick_nohz_idle_exit
<idle>-0 [003] dN.1 4228.547804: hrtimer_try_to_cancel <-hrtimer_cancel
<idle>-0 [003] dN.2 4228.547805: hrtimer_force_reprogram <-__remove_hrtimer
......@@ -2565,24 +2656,25 @@ Produces:
<idle>-0 [003] d..2 4228.547860: hrtimer_force_reprogram <-__rem
Notice that we lost the sys_nanosleep.
::
# cat set_ftrace_filter
hrtimer_run_queues
hrtimer_run_pending
hrtimer_init
hrtimer_cancel
hrtimer_try_to_cancel
hrtimer_forward
hrtimer_start
hrtimer_reprogram
hrtimer_force_reprogram
hrtimer_get_next_event
hrtimer_interrupt
hrtimer_nanosleep
hrtimer_wakeup
hrtimer_get_remaining
hrtimer_get_res
hrtimer_init_sleeper
hrtimer_run_queues
hrtimer_run_pending
hrtimer_init
hrtimer_cancel
hrtimer_try_to_cancel
hrtimer_forward
hrtimer_start
hrtimer_reprogram
hrtimer_force_reprogram
hrtimer_get_next_event
hrtimer_interrupt
hrtimer_nanosleep
hrtimer_wakeup
hrtimer_get_remaining
hrtimer_get_res
hrtimer_init_sleeper
This is because the '>' and '>>' act just like they do in bash.
......@@ -2590,7 +2682,7 @@ To rewrite the filters, use '>'
To append to the filters, use '>>'
To clear out a filter so that all functions will be recorded
again:
again::
# echo > set_ftrace_filter
# cat set_ftrace_filter
......@@ -2598,48 +2690,51 @@ again:
Again, now we want to append.
::
# echo sys_nanosleep > set_ftrace_filter
# cat set_ftrace_filter
sys_nanosleep
sys_nanosleep
# echo 'hrtimer_*' >> set_ftrace_filter
# cat set_ftrace_filter
hrtimer_run_queues
hrtimer_run_pending
hrtimer_init
hrtimer_cancel
hrtimer_try_to_cancel
hrtimer_forward
hrtimer_start
hrtimer_reprogram
hrtimer_force_reprogram
hrtimer_get_next_event
hrtimer_interrupt
sys_nanosleep
hrtimer_nanosleep
hrtimer_wakeup
hrtimer_get_remaining
hrtimer_get_res
hrtimer_init_sleeper
hrtimer_run_queues
hrtimer_run_pending
hrtimer_init
hrtimer_cancel
hrtimer_try_to_cancel
hrtimer_forward
hrtimer_start
hrtimer_reprogram
hrtimer_force_reprogram
hrtimer_get_next_event
hrtimer_interrupt
sys_nanosleep
hrtimer_nanosleep
hrtimer_wakeup
hrtimer_get_remaining
hrtimer_get_res
hrtimer_init_sleeper
The set_ftrace_notrace prevents those functions from being
traced.
::
# echo '*preempt*' '*lock*' > set_ftrace_notrace
Produces:
# tracer: function
#
# entries-in-buffer/entries-written: 39608/39608 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
Produces::
# tracer: function
#
# entries-in-buffer/entries-written: 39608/39608 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
bash-1994 [000] .... 4342.324896: file_ra_state_init <-do_dentry_open
bash-1994 [000] .... 4342.324897: open_check_o_direct <-do_last
bash-1994 [000] .... 4342.324897: ima_file_check <-do_last
......@@ -2664,12 +2759,12 @@ function tracer and the function-graph-tracer, there are some
special features only available in the function-graph tracer.
If you want to trace only one function and all of its children,
you just have to echo its name into set_graph_function:
you just have to echo its name into set_graph_function::
echo __do_fault > set_graph_function
will produce the following "expanded" trace of the __do_fault()
function:
function::
0) | __do_fault() {
0) | filemap_fault() {
......@@ -2706,13 +2801,13 @@ function:
0) 2.793 us | }
0) + 14.012 us | }
You can also expand several functions at once:
You can also expand several functions at once::
echo sys_open > set_graph_function
echo sys_close >> set_graph_function
Now if you want to go back to trace all functions you can clear
this special filter via:
this special filter via::
echo > set_graph_function
......@@ -2728,7 +2823,7 @@ also for any other uses (perf, kprobes, stack tracing, profiling, etc).
Please disable this with care.
This can be disable (and enabled) with:
This can be disable (and enabled) with::
sysctl kernel.ftrace_enabled=0
sysctl kernel.ftrace_enabled=1
......@@ -2743,13 +2838,13 @@ Filter commands
---------------
A few commands are supported by the set_ftrace_filter interface.
Trace commands have the following format:
Trace commands have the following format::
<function>:<command>:<parameter>
<function>:<command>:<parameter>
The following commands are supported:
- mod
- mod:
This command enables function filtering per module. The
parameter defines the module. For example, if only the write*
functions in the ext3 module are desired, run:
......@@ -2760,97 +2855,99 @@ The following commands are supported:
filtering based on function names. Thus, adding more functions
in a different module is accomplished by appending (>>) to the
filter file. Remove specific module functions by prepending
'!':
'!'::
echo '!writeback*:mod:ext3' >> set_ftrace_filter
Mod command supports module globbing. Disable tracing for all
functions except a specific module:
functions except a specific module::
echo '!*:mod:!ext3' >> set_ftrace_filter
Disable tracing for all modules, but still trace kernel:
Disable tracing for all modules, but still trace kernel::
echo '!*:mod:*' >> set_ftrace_filter
Enable filter only for kernel:
Enable filter only for kernel::
echo '*write*:mod:!*' >> set_ftrace_filter
Enable filter for module globbing:
Enable filter for module globbing::
echo '*write*:mod:*snd*' >> set_ftrace_filter
- traceon/traceoff
- traceon/traceoff:
These commands turn tracing on and off when the specified
functions are hit. The parameter determines how many times the
tracing system is turned on and off. If unspecified, there is
no limit. For example, to disable tracing when a schedule bug
is hit the first 5 times, run:
is hit the first 5 times, run::
echo '__schedule_bug:traceoff:5' > set_ftrace_filter
To always disable tracing when __schedule_bug is hit:
To always disable tracing when __schedule_bug is hit::
echo '__schedule_bug:traceoff' > set_ftrace_filter
These commands are cumulative whether or not they are appended
to set_ftrace_filter. To remove a command, prepend it by '!'
and drop the parameter:
and drop the parameter::
echo '!__schedule_bug:traceoff:0' > set_ftrace_filter
The above removes the traceoff command for __schedule_bug
that have a counter. To remove commands without counters:
that have a counter. To remove commands without counters::
echo '!__schedule_bug:traceoff' > set_ftrace_filter
- snapshot
- snapshot:
Will cause a snapshot to be triggered when the function is hit.
::
echo 'native_flush_tlb_others:snapshot' > set_ftrace_filter
To only snapshot once:
::
echo 'native_flush_tlb_others:snapshot:1' > set_ftrace_filter
To remove the above commands:
To remove the above commands::
echo '!native_flush_tlb_others:snapshot' > set_ftrace_filter
echo '!native_flush_tlb_others:snapshot:0' > set_ftrace_filter
- enable_event/disable_event
- enable_event/disable_event:
These commands can enable or disable a trace event. Note, because
function tracing callbacks are very sensitive, when these commands
are registered, the trace point is activated, but disabled in
a "soft" mode. That is, the tracepoint will be called, but
just will not be traced. The event tracepoint stays in this mode
as long as there's a command that triggers it.
::
echo 'try_to_wake_up:enable_event:sched:sched_switch:2' > \
set_ftrace_filter
The format is:
The format is::
<function>:enable_event:<system>:<event>[:count]
<function>:disable_event:<system>:<event>[:count]
To remove the events commands:
To remove the events commands::
echo '!try_to_wake_up:enable_event:sched:sched_switch:0' > \
set_ftrace_filter
echo '!schedule:disable_event:sched:sched_switch' > \
set_ftrace_filter
- dump
- dump:
When the function is hit, it will dump the contents of the ftrace
ring buffer to the console. This is useful if you need to debug
something, and want to dump the trace when a certain function
is hit. Perhaps its a function that is called before a tripple
fault happens and does not allow you to get a regular dump.
- cpudump
- cpudump:
When the function is hit, it will dump the contents of the ftrace
ring buffer for the current CPU to the console. Unlike the "dump"
command, it only prints out the contents of the ring buffer for the
......@@ -2863,25 +2960,26 @@ The trace_pipe outputs the same content as the trace file, but
the effect on the tracing is different. Every read from
trace_pipe is consumed. This means that subsequent reads will be
different. The trace is live.
::
# echo function > current_tracer
# cat trace_pipe > /tmp/trace.out &
[1] 4153
[1] 4153
# echo 1 > tracing_on
# usleep 1
# echo 0 > tracing_on
# cat trace
# tracer: function
#
# entries-in-buffer/entries-written: 0/0 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
# tracer: function
#
# entries-in-buffer/entries-written: 0/0 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
#
# cat /tmp/trace.out
......@@ -2908,44 +3006,51 @@ used to modify the size of the internal trace buffers. The
number listed is the number of entries that can be recorded per
CPU. To know the full size, multiply the number of possible CPUs
with the number of entries.
::
# cat buffer_size_kb
1408 (units kilobytes)
1408 (units kilobytes)
Or simply read buffer_total_size_kb
::
# cat buffer_total_size_kb
5632
5632
To modify the buffer, simple echo in a number (in 1024 byte segments).
::
# echo 10000 > buffer_size_kb
# cat buffer_size_kb
10000 (units kilobytes)
10000 (units kilobytes)
It will try to allocate as much as possible. If you allocate too
much, it can cause Out-Of-Memory to trigger.
::
# echo 1000000000000 > buffer_size_kb
-bash: echo: write error: Cannot allocate memory
-bash: echo: write error: Cannot allocate memory
# cat buffer_size_kb
85
85
The per_cpu buffers can be changed individually as well:
::
# echo 10000 > per_cpu/cpu0/buffer_size_kb
# echo 100 > per_cpu/cpu1/buffer_size_kb
When the per_cpu buffers are not the same, the buffer_size_kb
at the top level will just show an X
::
# cat buffer_size_kb
X
X
This is where the buffer_total_size_kb is useful:
::
# cat buffer_total_size_kb
12916
12916
Writing to the top level buffer_size_kb will reset all the buffers
to be the same again.
......@@ -2979,59 +3084,62 @@ feature:
snapshot contents.
More details are shown in the table below.
status\input | 0 | 1 | else |
--------------+------------+------------+------------+
not allocated |(do nothing)| alloc+swap |(do nothing)|
--------------+------------+------------+------------+
allocated | free | swap | clear |
--------------+------------+------------+------------+
+--------------+------------+------------+------------+
|status\\input | 0 | 1 | else |
+==============+============+============+============+
|not allocated |(do nothing)| alloc+swap |(do nothing)|
+--------------+------------+------------+------------+
|allocated | free | swap | clear |
+--------------+------------+------------+------------+
Here is an example of using the snapshot feature.
::
# echo 1 > events/sched/enable
# echo 1 > snapshot
# cat snapshot
# tracer: nop
#
# entries-in-buffer/entries-written: 71/71 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
# tracer: nop
#
# entries-in-buffer/entries-written: 71/71 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<idle>-0 [005] d... 2440.603828: sched_switch: prev_comm=swapper/5 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2242 next_prio=120
sleep-2242 [005] d... 2440.603846: sched_switch: prev_comm=snapshot-test-2 prev_pid=2242 prev_prio=120 prev_state=R ==> next_comm=kworker/5:1 next_pid=60 next_prio=120
[...]
[...]
<idle>-0 [002] d... 2440.707230: sched_switch: prev_comm=swapper/2 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2229 next_prio=120
# cat trace
# tracer: nop
#
# entries-in-buffer/entries-written: 77/77 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
# tracer: nop
#
# entries-in-buffer/entries-written: 77/77 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<idle>-0 [007] d... 2440.707395: sched_switch: prev_comm=swapper/7 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2243 next_prio=120
snapshot-test-2-2229 [002] d... 2440.707438: sched_switch: prev_comm=snapshot-test-2 prev_pid=2229 prev_prio=120 prev_state=S ==> next_comm=swapper/2 next_pid=0 next_prio=120
[...]
[...]
If you try to use this snapshot feature when current tracer is
one of the latency tracers, you will get the following results.
::
# echo wakeup > current_tracer
# echo 1 > snapshot
bash: echo: write error: Device or resource busy
bash: echo: write error: Device or resource busy
# cat snapshot
cat: snapshot: Device or resource busy
cat: snapshot: Device or resource busy
Instances
......@@ -3041,12 +3149,13 @@ This directory can have new directories created inside of it using
mkdir, and removing directories with rmdir. The directory created
with mkdir in this directory will already contain files and other
directories after it is created.
::
# mkdir instances/foo
# ls instances/foo
buffer_size_kb buffer_total_size_kb events free_buffer per_cpu
set_event snapshot trace trace_clock trace_marker trace_options
trace_pipe tracing_on
buffer_size_kb buffer_total_size_kb events free_buffer per_cpu
set_event snapshot trace trace_clock trace_marker trace_options
trace_pipe tracing_on
As you can see, the new directory looks similar to the tracing directory
itself. In fact, it is very similar, except that the buffer and
......@@ -3064,6 +3173,7 @@ may become specific to the instance they reside in.
Notice that none of the function tracer files are there, nor is
current_tracer and available_tracers. This is because the buffers
can currently only have events enabled for them.
::
# mkdir instances/foo
# mkdir instances/bar
......@@ -3078,7 +3188,7 @@ can currently only have events enabled for them.
# echo 1 > instances/bar/events/irq/enable
# echo 1 > instances/zoot/events/syscalls/enable
# cat trace_pipe
CPU:2 [LOST 11745 EVENTS]
CPU:2 [LOST 11745 EVENTS]
bash-2044 [002] .... 10594.481032: _raw_spin_lock_irqsave <-get_page_from_freelist
bash-2044 [002] d... 10594.481032: add_preempt_count <-_raw_spin_lock_irqsave
bash-2044 [002] d..1 10594.481032: __rmqueue <-get_page_from_freelist
......@@ -3090,7 +3200,7 @@ CPU:2 [LOST 11745 EVENTS]
bash-2044 [002] d... 10594.481034: __inc_zone_state <-zone_statistics
bash-2044 [002] d... 10594.481034: __inc_zone_state <-zone_statistics
bash-2044 [002] .... 10594.481035: arch_dup_task_struct <-copy_process
[...]
[...]
# cat instances/foo/trace_pipe
bash-1998 [000] d..4 136.676759: sched_wakeup: comm=kworker/0:1 pid=59 prio=120 success=1 target_cpu=000
......@@ -3103,7 +3213,7 @@ CPU:2 [LOST 11745 EVENTS]
bash-1998 [000] d..3 136.677018: sched_switch: prev_comm=bash prev_pid=1998 prev_prio=120 prev_state=R+ ==> next_comm=kworker/0:1 next_pid=59 next_prio=120
kworker/0:1-59 [000] d..4 136.677022: sched_wakeup: comm=sshd pid=1995 prio=120 success=1 target_cpu=001
kworker/0:1-59 [000] d..3 136.677025: sched_switch: prev_comm=kworker/0:1 prev_pid=59 prev_prio=120 prev_state=S ==> next_comm=bash next_pid=1998 next_prio=120
[...]
[...]
# cat instances/bar/trace_pipe
migration/1-14 [001] d.h3 138.732674: softirq_raise: vec=3 [action=NET_RX]
......@@ -3118,20 +3228,20 @@ CPU:2 [LOST 11745 EVENTS]
sshd-1995 [001] d.h1 138.733280: irq_handler_exit: irq=21 ret=unhandled
sshd-1995 [001] d.h1 138.733281: irq_handler_entry: irq=21 name=eth0
sshd-1995 [001] d.h1 138.733283: irq_handler_exit: irq=21 ret=handled
[...]
[...]
# cat instances/zoot/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 18996/18996 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
# tracer: nop
#
# entries-in-buffer/entries-written: 18996/18996 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
bash-1998 [000] d... 140.733501: sys_write -> 0x2
bash-1998 [000] d... 140.733504: sys_dup2(oldfd: a, newfd: 1)
bash-1998 [000] d... 140.733506: sys_dup2 -> 0x1
......@@ -3149,6 +3259,7 @@ the function tracing. The foo instance displays wakeups and task
switches.
To remove the instances, simply delete their directories:
::
# rmdir instances/foo
# rmdir instances/bar
......@@ -3174,6 +3285,7 @@ at every function call. This is enabled via the stack tracer.
CONFIG_STACK_TRACER enables the ftrace stack tracing functionality.
To enable it, write a '1' into /proc/sys/kernel/stack_tracer_enabled.
::
# echo 1 > /proc/sys/kernel/stack_tracer_enabled
......@@ -3182,9 +3294,10 @@ the stack size of the kernel during boot up, by adding "stacktrace"
to the kernel command line parameter.
After running it for a few minutes, the output looks like:
::
# cat stack_max_size
2928
2928
# cat stack_trace
Depth Size Location (18 entries)
......@@ -3214,7 +3327,6 @@ are not tested by the stack tracer when -mfentry is used.
Currently, -mfentry is used by gcc 4.6.0 and above on x86 only.
---------
More details can be found in the source code, in the
kernel/trace/*.c files.
More
----
More details can be found in the source code, in the `kernel/trace/*.c` files.
Documentation/trace/index.rst
View file @ 1f198e22
......@@ -7,4 +7,5 @@ Linux Tracing Technologies
ftrace-design
tracepoint-analysis
ftrace
ftrace-uses
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7