Skip to content

L
linux
Commit 1f198e22 authored by Changbin Du's avatar Changbin Du Committed by Jonathan Corbet
Browse files
Options

trace doc: convert trace/ftrace.txt to rst format 

This converts the plain text documentation to reStructuredText format and
add it into Sphinx TOC tree. No essential content change.

Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: default avatarChangbin Du <changbin.du@intel.com>
Signed-off-by: default avatarJonathan Corbet <corbet@lwn.net>
parent 8fa4e720
Show whitespace changes
Inline Side-by-side
Showing with 3333 additions and 0 deletions
Documentation/trace/ftrace.txt → Documentation/trace/ftrace.rst
View file @ 1f198e22
ftrace - Function Tracer ========================
======================== ftrace - Function Tracer
========================
Copyright 2008 Red Hat Inc. Copyright 2008 Red Hat Inc.
Author: Steven Rostedt <srostedt@redhat.com>
License: The GNU Free Documentation License, Version 1.2 :Author: Steven Rostedt <srostedt@redhat.com>
:License: The GNU Free Documentation License, Version 1.2
(dual licensed under the GPL v2) (dual licensed under the GPL v2)
Original Reviewers: Elias Oltmanns, Randy Dunlap, Andrew Morton, :Original Reviewers: Elias Oltmanns, Randy Dunlap, Andrew Morton,
John Kacur, and David Teigland. John Kacur, and David Teigland.
Written for: 2.6.28-rc2
Updated for: 3.10 - Written for: 2.6.28-rc2
Updated for: 4.13 - Copyright 2017 VMware Inc. Steven Rostedt - Updated for: 3.10
- Updated for: 4.13 - Copyright 2017 VMware Inc. Steven Rostedt
- Converted to rst format - Changbin Du <changbin.du@intel.com>
Introduction Introduction
------------ ------------
...@@ -36,7 +40,7 @@ See events.txt for more information. ...@@ -36,7 +40,7 @@ See events.txt for more information.
Implementation Details Implementation Details
---------------------- ----------------------
See ftrace-design.txt for details for arch porters and such. See :doc:`ftrace-design` for details for arch porters and such.
The File System The File System
...@@ -47,38 +51,38 @@ well as the files to display output. ...@@ -47,38 +51,38 @@ well as the files to display output.
When tracefs is configured into the kernel (which selecting any ftrace When tracefs is configured into the kernel (which selecting any ftrace
option will do) the directory /sys/kernel/tracing will be created. To mount option will do) the directory /sys/kernel/tracing will be created. To mount
this directory, you can add to your /etc/fstab file: this directory, you can add to your /etc/fstab file::
tracefs /sys/kernel/tracing tracefs defaults 0 0 tracefs /sys/kernel/tracing tracefs defaults 0 0
Or you can mount it at run time with: Or you can mount it at run time with::
mount -t tracefs nodev /sys/kernel/tracing mount -t tracefs nodev /sys/kernel/tracing
For quicker access to that directory you may want to make a soft link to For quicker access to that directory you may want to make a soft link to
it: it::
ln -s /sys/kernel/tracing /tracing ln -s /sys/kernel/tracing /tracing
*** NOTICE *** .. attention::
Before 4.1, all ftrace tracing control files were within the debugfs Before 4.1, all ftrace tracing control files were within the debugfs
file system, which is typically located at /sys/kernel/debug/tracing. file system, which is typically located at /sys/kernel/debug/tracing.
For backward compatibility, when mounting the debugfs file system, For backward compatibility, when mounting the debugfs file system,
the tracefs file system will be automatically mounted at: the tracefs file system will be automatically mounted at:
/sys/kernel/debug/tracing /sys/kernel/debug/tracing
All files located in the tracefs file system will be located in that All files located in the tracefs file system will be located in that
debugfs file system directory as well. debugfs file system directory as well.
*** NOTICE *** .. attention::
Any selected ftrace option will also create the tracefs file system. Any selected ftrace option will also create the tracefs file system.
The rest of the document will assume that you are in the ftrace directory The rest of the document will assume that you are in the ftrace directory
(cd /sys/kernel/tracing) and will only concentrate on the files within that (cd /sys/kernel/tracing) and will only concentrate on the files within that
directory and not distract from the content with the extended directory and not distract from the content with the extended
"/sys/kernel/tracing" path name. "/sys/kernel/tracing" path name.
That's it! (assuming that you have ftrace configured into your kernel) That's it! (assuming that you have ftrace configured into your kernel)
...@@ -407,41 +411,48 @@ of ftrace. Here is a list of some of the key files: ...@@ -407,41 +411,48 @@ of ftrace. Here is a list of some of the key files:
CPUs. In other words, the local clocks may not be in sync CPUs. In other words, the local clocks may not be in sync
with local clocks on other CPUs. with local clocks on other CPUs.
Usual clocks for tracing: Usual clocks for tracing::
# cat trace_clock # cat trace_clock
[local] global counter x86-tsc [local] global counter x86-tsc
The clock with the square brackets around it is the one The clock with the square brackets around it is the one in effect.
in effect.
local: Default clock, but may not be in sync across CPUs local:
Default clock, but may not be in sync across CPUs
global: This clock is in sync with all CPUs but may global:
This clock is in sync with all CPUs but may
be a bit slower than the local clock. be a bit slower than the local clock.
counter: This is not a clock at all, but literally an atomic counter:
This is not a clock at all, but literally an atomic
counter. It counts up one by one, but is in sync counter. It counts up one by one, but is in sync
with all CPUs. This is useful when you need to with all CPUs. This is useful when you need to
know exactly the order events occurred with respect to know exactly the order events occurred with respect to
each other on different CPUs. each other on different CPUs.
uptime: This uses the jiffies counter and the time stamp uptime:
This uses the jiffies counter and the time stamp
is relative to the time since boot up. is relative to the time since boot up.
perf: This makes ftrace use the same clock that perf uses. perf:
This makes ftrace use the same clock that perf uses.
Eventually perf will be able to read ftrace buffers Eventually perf will be able to read ftrace buffers
and this will help out in interleaving the data. and this will help out in interleaving the data.
x86-tsc: Architectures may define their own clocks. For x86-tsc:
Architectures may define their own clocks. For
example, x86 uses its own TSC cycle clock here. example, x86 uses its own TSC cycle clock here.
ppc-tb: This uses the powerpc timebase register value. ppc-tb:
This uses the powerpc timebase register value.
This is in sync across CPUs and can also be used This is in sync across CPUs and can also be used
to correlate events across hypervisor/guest if to correlate events across hypervisor/guest if
tb_offset is known. tb_offset is known.
mono: This uses the fast monotonic clock (CLOCK_MONOTONIC) mono:
This uses the fast monotonic clock (CLOCK_MONOTONIC)
which is monotonic and is subject to NTP rate adjustments. which is monotonic and is subject to NTP rate adjustments.
mono_raw: mono_raw:
...@@ -449,7 +460,8 @@ of ftrace. Here is a list of some of the key files: ...@@ -449,7 +460,8 @@ of ftrace. Here is a list of some of the key files:
which is montonic but is not subject to any rate adjustments which is montonic but is not subject to any rate adjustments
and ticks at the same rate as the hardware clocksource. and ticks at the same rate as the hardware clocksource.
boot: This is the boot clock (CLOCK_BOOTTIME) and is based on the boot:
This is the boot clock (CLOCK_BOOTTIME) and is based on the
fast monotonic clock, but also accounts for time spent in fast monotonic clock, but also accounts for time spent in
suspend. Since the clock access is designed for use in suspend. Since the clock access is designed for use in
tracing in the suspend path, some side effects are possible tracing in the suspend path, some side effects are possible
...@@ -461,7 +473,7 @@ of ftrace. Here is a list of some of the key files: ...@@ -461,7 +473,7 @@ of ftrace. Here is a list of some of the key files:
processing should be able to handle them. See comments in the processing should be able to handle them. See comments in the
ktime_get_boot_fast_ns() function for more information. ktime_get_boot_fast_ns() function for more information.
To set a clock, simply echo the clock name into this file. To set a clock, simply echo the clock name into this file::
echo global > trace_clock echo global > trace_clock
...@@ -473,7 +485,7 @@ of ftrace. Here is a list of some of the key files: ...@@ -473,7 +485,7 @@ of ftrace. Here is a list of some of the key files:
It is useful in applications to open this file at the start It is useful in applications to open this file at the start
of the application and just reference the file descriptor of the application and just reference the file descriptor
for the file. for the file::
void trace_write(const char *fmt, ...) void trace_write(const char *fmt, ...)
{ {
...@@ -491,7 +503,7 @@ of ftrace. Here is a list of some of the key files: ...@@ -491,7 +503,7 @@ of ftrace. Here is a list of some of the key files:
write(trace_fd, buf, n); write(trace_fd, buf, n);
} }
start: start::
trace_fd = open("trace_marker", WR_ONLY); trace_fd = open("trace_marker", WR_ONLY);
...@@ -597,25 +609,33 @@ of ftrace. Here is a list of some of the key files: ...@@ -597,25 +609,33 @@ of ftrace. Here is a list of some of the key files:
This displays certain stats about the ring buffer: This displays certain stats about the ring buffer:
entries: The number of events that are still in the buffer. entries:
The number of events that are still in the buffer.
overrun: The number of lost events due to overwriting when overrun:
The number of lost events due to overwriting when
the buffer was full. the buffer was full.
commit overrun: Should always be zero. commit overrun:
Should always be zero.
This gets set if so many events happened within a nested This gets set if so many events happened within a nested
event (ring buffer is re-entrant), that it fills the event (ring buffer is re-entrant), that it fills the
buffer and starts dropping events. buffer and starts dropping events.
bytes: Bytes actually read (not overwritten). bytes:
Bytes actually read (not overwritten).
oldest event ts: The oldest timestamp in the buffer oldest event ts:
The oldest timestamp in the buffer
now ts: The current timestamp now ts:
The current timestamp
dropped events: Events lost due to overwrite option being off. dropped events:
Events lost due to overwrite option being off.
read events: The number of events read. read events:
The number of events read.
The Tracers The Tracers
----------- -----------
...@@ -716,20 +736,19 @@ user-land utilities). ...@@ -716,20 +736,19 @@ user-land utilities).
Output format: Output format:
-------------- --------------
Here is an example of the output format of the file "trace" Here is an example of the output format of the file "trace"::
-------- # tracer: function
# tracer: function #
# # entries-in-buffer/entries-written: 140080/250280 #P:4
# entries-in-buffer/entries-written: 140080/250280 #P:4 #
# # _-----=> irqs-off
# _-----=> irqs-off # / _----=> need-resched
# / _----=> need-resched # | / _---=> hardirq/softirq
# | / _---=> hardirq/softirq # || / _--=> preempt-depth
# || / _--=> preempt-depth # ||| / delay
# ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | |
# | | | |||| | |
bash-1977 [000] .... 17284.993652: sys_close <-system_call_fastpath bash-1977 [000] .... 17284.993652: sys_close <-system_call_fastpath
bash-1977 [000] .... 17284.993653: __close_fd <-sys_close bash-1977 [000] .... 17284.993653: __close_fd <-sys_close
bash-1977 [000] .... 17284.993653: _raw_spin_lock <-__close_fd bash-1977 [000] .... 17284.993653: _raw_spin_lock <-__close_fd
...@@ -740,7 +759,7 @@ Here is an example of the output format of the file "trace" ...@@ -740,7 +759,7 @@ Here is an example of the output format of the file "trace"
bash-1977 [000] .... 17284.993657: filp_close <-__close_fd bash-1977 [000] .... 17284.993657: filp_close <-__close_fd
bash-1977 [000] .... 17284.993657: dnotify_flush <-filp_close bash-1977 [000] .... 17284.993657: dnotify_flush <-filp_close
sshd-1974 [003] .... 17284.993658: sys_select <-system_call_fastpath sshd-1974 [003] .... 17284.993658: sys_select <-system_call_fastpath
-------- ....
A header is printed with the tracer name that is represented by A header is printed with the tracer name that is represented by
the trace. In this case the tracer is "function". Then it shows the the trace. In this case the tracer is "function". Then it shows the
...@@ -761,28 +780,28 @@ Latency trace format ...@@ -761,28 +780,28 @@ Latency trace format
When the latency-format option is enabled or when one of the latency When the latency-format option is enabled or when one of the latency
tracers is set, the trace file gives somewhat more information to see tracers is set, the trace file gives somewhat more information to see
why a latency happened. Here is a typical trace. why a latency happened. Here is a typical trace::
# tracer: irqsoff # tracer: irqsoff
# #
# irqsoff latency trace v1.1.5 on 3.8.0-test+ # irqsoff latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 259 us, #4/4, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 259 us, #4/4, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: ps-6143 (uid:0 nice:0 policy:0 rt_prio:0) # | task: ps-6143 (uid:0 nice:0 policy:0 rt_prio:0)
# ----------------- # -----------------
# => started at: __lock_task_sighand # => started at: __lock_task_sighand
# => ended at: _raw_spin_unlock_irqrestore # => ended at: _raw_spin_unlock_irqrestore
# #
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
ps-6143 2d... 0us!: trace_hardirqs_off <-__lock_task_sighand ps-6143 2d... 0us!: trace_hardirqs_off <-__lock_task_sighand
ps-6143 2d..1 259us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore ps-6143 2d..1 259us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore
ps-6143 2d..1 263us+: time_hardirqs_on <-_raw_spin_unlock_irqrestore ps-6143 2d..1 263us+: time_hardirqs_on <-_raw_spin_unlock_irqrestore
...@@ -813,8 +832,8 @@ occurred. (ps pid: 6143). ...@@ -813,8 +832,8 @@ occurred. (ps pid: 6143).
The start and stop (the functions in which the interrupts were The start and stop (the functions in which the interrupts were
disabled and enabled respectively) that caused the latencies: disabled and enabled respectively) that caused the latencies:
__lock_task_sighand is where the interrupts were disabled. - __lock_task_sighand is where the interrupts were disabled.
_raw_spin_unlock_irqrestore is where they were enabled again. - _raw_spin_unlock_irqrestore is where they were enabled again.
The next lines after the header are the trace itself. The header The next lines after the header are the trace itself. The header
explains which is which. explains which is which.
...@@ -826,44 +845,47 @@ explains which is which. ...@@ -826,44 +845,47 @@ explains which is which.
CPU#: The CPU which the process was running on. CPU#: The CPU which the process was running on.
irqs-off: 'd' interrupts are disabled. '.' otherwise. irqs-off: 'd' interrupts are disabled. '.' otherwise.
Note: If the architecture does not support a way to .. caution:: If the architecture does not support a way to
read the irq flags variable, an 'X' will always read the irq flags variable, an 'X' will always
be printed here. be printed here.
need-resched: need-resched:
'N' both TIF_NEED_RESCHED and PREEMPT_NEED_RESCHED is set, - 'N' both TIF_NEED_RESCHED and PREEMPT_NEED_RESCHED is set,
'n' only TIF_NEED_RESCHED is set, - 'n' only TIF_NEED_RESCHED is set,
'p' only PREEMPT_NEED_RESCHED is set, - 'p' only PREEMPT_NEED_RESCHED is set,
'.' otherwise. - '.' otherwise.
hardirq/softirq: hardirq/softirq:
'Z' - NMI occurred inside a hardirq - 'Z' - NMI occurred inside a hardirq
'z' - NMI is running - 'z' - NMI is running
'H' - hard irq occurred inside a softirq. - 'H' - hard irq occurred inside a softirq.
'h' - hard irq is running - 'h' - hard irq is running
's' - soft irq is running - 's' - soft irq is running
'.' - normal context. - '.' - normal context.
preempt-depth: The level of preempt_disabled preempt-depth: The level of preempt_disabled
The above is mostly meaningful for kernel developers. The above is mostly meaningful for kernel developers.
time: When the latency-format option is enabled, the trace file time:
When the latency-format option is enabled, the trace file
output includes a timestamp relative to the start of the output includes a timestamp relative to the start of the
trace. This differs from the output when latency-format trace. This differs from the output when latency-format
is disabled, which includes an absolute timestamp. is disabled, which includes an absolute timestamp.
delay: This is just to help catch your eye a bit better. And delay:
This is just to help catch your eye a bit better. And
needs to be fixed to be only relative to the same CPU. needs to be fixed to be only relative to the same CPU.
The marks are determined by the difference between this The marks are determined by the difference between this
current trace and the next trace. current trace and the next trace.
'$' - greater than 1 second
'@' - greater than 100 milisecond - '$' - greater than 1 second
'*' - greater than 10 milisecond - '@' - greater than 100 milisecond
'#' - greater than 1000 microsecond - '*' - greater than 10 milisecond
'!' - greater than 100 microsecond - '#' - greater than 1000 microsecond
'+' - greater than 10 microsecond - '!' - greater than 100 microsecond
' ' - less than or equal to 10 microsecond. - '+' - greater than 10 microsecond
- ' ' - less than or equal to 10 microsecond.
The rest is the same as the 'trace' file. The rest is the same as the 'trace' file.
...@@ -875,50 +897,52 @@ trace_options ...@@ -875,50 +897,52 @@ trace_options
The trace_options file (or the options directory) is used to control The trace_options file (or the options directory) is used to control
what gets printed in the trace output, or manipulate the tracers. what gets printed in the trace output, or manipulate the tracers.
To see what is available, simply cat the file: To see what is available, simply cat the file::
cat trace_options cat trace_options
print-parent print-parent
nosym-offset nosym-offset
nosym-addr nosym-addr
noverbose noverbose
noraw noraw
nohex nohex
nobin nobin
noblock noblock
trace_printk trace_printk
annotate annotate
nouserstacktrace nouserstacktrace
nosym-userobj nosym-userobj
noprintk-msg-only noprintk-msg-only
context-info context-info
nolatency-format nolatency-format
record-cmd record-cmd
norecord-tgid norecord-tgid
overwrite overwrite
nodisable_on_free nodisable_on_free
irq-info irq-info
markers markers
noevent-fork noevent-fork
function-trace function-trace
nofunction-fork nofunction-fork
nodisplay-graph nodisplay-graph
nostacktrace nostacktrace
nobranch nobranch
To disable one of the options, echo in the option prepended with To disable one of the options, echo in the option prepended with
"no". "no"::
echo noprint-parent > trace_options echo noprint-parent > trace_options
To enable an option, leave off the "no". To enable an option, leave off the "no"::
echo sym-offset > trace_options echo sym-offset > trace_options
Here are the available options: Here are the available options:
print-parent - On function traces, display the calling (parent) print-parent
On function traces, display the calling (parent)
function as well as the function being traced. function as well as the function being traced.
::
print-parent: print-parent:
bash-4000 [01] 1477.606694: simple_strtoul <-kstrtoul bash-4000 [01] 1477.606694: simple_strtoul <-kstrtoul
...@@ -927,61 +951,74 @@ Here are the available options: ...@@ -927,61 +951,74 @@ Here are the available options:
bash-4000 [01] 1477.606694: simple_strtoul bash-4000 [01] 1477.606694: simple_strtoul
sym-offset - Display not only the function name, but also the sym-offset
Display not only the function name, but also the
offset in the function. For example, instead of offset in the function. For example, instead of
seeing just "ktime_get", you will see seeing just "ktime_get", you will see
"ktime_get+0xb/0x20". "ktime_get+0xb/0x20".
::
sym-offset: sym-offset:
bash-4000 [01] 1477.606694: simple_strtoul+0x6/0xa0 bash-4000 [01] 1477.606694: simple_strtoul+0x6/0xa0
sym-addr - this will also display the function address as well sym-addr
This will also display the function address as well
as the function name. as the function name.
::
sym-addr: sym-addr:
bash-4000 [01] 1477.606694: simple_strtoul <c0339346> bash-4000 [01] 1477.606694: simple_strtoul <c0339346>
verbose - This deals with the trace file when the verbose
This deals with the trace file when the
latency-format option is enabled. latency-format option is enabled.
::
bash 4000 1 0 00000000 00010a95 [58127d26] 1720.415ms \ bash 4000 1 0 00000000 00010a95 [58127d26] 1720.415ms \
(+0.000ms): simple_strtoul (kstrtoul) (+0.000ms): simple_strtoul (kstrtoul)
raw - This will display raw numbers. This option is best for raw
This will display raw numbers. This option is best for
use with user applications that can translate the raw use with user applications that can translate the raw
numbers better than having it done in the kernel. numbers better than having it done in the kernel.
hex - Similar to raw, but the numbers will be in a hexadecimal hex
format. Similar to raw, but the numbers will be in a hexadecimal format.
bin - This will print out the formats in raw binary. bin
This will print out the formats in raw binary.
block - When set, reading trace_pipe will not block when polled. block
When set, reading trace_pipe will not block when polled.
trace_printk - Can disable trace_printk() from writing into the buffer. trace_printk
Can disable trace_printk() from writing into the buffer.
annotate - It is sometimes confusing when the CPU buffers are full annotate
It is sometimes confusing when the CPU buffers are full
and one CPU buffer had a lot of events recently, thus and one CPU buffer had a lot of events recently, thus
a shorter time frame, were another CPU may have only had a shorter time frame, were another CPU may have only had
a few events, which lets it have older events. When a few events, which lets it have older events. When
the trace is reported, it shows the oldest events first, the trace is reported, it shows the oldest events first,
and it may look like only one CPU ran (the one with the and it may look like only one CPU ran (the one with the
oldest events). When the annotate option is set, it will oldest events). When the annotate option is set, it will
display when a new CPU buffer started: display when a new CPU buffer started::
<idle>-0 [001] dNs4 21169.031481: wake_up_idle_cpu <-add_timer_on <idle>-0 [001] dNs4 21169.031481: wake_up_idle_cpu <-add_timer_on
<idle>-0 [001] dNs4 21169.031482: _raw_spin_unlock_irqrestore <-add_timer_on <idle>-0 [001] dNs4 21169.031482: _raw_spin_unlock_irqrestore <-add_timer_on
<idle>-0 [001] .Ns4 21169.031484: sub_preempt_count <-_raw_spin_unlock_irqrestore <idle>-0 [001] .Ns4 21169.031484: sub_preempt_count <-_raw_spin_unlock_irqrestore
##### CPU 2 buffer started #### ##### CPU 2 buffer started ####
<idle>-0 [002] .N.1 21169.031484: rcu_idle_exit <-cpu_idle <idle>-0 [002] .N.1 21169.031484: rcu_idle_exit <-cpu_idle
<idle>-0 [001] .Ns3 21169.031484: _raw_spin_unlock <-clocksource_watchdog <idle>-0 [001] .Ns3 21169.031484: _raw_spin_unlock <-clocksource_watchdog
<idle>-0 [001] .Ns3 21169.031485: sub_preempt_count <-_raw_spin_unlock <idle>-0 [001] .Ns3 21169.031485: sub_preempt_count <-_raw_spin_unlock
userstacktrace - This option changes the trace. It records a userstacktrace
This option changes the trace. It records a
stacktrace of the current user space thread after stacktrace of the current user space thread after
each trace event. each trace event.
sym-userobj - when user stacktrace are enabled, look up which sym-userobj
when user stacktrace are enabled, look up which
object the address belongs to, and print a object the address belongs to, and print a
relative address. This is especially useful when relative address. This is especially useful when
ASLR is on, otherwise you don't get a chance to ASLR is on, otherwise you don't get a chance to
...@@ -989,91 +1026,106 @@ Here are the available options: ...@@ -989,91 +1026,106 @@ Here are the available options:
the app is no longer running the app is no longer running
The lookup is performed when you read The lookup is performed when you read
trace,trace_pipe. Example: trace,trace_pipe. Example::
a.out-1623 [000] 40874.465068: /root/a.out[+0x480] <-/root/a.out[+0 a.out-1623 [000] 40874.465068: /root/a.out[+0x480] <-/root/a.out[+0
x494] <- /root/a.out[+0x4a8] <- /lib/libc-2.7.so[+0x1e1a6] x494] <- /root/a.out[+0x4a8] <- /lib/libc-2.7.so[+0x1e1a6]
printk-msg-only - When set, trace_printk()s will only show the format printk-msg-only
When set, trace_printk()s will only show the format
and not their parameters (if trace_bprintk() or and not their parameters (if trace_bprintk() or
trace_bputs() was used to save the trace_printk()). trace_bputs() was used to save the trace_printk()).
context-info - Show only the event data. Hides the comm, PID, context-info
Show only the event data. Hides the comm, PID,
timestamp, CPU, and other useful data. timestamp, CPU, and other useful data.
latency-format - This option changes the trace output. When it is enabled, latency-format
This option changes the trace output. When it is enabled,
the trace displays additional information about the the trace displays additional information about the
latency, as described in "Latency trace format". latency, as described in "Latency trace format".
record-cmd - When any event or tracer is enabled, a hook is enabled record-cmd
When any event or tracer is enabled, a hook is enabled
in the sched_switch trace point to fill comm cache in the sched_switch trace point to fill comm cache
with mapped pids and comms. But this may cause some with mapped pids and comms. But this may cause some
overhead, and if you only care about pids, and not the overhead, and if you only care about pids, and not the
name of the task, disabling this option can lower the name of the task, disabling this option can lower the
impact of tracing. See "saved_cmdlines". impact of tracing. See "saved_cmdlines".
record-tgid - When any event or tracer is enabled, a hook is enabled record-tgid
When any event or tracer is enabled, a hook is enabled
in the sched_switch trace point to fill the cache of in the sched_switch trace point to fill the cache of
mapped Thread Group IDs (TGID) mapping to pids. See mapped Thread Group IDs (TGID) mapping to pids. See
"saved_tgids". "saved_tgids".
overwrite - This controls what happens when the trace buffer is overwrite
This controls what happens when the trace buffer is
full. If "1" (default), the oldest events are full. If "1" (default), the oldest events are
discarded and overwritten. If "0", then the newest discarded and overwritten. If "0", then the newest
events are discarded. events are discarded.
(see per_cpu/cpu0/stats for overrun and dropped) (see per_cpu/cpu0/stats for overrun and dropped)
disable_on_free - When the free_buffer is closed, tracing will disable_on_free
When the free_buffer is closed, tracing will
stop (tracing_on set to 0). stop (tracing_on set to 0).
irq-info - Shows the interrupt, preempt count, need resched data. irq-info
When disabled, the trace looks like: Shows the interrupt, preempt count, need resched data.
When disabled, the trace looks like::
# tracer: function # tracer: function
# #
# entries-in-buffer/entries-written: 144405/9452052 #P:4 # entries-in-buffer/entries-written: 144405/9452052 #P:4
# #
# TASK-PID CPU# TIMESTAMP FUNCTION # TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | | # | | | | |
<idle>-0 [002] 23636.756054: ttwu_do_activate.constprop.89 <-try_to_wake_up <idle>-0 [002] 23636.756054: ttwu_do_activate.constprop.89 <-try_to_wake_up
<idle>-0 [002] 23636.756054: activate_task <-ttwu_do_activate.constprop.89 <idle>-0 [002] 23636.756054: activate_task <-ttwu_do_activate.constprop.89
<idle>-0 [002] 23636.756055: enqueue_task <-activate_task <idle>-0 [002] 23636.756055: enqueue_task <-activate_task
markers - When set, the trace_marker is writable (only by root). markers
When set, the trace_marker is writable (only by root).
When disabled, the trace_marker will error with EINVAL When disabled, the trace_marker will error with EINVAL
on write. on write.
event-fork - When set, tasks with PIDs listed in set_event_pid will have event-fork
When set, tasks with PIDs listed in set_event_pid will have
the PIDs of their children added to set_event_pid when those the PIDs of their children added to set_event_pid when those
tasks fork. Also, when tasks with PIDs in set_event_pid exit, tasks fork. Also, when tasks with PIDs in set_event_pid exit,
their PIDs will be removed from the file. their PIDs will be removed from the file.
function-trace - The latency tracers will enable function tracing function-trace
The latency tracers will enable function tracing
if this option is enabled (default it is). When if this option is enabled (default it is). When
it is disabled, the latency tracers do not trace it is disabled, the latency tracers do not trace
functions. This keeps the overhead of the tracer down functions. This keeps the overhead of the tracer down
when performing latency tests. when performing latency tests.
function-fork - When set, tasks with PIDs listed in set_ftrace_pid will function-fork
When set, tasks with PIDs listed in set_ftrace_pid will
have the PIDs of their children added to set_ftrace_pid have the PIDs of their children added to set_ftrace_pid
when those tasks fork. Also, when tasks with PIDs in when those tasks fork. Also, when tasks with PIDs in
set_ftrace_pid exit, their PIDs will be removed from the set_ftrace_pid exit, their PIDs will be removed from the
file. file.
display-graph - When set, the latency tracers (irqsoff, wakeup, etc) will display-graph
When set, the latency tracers (irqsoff, wakeup, etc) will
use function graph tracing instead of function tracing. use function graph tracing instead of function tracing.
stacktrace - When set, a stack trace is recorded after any trace event stacktrace
When set, a stack trace is recorded after any trace event
is recorded. is recorded.
branch - Enable branch tracing with the tracer. This enables branch branch
Enable branch tracing with the tracer. This enables branch
tracer along with the currently set tracer. Enabling this tracer along with the currently set tracer. Enabling this
with the "nop" tracer is the same as just enabling the with the "nop" tracer is the same as just enabling the
"branch" tracer. "branch" tracer.
Note: Some tracers have their own options. They only appear in this .. tip:: Some tracers have their own options. They only appear in this
file when the tracer is active. They always appear in the file when the tracer is active. They always appear in the
options directory. options directory.
...@@ -1082,7 +1134,8 @@ Here are the per tracer options: ...@@ -1082,7 +1134,8 @@ Here are the per tracer options:
Options for function tracer: Options for function tracer:
func_stack_trace - When set, a stack trace is recorded after every func_stack_trace
When set, a stack trace is recorded after every
function that is recorded. NOTE! Limit the functions function that is recorded. NOTE! Limit the functions
that are recorded before enabling this, with that are recorded before enabling this, with
"set_ftrace_filter" otherwise the system performance "set_ftrace_filter" otherwise the system performance
...@@ -1094,7 +1147,8 @@ Options for function_graph tracer: ...@@ -1094,7 +1147,8 @@ Options for function_graph tracer:
Since the function_graph tracer has a slightly different output Since the function_graph tracer has a slightly different output
it has its own options to control what is displayed. it has its own options to control what is displayed.
funcgraph-overrun - When set, the "overrun" of the graph stack is funcgraph-overrun
When set, the "overrun" of the graph stack is
displayed after each function traced. The displayed after each function traced. The
overrun, is when the stack depth of the calls overrun, is when the stack depth of the calls
is greater than what is reserved for each task. is greater than what is reserved for each task.
...@@ -1104,41 +1158,49 @@ Options for function_graph tracer: ...@@ -1104,41 +1158,49 @@ Options for function_graph tracer:
The overrun is the number of functions missed The overrun is the number of functions missed
due to exceeding this array. due to exceeding this array.
funcgraph-cpu - When set, the CPU number of the CPU where the trace funcgraph-cpu
When set, the CPU number of the CPU where the trace
occurred is displayed. occurred is displayed.
funcgraph-overhead - When set, if the function takes longer than funcgraph-overhead
When set, if the function takes longer than
A certain amount, then a delay marker is A certain amount, then a delay marker is
displayed. See "delay" above, under the displayed. See "delay" above, under the
header description. header description.
funcgraph-proc - Unlike other tracers, the process' command line funcgraph-proc
Unlike other tracers, the process' command line
is not displayed by default, but instead only is not displayed by default, but instead only
when a task is traced in and out during a context when a task is traced in and out during a context
switch. Enabling this options has the command switch. Enabling this options has the command
of each process displayed at every line. of each process displayed at every line.
funcgraph-duration - At the end of each function (the return) funcgraph-duration
At the end of each function (the return)
the duration of the amount of time in the the duration of the amount of time in the
function is displayed in microseconds. function is displayed in microseconds.
funcgraph-abstime - When set, the timestamp is displayed at each funcgraph-abstime
line. When set, the timestamp is displayed at each line.
funcgraph-irqs - When disabled, functions that happen inside an funcgraph-irqs
When disabled, functions that happen inside an
interrupt will not be traced. interrupt will not be traced.
funcgraph-tail - When set, the return event will include the function funcgraph-tail
When set, the return event will include the function
that it represents. By default this is off, and that it represents. By default this is off, and
only a closing curly bracket "}" is displayed for only a closing curly bracket "}" is displayed for
the return of a function. the return of a function.
sleep-time - When running function graph tracer, to include sleep-time
When running function graph tracer, to include
the time a task schedules out in its function. the time a task schedules out in its function.
When enabled, it will account time the task has been When enabled, it will account time the task has been
scheduled out as part of the function call. scheduled out as part of the function call.
graph-time - When running function profiler with function graph tracer, graph-time
When running function profiler with function graph tracer,
to include the time to call nested functions. When this is to include the time to call nested functions. When this is
not set, the time reported for the function will only not set, the time reported for the function will only
include the time the function itself executed for, not the include the time the function itself executed for, not the
...@@ -1146,7 +1208,8 @@ Options for function_graph tracer: ...@@ -1146,7 +1208,8 @@ Options for function_graph tracer:
Options for blk tracer: Options for blk tracer:
blk_classic - Shows a more minimalistic output. blk_classic
Shows a more minimalistic output.
irqsoff irqsoff
...@@ -1165,7 +1228,7 @@ new maximum is reached, the old saved trace is discarded and the ...@@ -1165,7 +1228,7 @@ new maximum is reached, the old saved trace is discarded and the
new trace is saved. new trace is saved.
To reset the maximum, echo 0 into tracing_max_latency. Here is To reset the maximum, echo 0 into tracing_max_latency. Here is
an example: an example::
# echo 0 > options/function-trace # echo 0 > options/function-trace
# echo irqsoff > current_tracer # echo irqsoff > current_tracer
...@@ -1175,26 +1238,26 @@ an example: ...@@ -1175,26 +1238,26 @@ an example:
[...] [...]
# echo 0 > tracing_on # echo 0 > tracing_on
# cat trace # cat trace
# tracer: irqsoff # tracer: irqsoff
# #
# irqsoff latency trace v1.1.5 on 3.8.0-test+ # irqsoff latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 16 us, #4/4, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 16 us, #4/4, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: swapper/0-0 (uid:0 nice:0 policy:0 rt_prio:0) # | task: swapper/0-0 (uid:0 nice:0 policy:0 rt_prio:0)
# ----------------- # -----------------
# => started at: run_timer_softirq # => started at: run_timer_softirq
# => ended at: run_timer_softirq # => ended at: run_timer_softirq
# #
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
<idle>-0 0d.s2 0us+: _raw_spin_lock_irq <-run_timer_softirq <idle>-0 0d.s2 0us+: _raw_spin_lock_irq <-run_timer_softirq
<idle>-0 0dNs3 17us : _raw_spin_unlock_irq <-run_timer_softirq <idle>-0 0dNs3 17us : _raw_spin_unlock_irq <-run_timer_softirq
<idle>-0 0dNs3 17us+: trace_hardirqs_on <-run_timer_softirq <idle>-0 0dNs3 17us+: trace_hardirqs_on <-run_timer_softirq
...@@ -1222,30 +1285,30 @@ between the time of recording the max latency and the time of ...@@ -1222,30 +1285,30 @@ between the time of recording the max latency and the time of
recording the function that had that latency. recording the function that had that latency.
Note the above example had function-trace not set. If we set Note the above example had function-trace not set. If we set
function-trace, we get a much larger output: function-trace, we get a much larger output::
with echo 1 > options/function-trace with echo 1 > options/function-trace
# tracer: irqsoff # tracer: irqsoff
# #
# irqsoff latency trace v1.1.5 on 3.8.0-test+ # irqsoff latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 71 us, #168/168, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 71 us, #168/168, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: bash-2042 (uid:0 nice:0 policy:0 rt_prio:0) # | task: bash-2042 (uid:0 nice:0 policy:0 rt_prio:0)
# ----------------- # -----------------
# => started at: ata_scsi_queuecmd # => started at: ata_scsi_queuecmd
# => ended at: ata_scsi_queuecmd # => ended at: ata_scsi_queuecmd
# #
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
bash-2042 3d... 0us : _raw_spin_lock_irqsave <-ata_scsi_queuecmd bash-2042 3d... 0us : _raw_spin_lock_irqsave <-ata_scsi_queuecmd
bash-2042 3d... 0us : add_preempt_count <-_raw_spin_lock_irqsave bash-2042 3d... 0us : add_preempt_count <-_raw_spin_lock_irqsave
bash-2042 3d..1 1us : ata_scsi_find_dev <-ata_scsi_queuecmd bash-2042 3d..1 1us : ata_scsi_find_dev <-ata_scsi_queuecmd
...@@ -1255,7 +1318,7 @@ function-trace, we get a much larger output: ...@@ -1255,7 +1318,7 @@ function-trace, we get a much larger output:
bash-2042 3d..1 3us : ata_sg_init <-__ata_scsi_queuecmd bash-2042 3d..1 3us : ata_sg_init <-__ata_scsi_queuecmd
bash-2042 3d..1 4us : ata_scsi_rw_xlat <-__ata_scsi_queuecmd bash-2042 3d..1 4us : ata_scsi_rw_xlat <-__ata_scsi_queuecmd
bash-2042 3d..1 4us : ata_build_rw_tf <-ata_scsi_rw_xlat bash-2042 3d..1 4us : ata_build_rw_tf <-ata_scsi_rw_xlat
[...] [...]
bash-2042 3d..1 67us : delay_tsc <-__delay bash-2042 3d..1 67us : delay_tsc <-__delay
bash-2042 3d..1 67us : add_preempt_count <-delay_tsc bash-2042 3d..1 67us : add_preempt_count <-delay_tsc
bash-2042 3d..2 67us : sub_preempt_count <-delay_tsc bash-2042 3d..2 67us : sub_preempt_count <-delay_tsc
...@@ -1312,6 +1375,7 @@ The preemptoff tracer traces the places that disable preemption. ...@@ -1312,6 +1375,7 @@ The preemptoff tracer traces the places that disable preemption.
Like the irqsoff tracer, it records the maximum latency for Like the irqsoff tracer, it records the maximum latency for
which preemption was disabled. The control of preemptoff tracer which preemption was disabled. The control of preemptoff tracer
is much like the irqsoff tracer. is much like the irqsoff tracer.
::
# echo 0 > options/function-trace # echo 0 > options/function-trace
# echo preemptoff > current_tracer # echo preemptoff > current_tracer
...@@ -1321,26 +1385,26 @@ is much like the irqsoff tracer. ...@@ -1321,26 +1385,26 @@ is much like the irqsoff tracer.
[...] [...]
# echo 0 > tracing_on # echo 0 > tracing_on
# cat trace # cat trace
# tracer: preemptoff # tracer: preemptoff
# #
# preemptoff latency trace v1.1.5 on 3.8.0-test+ # preemptoff latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 46 us, #4/4, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 46 us, #4/4, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: sshd-1991 (uid:0 nice:0 policy:0 rt_prio:0) # | task: sshd-1991 (uid:0 nice:0 policy:0 rt_prio:0)
# ----------------- # -----------------
# => started at: do_IRQ # => started at: do_IRQ
# => ended at: do_IRQ # => ended at: do_IRQ
# #
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
sshd-1991 1d.h. 0us+: irq_enter <-do_IRQ sshd-1991 1d.h. 0us+: irq_enter <-do_IRQ
sshd-1991 1d..1 46us : irq_exit <-do_IRQ sshd-1991 1d..1 46us : irq_exit <-do_IRQ
sshd-1991 1d..1 47us+: trace_preempt_on <-do_IRQ sshd-1991 1d..1 47us+: trace_preempt_on <-do_IRQ
...@@ -1357,33 +1421,34 @@ But we also see that interrupts have been disabled when entering ...@@ -1357,33 +1421,34 @@ But we also see that interrupts have been disabled when entering
the preempt off section and leaving it (the 'd'). We do not know if the preempt off section and leaving it (the 'd'). We do not know if
interrupts were enabled in the mean time or shortly after this interrupts were enabled in the mean time or shortly after this
was over. was over.
::
# tracer: preemptoff # tracer: preemptoff
# #
# preemptoff latency trace v1.1.5 on 3.8.0-test+ # preemptoff latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 83 us, #241/241, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 83 us, #241/241, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: bash-1994 (uid:0 nice:0 policy:0 rt_prio:0) # | task: bash-1994 (uid:0 nice:0 policy:0 rt_prio:0)
# ----------------- # -----------------
# => started at: wake_up_new_task # => started at: wake_up_new_task
# => ended at: task_rq_unlock # => ended at: task_rq_unlock
# #
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
bash-1994 1d..1 0us : _raw_spin_lock_irqsave <-wake_up_new_task bash-1994 1d..1 0us : _raw_spin_lock_irqsave <-wake_up_new_task
bash-1994 1d..1 0us : select_task_rq_fair <-select_task_rq bash-1994 1d..1 0us : select_task_rq_fair <-select_task_rq
bash-1994 1d..1 1us : __rcu_read_lock <-select_task_rq_fair bash-1994 1d..1 1us : __rcu_read_lock <-select_task_rq_fair
bash-1994 1d..1 1us : source_load <-select_task_rq_fair bash-1994 1d..1 1us : source_load <-select_task_rq_fair
bash-1994 1d..1 1us : source_load <-select_task_rq_fair bash-1994 1d..1 1us : source_load <-select_task_rq_fair
[...] [...]
bash-1994 1d..1 12us : irq_enter <-smp_apic_timer_interrupt bash-1994 1d..1 12us : irq_enter <-smp_apic_timer_interrupt
bash-1994 1d..1 12us : rcu_irq_enter <-irq_enter bash-1994 1d..1 12us : rcu_irq_enter <-irq_enter
bash-1994 1d..1 13us : add_preempt_count <-irq_enter bash-1994 1d..1 13us : add_preempt_count <-irq_enter
...@@ -1392,7 +1457,7 @@ was over. ...@@ -1392,7 +1457,7 @@ was over.
bash-1994 1d.h1 13us : _raw_spin_lock <-hrtimer_interrupt bash-1994 1d.h1 13us : _raw_spin_lock <-hrtimer_interrupt
bash-1994 1d.h1 14us : add_preempt_count <-_raw_spin_lock bash-1994 1d.h1 14us : add_preempt_count <-_raw_spin_lock
bash-1994 1d.h2 14us : ktime_get_update_offsets <-hrtimer_interrupt bash-1994 1d.h2 14us : ktime_get_update_offsets <-hrtimer_interrupt
[...] [...]
bash-1994 1d.h1 35us : lapic_next_event <-clockevents_program_event bash-1994 1d.h1 35us : lapic_next_event <-clockevents_program_event
bash-1994 1d.h1 35us : irq_exit <-smp_apic_timer_interrupt bash-1994 1d.h1 35us : irq_exit <-smp_apic_timer_interrupt
bash-1994 1d.h1 36us : sub_preempt_count <-irq_exit bash-1994 1d.h1 36us : sub_preempt_count <-irq_exit
...@@ -1403,7 +1468,7 @@ was over. ...@@ -1403,7 +1468,7 @@ was over.
bash-1994 1d.s3 38us : _raw_spin_unlock <-run_timer_softirq bash-1994 1d.s3 38us : _raw_spin_unlock <-run_timer_softirq
bash-1994 1d.s3 39us : sub_preempt_count <-_raw_spin_unlock bash-1994 1d.s3 39us : sub_preempt_count <-_raw_spin_unlock
bash-1994 1d.s2 39us : call_timer_fn <-run_timer_softirq bash-1994 1d.s2 39us : call_timer_fn <-run_timer_softirq
[...] [...]
bash-1994 1dNs2 81us : cpu_needs_another_gp <-rcu_process_callbacks bash-1994 1dNs2 81us : cpu_needs_another_gp <-rcu_process_callbacks
bash-1994 1dNs2 82us : __local_bh_enable <-__do_softirq bash-1994 1dNs2 82us : __local_bh_enable <-__do_softirq
bash-1994 1dNs2 82us : sub_preempt_count <-__local_bh_enable bash-1994 1dNs2 82us : sub_preempt_count <-__local_bh_enable
...@@ -1437,7 +1502,7 @@ preemption disabled for the longest times is helpful. But ...@@ -1437,7 +1502,7 @@ preemption disabled for the longest times is helpful. But
sometimes we would like to know when either preemption and/or sometimes we would like to know when either preemption and/or
interrupts are disabled. interrupts are disabled.
Consider the following code: Consider the following code::
local_irq_disable(); local_irq_disable();
call_function_with_irqs_off(); call_function_with_irqs_off();
...@@ -1462,6 +1527,7 @@ tracer. ...@@ -1462,6 +1527,7 @@ tracer.
Again, using this trace is much like the irqsoff and preemptoff Again, using this trace is much like the irqsoff and preemptoff
tracers. tracers.
::
# echo 0 > options/function-trace # echo 0 > options/function-trace
# echo preemptirqsoff > current_tracer # echo preemptirqsoff > current_tracer
...@@ -1471,26 +1537,26 @@ tracers. ...@@ -1471,26 +1537,26 @@ tracers.
[...] [...]
# echo 0 > tracing_on # echo 0 > tracing_on
# cat trace # cat trace
# tracer: preemptirqsoff # tracer: preemptirqsoff
# #
# preemptirqsoff latency trace v1.1.5 on 3.8.0-test+ # preemptirqsoff latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 100 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 100 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: ls-2230 (uid:0 nice:0 policy:0 rt_prio:0) # | task: ls-2230 (uid:0 nice:0 policy:0 rt_prio:0)
# ----------------- # -----------------
# => started at: ata_scsi_queuecmd # => started at: ata_scsi_queuecmd
# => ended at: ata_scsi_queuecmd # => ended at: ata_scsi_queuecmd
# #
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
ls-2230 3d... 0us+: _raw_spin_lock_irqsave <-ata_scsi_queuecmd ls-2230 3d... 0us+: _raw_spin_lock_irqsave <-ata_scsi_queuecmd
ls-2230 3...1 100us : _raw_spin_unlock_irqrestore <-ata_scsi_queuecmd ls-2230 3...1 100us : _raw_spin_unlock_irqrestore <-ata_scsi_queuecmd
ls-2230 3...1 101us+: trace_preempt_on <-ata_scsi_queuecmd ls-2230 3...1 101us+: trace_preempt_on <-ata_scsi_queuecmd
...@@ -1522,53 +1588,53 @@ function tracing, we do not know if interrupts were enabled ...@@ -1522,53 +1588,53 @@ function tracing, we do not know if interrupts were enabled
within the preemption points. We do see that it started with within the preemption points. We do see that it started with
preemption enabled. preemption enabled.
Here is a trace with function-trace set: Here is a trace with function-trace set::
# tracer: preemptirqsoff # tracer: preemptirqsoff
# #
# preemptirqsoff latency trace v1.1.5 on 3.8.0-test+ # preemptirqsoff latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 161 us, #339/339, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 161 us, #339/339, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: ls-2269 (uid:0 nice:0 policy:0 rt_prio:0) # | task: ls-2269 (uid:0 nice:0 policy:0 rt_prio:0)
# ----------------- # -----------------
# => started at: schedule # => started at: schedule
# => ended at: mutex_unlock # => ended at: mutex_unlock
# #
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
kworker/-59 3...1 0us : __schedule <-schedule kworker/-59 3...1 0us : __schedule <-schedule
kworker/-59 3d..1 0us : rcu_preempt_qs <-rcu_note_context_switch kworker/-59 3d..1 0us : rcu_preempt_qs <-rcu_note_context_switch
kworker/-59 3d..1 1us : add_preempt_count <-_raw_spin_lock_irq kworker/-59 3d..1 1us : add_preempt_count <-_raw_spin_lock_irq
kworker/-59 3d..2 1us : deactivate_task <-__schedule kworker/-59 3d..2 1us : deactivate_task <-__schedule
kworker/-59 3d..2 1us : dequeue_task <-deactivate_task kworker/-59 3d..2 1us : dequeue_task <-deactivate_task
kworker/-59 3d..2 2us : update_rq_clock <-dequeue_task kworker/-59 3d..2 2us : update_rq_clock <-dequeue_task
kworker/-59 3d..2 2us : dequeue_task_fair <-dequeue_task kworker/-59 3d..2 2us : dequeue_task_fair <-dequeue_task
kworker/-59 3d..2 2us : update_curr <-dequeue_task_fair kworker/-59 3d..2 2us : update_curr <-dequeue_task_fair
kworker/-59 3d..2 2us : update_min_vruntime <-update_curr kworker/-59 3d..2 2us : update_min_vruntime <-update_curr
kworker/-59 3d..2 3us : cpuacct_charge <-update_curr kworker/-59 3d..2 3us : cpuacct_charge <-update_curr
kworker/-59 3d..2 3us : __rcu_read_lock <-cpuacct_charge kworker/-59 3d..2 3us : __rcu_read_lock <-cpuacct_charge
kworker/-59 3d..2 3us : __rcu_read_unlock <-cpuacct_charge kworker/-59 3d..2 3us : __rcu_read_unlock <-cpuacct_charge
kworker/-59 3d..2 3us : update_cfs_rq_blocked_load <-dequeue_task_fair kworker/-59 3d..2 3us : update_cfs_rq_blocked_load <-dequeue_task_fair
kworker/-59 3d..2 4us : clear_buddies <-dequeue_task_fair kworker/-59 3d..2 4us : clear_buddies <-dequeue_task_fair
kworker/-59 3d..2 4us : account_entity_dequeue <-dequeue_task_fair kworker/-59 3d..2 4us : account_entity_dequeue <-dequeue_task_fair
kworker/-59 3d..2 4us : update_min_vruntime <-dequeue_task_fair kworker/-59 3d..2 4us : update_min_vruntime <-dequeue_task_fair
kworker/-59 3d..2 4us : update_cfs_shares <-dequeue_task_fair kworker/-59 3d..2 4us : update_cfs_shares <-dequeue_task_fair
kworker/-59 3d..2 5us : hrtick_update <-dequeue_task_fair kworker/-59 3d..2 5us : hrtick_update <-dequeue_task_fair
kworker/-59 3d..2 5us : wq_worker_sleeping <-__schedule kworker/-59 3d..2 5us : wq_worker_sleeping <-__schedule
kworker/-59 3d..2 5us : kthread_data <-wq_worker_sleeping kworker/-59 3d..2 5us : kthread_data <-wq_worker_sleeping
kworker/-59 3d..2 5us : put_prev_task_fair <-__schedule kworker/-59 3d..2 5us : put_prev_task_fair <-__schedule
kworker/-59 3d..2 6us : pick_next_task_fair <-pick_next_task kworker/-59 3d..2 6us : pick_next_task_fair <-pick_next_task
kworker/-59 3d..2 6us : clear_buddies <-pick_next_task_fair kworker/-59 3d..2 6us : clear_buddies <-pick_next_task_fair
kworker/-59 3d..2 6us : set_next_entity <-pick_next_task_fair kworker/-59 3d..2 6us : set_next_entity <-pick_next_task_fair
kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity
ls-2269 3d..2 7us : finish_task_switch <-__schedule ls-2269 3d..2 7us : finish_task_switch <-__schedule
ls-2269 3d..2 7us : _raw_spin_unlock_irq <-finish_task_switch ls-2269 3d..2 7us : _raw_spin_unlock_irq <-finish_task_switch
ls-2269 3d..2 8us : do_IRQ <-ret_from_intr ls-2269 3d..2 8us : do_IRQ <-ret_from_intr
...@@ -1576,7 +1642,7 @@ kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity ...@@ -1576,7 +1642,7 @@ kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity
ls-2269 3d..2 8us : rcu_irq_enter <-irq_enter ls-2269 3d..2 8us : rcu_irq_enter <-irq_enter
ls-2269 3d..2 9us : add_preempt_count <-irq_enter ls-2269 3d..2 9us : add_preempt_count <-irq_enter
ls-2269 3d.h2 9us : exit_idle <-do_IRQ ls-2269 3d.h2 9us : exit_idle <-do_IRQ
[...] [...]
ls-2269 3d.h3 20us : sub_preempt_count <-_raw_spin_unlock ls-2269 3d.h3 20us : sub_preempt_count <-_raw_spin_unlock
ls-2269 3d.h2 20us : irq_exit <-do_IRQ ls-2269 3d.h2 20us : irq_exit <-do_IRQ
ls-2269 3d.h2 21us : sub_preempt_count <-irq_exit ls-2269 3d.h2 21us : sub_preempt_count <-irq_exit
...@@ -1588,14 +1654,14 @@ kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity ...@@ -1588,14 +1654,14 @@ kworker/-59 3d..2 6us : update_stats_wait_end <-set_next_entity
ls-2269 3d.s5 31us : do_IRQ <-ret_from_intr ls-2269 3d.s5 31us : do_IRQ <-ret_from_intr
ls-2269 3d.s5 31us : irq_enter <-do_IRQ ls-2269 3d.s5 31us : irq_enter <-do_IRQ
ls-2269 3d.s5 31us : rcu_irq_enter <-irq_enter ls-2269 3d.s5 31us : rcu_irq_enter <-irq_enter
[...] [...]
ls-2269 3d.s5 31us : rcu_irq_enter <-irq_enter ls-2269 3d.s5 31us : rcu_irq_enter <-irq_enter
ls-2269 3d.s5 32us : add_preempt_count <-irq_enter ls-2269 3d.s5 32us : add_preempt_count <-irq_enter
ls-2269 3d.H5 32us : exit_idle <-do_IRQ ls-2269 3d.H5 32us : exit_idle <-do_IRQ
ls-2269 3d.H5 32us : handle_irq <-do_IRQ ls-2269 3d.H5 32us : handle_irq <-do_IRQ
ls-2269 3d.H5 32us : irq_to_desc <-handle_irq ls-2269 3d.H5 32us : irq_to_desc <-handle_irq
ls-2269 3d.H5 33us : handle_fasteoi_irq <-handle_irq ls-2269 3d.H5 33us : handle_fasteoi_irq <-handle_irq
[...] [...]
ls-2269 3d.s5 158us : _raw_spin_unlock_irqrestore <-rtl8139_poll ls-2269 3d.s5 158us : _raw_spin_unlock_irqrestore <-rtl8139_poll
ls-2269 3d.s3 158us : net_rps_action_and_irq_enable.isra.65 <-net_rx_action ls-2269 3d.s3 158us : net_rps_action_and_irq_enable.isra.65 <-net_rx_action
ls-2269 3d.s3 159us : __local_bh_enable <-__do_softirq ls-2269 3d.s3 159us : __local_bh_enable <-__do_softirq
...@@ -1631,7 +1697,7 @@ time it takes for a task that is woken to actually wake up. ...@@ -1631,7 +1697,7 @@ time it takes for a task that is woken to actually wake up.
Now for non Real-Time tasks, this can be arbitrary. But tracing Now for non Real-Time tasks, this can be arbitrary. But tracing
it none the less can be interesting. it none the less can be interesting.
Without function tracing: Without function tracing::
# echo 0 > options/function-trace # echo 0 > options/function-trace
# echo wakeup > current_tracer # echo wakeup > current_tracer
...@@ -1640,23 +1706,23 @@ Without function tracing: ...@@ -1640,23 +1706,23 @@ Without function tracing:
# chrt -f 5 sleep 1 # chrt -f 5 sleep 1
# echo 0 > tracing_on # echo 0 > tracing_on
# cat trace # cat trace
# tracer: wakeup # tracer: wakeup
# #
# wakeup latency trace v1.1.5 on 3.8.0-test+ # wakeup latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 15 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 15 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: kworker/3:1H-312 (uid:0 nice:-20 policy:0 rt_prio:0) # | task: kworker/3:1H-312 (uid:0 nice:-20 policy:0 rt_prio:0)
# ----------------- # -----------------
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
<idle>-0 3dNs7 0us : 0:120:R + [003] 312:100:R kworker/3:1H <idle>-0 3dNs7 0us : 0:120:R + [003] 312:100:R kworker/3:1H
<idle>-0 3dNs7 1us+: ttwu_do_activate.constprop.87 <-try_to_wake_up <idle>-0 3dNs7 1us+: ttwu_do_activate.constprop.87 <-try_to_wake_up
<idle>-0 3d..3 15us : __schedule <-schedule <idle>-0 3d..3 15us : __schedule <-schedule
...@@ -1698,6 +1764,7 @@ Since this tracer only deals with RT tasks, we will run this ...@@ -1698,6 +1764,7 @@ Since this tracer only deals with RT tasks, we will run this
slightly differently than we did with the previous tracers. slightly differently than we did with the previous tracers.
Instead of performing an 'ls', we will run 'sleep 1' under Instead of performing an 'ls', we will run 'sleep 1' under
'chrt' which changes the priority of the task. 'chrt' which changes the priority of the task.
::
# echo 0 > options/function-trace # echo 0 > options/function-trace
# echo wakeup_rt > current_tracer # echo wakeup_rt > current_tracer
...@@ -1706,25 +1773,25 @@ Instead of performing an 'ls', we will run 'sleep 1' under ...@@ -1706,25 +1773,25 @@ Instead of performing an 'ls', we will run 'sleep 1' under
# chrt -f 5 sleep 1 # chrt -f 5 sleep 1
# echo 0 > tracing_on # echo 0 > tracing_on
# cat trace # cat trace
# tracer: wakeup # tracer: wakeup
# #
# tracer: wakeup_rt # tracer: wakeup_rt
# #
# wakeup_rt latency trace v1.1.5 on 3.8.0-test+ # wakeup_rt latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 5 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 5 us, #4/4, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: sleep-2389 (uid:0 nice:0 policy:1 rt_prio:5) # | task: sleep-2389 (uid:0 nice:0 policy:1 rt_prio:5)
# ----------------- # -----------------
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
<idle>-0 3d.h4 0us : 0:120:R + [003] 2389: 94:R sleep <idle>-0 3d.h4 0us : 0:120:R + [003] 2389: 94:R sleep
<idle>-0 3d.h4 1us+: ttwu_do_activate.constprop.87 <-try_to_wake_up <idle>-0 3d.h4 1us+: ttwu_do_activate.constprop.87 <-try_to_wake_up
<idle>-0 3d..3 5us : __schedule <-schedule <idle>-0 3d..3 5us : __schedule <-schedule
...@@ -1743,6 +1810,7 @@ and not the internal kernel priority. The policy is 1 for ...@@ -1743,6 +1810,7 @@ and not the internal kernel priority. The policy is 1 for
SCHED_FIFO and 2 for SCHED_RR. SCHED_FIFO and 2 for SCHED_RR.
Note, that the trace data shows the internal priority (99 - rtprio). Note, that the trace data shows the internal priority (99 - rtprio).
::
<idle>-0 3d..3 5us : 0:120:R ==> [003] 2389: 94:R sleep <idle>-0 3d..3 5us : 0:120:R ==> [003] 2389: 94:R sleep
...@@ -1752,26 +1820,27 @@ and in the running state 'R'. The sleep task was scheduled in with ...@@ -1752,26 +1820,27 @@ and in the running state 'R'. The sleep task was scheduled in with
and it too is in the running state. and it too is in the running state.
Doing the same with chrt -r 5 and function-trace set. Doing the same with chrt -r 5 and function-trace set.
::
echo 1 > options/function-trace echo 1 > options/function-trace
# tracer: wakeup_rt # tracer: wakeup_rt
# #
# wakeup_rt latency trace v1.1.5 on 3.8.0-test+ # wakeup_rt latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 29 us, #85/85, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 29 us, #85/85, CPU#3 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: sleep-2448 (uid:0 nice:0 policy:1 rt_prio:5) # | task: sleep-2448 (uid:0 nice:0 policy:1 rt_prio:5)
# ----------------- # -----------------
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
<idle>-0 3d.h4 1us+: 0:120:R + [003] 2448: 94:R sleep <idle>-0 3d.h4 1us+: 0:120:R + [003] 2448: 94:R sleep
<idle>-0 3d.h4 2us : ttwu_do_activate.constprop.87 <-try_to_wake_up <idle>-0 3d.h4 2us : ttwu_do_activate.constprop.87 <-try_to_wake_up
<idle>-0 3d.h3 3us : check_preempt_curr <-ttwu_do_wakeup <idle>-0 3d.h3 3us : check_preempt_curr <-ttwu_do_wakeup
...@@ -1871,6 +1940,7 @@ As function tracing can induce a much larger latency, but without ...@@ -1871,6 +1940,7 @@ As function tracing can induce a much larger latency, but without
seeing what happens within the latency it is hard to know what seeing what happens within the latency it is hard to know what
caused it. There is a middle ground, and that is with enabling caused it. There is a middle ground, and that is with enabling
events. events.
::
# echo 0 > options/function-trace # echo 0 > options/function-trace
# echo wakeup_rt > current_tracer # echo wakeup_rt > current_tracer
...@@ -1880,23 +1950,23 @@ events. ...@@ -1880,23 +1950,23 @@ events.
# chrt -f 5 sleep 1 # chrt -f 5 sleep 1
# echo 0 > tracing_on # echo 0 > tracing_on
# cat trace # cat trace
# tracer: wakeup_rt # tracer: wakeup_rt
# #
# wakeup_rt latency trace v1.1.5 on 3.8.0-test+ # wakeup_rt latency trace v1.1.5 on 3.8.0-test+
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# latency: 6 us, #12/12, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4) # latency: 6 us, #12/12, CPU#2 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# ----------------- # -----------------
# | task: sleep-5882 (uid:0 nice:0 policy:1 rt_prio:5) # | task: sleep-5882 (uid:0 nice:0 policy:1 rt_prio:5)
# ----------------- # -----------------
# #
# _------=> CPU# # _------=> CPU#
# / _-----=> irqs-off # / _-----=> irqs-off
# | / _----=> need-resched # | / _----=> need-resched
# || / _---=> hardirq/softirq # || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth # ||| / _--=> preempt-depth
# |||| / delay # |||| / delay
# cmd pid ||||| time | caller # cmd pid ||||| time | caller
# \ / ||||| \ | / # \ / ||||| \ | /
<idle>-0 2d.h4 0us : 0:120:R + [002] 5882: 94:R sleep <idle>-0 2d.h4 0us : 0:120:R + [002] 5882: 94:R sleep
<idle>-0 2d.h4 0us : ttwu_do_activate.constprop.87 <-try_to_wake_up <idle>-0 2d.h4 0us : ttwu_do_activate.constprop.87 <-try_to_wake_up
<idle>-0 2d.h4 1us : sched_wakeup: comm=sleep pid=5882 prio=94 success=1 target_cpu=002 <idle>-0 2d.h4 1us : sched_wakeup: comm=sleep pid=5882 prio=94 success=1 target_cpu=002
...@@ -1918,19 +1988,20 @@ The hardware latency detector is executed by enabling the "hwlat" tracer. ...@@ -1918,19 +1988,20 @@ The hardware latency detector is executed by enabling the "hwlat" tracer.
NOTE, this tracer will affect the performance of the system as it will NOTE, this tracer will affect the performance of the system as it will
periodically make a CPU constantly busy with interrupts disabled. periodically make a CPU constantly busy with interrupts disabled.
::
# echo hwlat > current_tracer # echo hwlat > current_tracer
# sleep 100 # sleep 100
# cat trace # cat trace
# tracer: hwlat # tracer: hwlat
# #
# _-----=> irqs-off # _-----=> irqs-off
# / _----=> need-resched # / _----=> need-resched
# | / _---=> hardirq/softirq # | / _---=> hardirq/softirq
# || / _--=> preempt-depth # || / _--=> preempt-depth
# ||| / delay # ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | | # | | | |||| | |
<...>-3638 [001] d... 19452.055471: #1 inner/outer(us): 12/14 ts:1499801089.066141940 <...>-3638 [001] d... 19452.055471: #1 inner/outer(us): 12/14 ts:1499801089.066141940
<...>-3638 [003] d... 19454.071354: #2 inner/outer(us): 11/9 ts:1499801091.082164365 <...>-3638 [003] d... 19454.071354: #2 inner/outer(us): 11/9 ts:1499801091.082164365
<...>-3638 [002] dn.. 19461.126852: #3 inner/outer(us): 12/9 ts:1499801098.138150062 <...>-3638 [002] dn.. 19461.126852: #3 inner/outer(us): 12/9 ts:1499801098.138150062
...@@ -1942,7 +2013,8 @@ periodically make a CPU constantly busy with interrupts disabled. ...@@ -1942,7 +2013,8 @@ periodically make a CPU constantly busy with interrupts disabled.
The above output is somewhat the same in the header. All events will have The above output is somewhat the same in the header. All events will have
interrupts disabled 'd'. Under the FUNCTION title there is: interrupts disabled 'd'. Under the FUNCTION title there is:
#1 - This is the count of events recorded that were greater than the #1
This is the count of events recorded that were greater than the
tracing_threshold (See below). tracing_threshold (See below).
inner/outer(us): 12/14 inner/outer(us): 12/14
...@@ -1968,7 +2040,8 @@ interrupts disabled 'd'. Under the FUNCTION title there is: ...@@ -1968,7 +2040,8 @@ interrupts disabled 'd'. Under the FUNCTION title there is:
hwlat files: hwlat files:
tracing_threshold - This gets automatically set to "10" to represent 10 tracing_threshold
This gets automatically set to "10" to represent 10
microseconds. This is the threshold of latency that microseconds. This is the threshold of latency that
needs to be detected before the trace will be recorded. needs to be detected before the trace will be recorded.
...@@ -1976,14 +2049,16 @@ hwlat files: ...@@ -1976,14 +2049,16 @@ hwlat files:
written into "current_tracer"), the original value for written into "current_tracer"), the original value for
tracing_threshold is placed back into this file. tracing_threshold is placed back into this file.
hwlat_detector/width - The length of time the test runs with interrupts hwlat_detector/width
disabled. The length of time the test runs with interrupts disabled.
hwlat_detector/window - The length of time of the window which the test hwlat_detector/window
The length of time of the window which the test
runs. That is, the test will run for "width" runs. That is, the test will run for "width"
microseconds per "window" microseconds microseconds per "window" microseconds
tracing_cpumask - When the test is started. A kernel thread is created that tracing_cpumask
When the test is started. A kernel thread is created that
runs the test. This thread will alternate between CPUs runs the test. This thread will alternate between CPUs
listed in the tracing_cpumask between each period listed in the tracing_cpumask between each period
(one "window"). To limit the test to specific CPUs (one "window"). To limit the test to specific CPUs
...@@ -1997,6 +2072,7 @@ This tracer is the function tracer. Enabling the function tracer ...@@ -1997,6 +2072,7 @@ This tracer is the function tracer. Enabling the function tracer
can be done from the debug file system. Make sure the can be done from the debug file system. Make sure the
ftrace_enabled is set; otherwise this tracer is a nop. ftrace_enabled is set; otherwise this tracer is a nop.
See the "ftrace_enabled" section below. See the "ftrace_enabled" section below.
::
# sysctl kernel.ftrace_enabled=1 # sysctl kernel.ftrace_enabled=1
# echo function > current_tracer # echo function > current_tracer
...@@ -2004,17 +2080,17 @@ See the "ftrace_enabled" section below. ...@@ -2004,17 +2080,17 @@ See the "ftrace_enabled" section below.
# usleep 1 # usleep 1
# echo 0 > tracing_on # echo 0 > tracing_on
# cat trace # cat trace
# tracer: function # tracer: function
# #
# entries-in-buffer/entries-written: 24799/24799 #P:4 # entries-in-buffer/entries-written: 24799/24799 #P:4
# #
# _-----=> irqs-off # _-----=> irqs-off
# / _----=> need-resched # / _----=> need-resched
# | / _---=> hardirq/softirq # | / _---=> hardirq/softirq
# || / _--=> preempt-depth # || / _--=> preempt-depth
# ||| / delay # ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | | # | | | |||| | |
bash-1994 [002] .... 3082.063030: mutex_unlock <-rb_simple_write bash-1994 [002] .... 3082.063030: mutex_unlock <-rb_simple_write
bash-1994 [002] .... 3082.063031: __mutex_unlock_slowpath <-mutex_unlock bash-1994 [002] .... 3082.063031: __mutex_unlock_slowpath <-mutex_unlock
bash-1994 [002] .... 3082.063031: __fsnotify_parent <-fsnotify_modify bash-1994 [002] .... 3082.063031: __fsnotify_parent <-fsnotify_modify
...@@ -2023,7 +2099,7 @@ See the "ftrace_enabled" section below. ...@@ -2023,7 +2099,7 @@ See the "ftrace_enabled" section below.
bash-1994 [002] .... 3082.063032: add_preempt_count <-__srcu_read_lock bash-1994 [002] .... 3082.063032: add_preempt_count <-__srcu_read_lock
bash-1994 [002] ...1 3082.063032: sub_preempt_count <-__srcu_read_lock bash-1994 [002] ...1 3082.063032: sub_preempt_count <-__srcu_read_lock
bash-1994 [002] .... 3082.063033: __srcu_read_unlock <-fsnotify bash-1994 [002] .... 3082.063033: __srcu_read_unlock <-fsnotify
[...] [...]
Note: function tracer uses ring buffers to store the above Note: function tracer uses ring buffers to store the above
...@@ -2034,11 +2110,11 @@ record. For this reason, it is sometimes better to disable ...@@ -2034,11 +2110,11 @@ record. For this reason, it is sometimes better to disable
tracing directly from a program. This allows you to stop the tracing directly from a program. This allows you to stop the
tracing at the point that you hit the part that you are tracing at the point that you hit the part that you are
interested in. To disable the tracing directly from a C program, interested in. To disable the tracing directly from a C program,
something like following code snippet can be used: something like following code snippet can be used::
int trace_fd; int trace_fd;
[...] [...]
int main(int argc, char *argv[]) { int main(int argc, char *argv[]) {
[...] [...]
trace_fd = open(tracing_file("tracing_on"), O_WRONLY); trace_fd = open(tracing_file("tracing_on"), O_WRONLY);
[...] [...]
...@@ -2046,22 +2122,22 @@ int main(int argc, char *argv[]) { ...@@ -2046,22 +2122,22 @@ int main(int argc, char *argv[]) {
write(trace_fd, "0", 1); write(trace_fd, "0", 1);
} }
[...] [...]
} }
Single thread tracing Single thread tracing
--------------------- ---------------------
By writing into set_ftrace_pid you can trace a By writing into set_ftrace_pid you can trace a
single thread. For example: single thread. For example::
# cat set_ftrace_pid # cat set_ftrace_pid
no pid no pid
# echo 3111 > set_ftrace_pid # echo 3111 > set_ftrace_pid
# cat set_ftrace_pid # cat set_ftrace_pid
3111 3111
# echo function > current_tracer # echo function > current_tracer
# cat trace | head # cat trace | head
# tracer: function # tracer: function
# #
# TASK-PID CPU# TIMESTAMP FUNCTION # TASK-PID CPU# TIMESTAMP FUNCTION
...@@ -2072,8 +2148,8 @@ no pid ...@@ -2072,8 +2148,8 @@ no pid
yum-updatesd-3111 [003] 1637.254683: lock_hrtimer_base <-hrtimer_try_to_cancel yum-updatesd-3111 [003] 1637.254683: lock_hrtimer_base <-hrtimer_try_to_cancel
yum-updatesd-3111 [003] 1637.254685: fget_light <-do_sys_poll yum-updatesd-3111 [003] 1637.254685: fget_light <-do_sys_poll
yum-updatesd-3111 [003] 1637.254686: pipe_poll <-do_sys_poll yum-updatesd-3111 [003] 1637.254686: pipe_poll <-do_sys_poll
# echo > set_ftrace_pid # echo > set_ftrace_pid
# cat trace |head # cat trace |head
# tracer: function # tracer: function
# #
# TASK-PID CPU# TIMESTAMP FUNCTION # TASK-PID CPU# TIMESTAMP FUNCTION
...@@ -2086,22 +2162,23 @@ no pid ...@@ -2086,22 +2162,23 @@ no pid
yum-updatesd-3111 [003] 1701.957693: path_put <-audit_syscall_exit yum-updatesd-3111 [003] 1701.957693: path_put <-audit_syscall_exit
If you want to trace a function when executing, you could use If you want to trace a function when executing, you could use
something like this simple program: something like this simple program.
::
#include <stdio.h>
#include <stdlib.h> #include <stdio.h>
#include <sys/types.h> #include <stdlib.h>
#include <sys/stat.h> #include <sys/types.h>
#include <fcntl.h> #include <sys/stat.h>
#include <unistd.h> #include <fcntl.h>
#include <string.h> #include <unistd.h>
#include <string.h>
#define _STR(x) #x
#define STR(x) _STR(x) #define _STR(x) #x
#define MAX_PATH 256 #define STR(x) _STR(x)
#define MAX_PATH 256
const char *find_tracefs(void)
{ const char *find_tracefs(void)
{
static char tracefs[MAX_PATH+1]; static char tracefs[MAX_PATH+1];
static int tracefs_found; static int tracefs_found;
char type[100]; char type[100];
...@@ -2133,17 +2210,17 @@ const char *find_tracefs(void) ...@@ -2133,17 +2210,17 @@ const char *find_tracefs(void)
tracefs_found = 1; tracefs_found = 1;
return tracefs; return tracefs;
} }
const char *tracing_file(const char *file_name) const char *tracing_file(const char *file_name)
{ {
static char trace_file[MAX_PATH+1]; static char trace_file[MAX_PATH+1];
snprintf(trace_file, MAX_PATH, "%s/%s", find_tracefs(), file_name); snprintf(trace_file, MAX_PATH, "%s/%s", find_tracefs(), file_name);
return trace_file; return trace_file;
} }
int main (int argc, char **argv) int main (int argc, char **argv)
{ {
if (argc < 1) if (argc < 1)
exit(-1); exit(-1);
...@@ -2170,21 +2247,20 @@ int main (int argc, char **argv) ...@@ -2170,21 +2247,20 @@ int main (int argc, char **argv)
} }
return 0; return 0;
} }
Or this simple script! Or this simple script!
::
------ #!/bin/bash
#!/bin/bash
tracefs=`sed -ne 's/^tracefs \(.*\) tracefs.*/\1/p' /proc/mounts`
tracefs=`sed -ne 's/^tracefs \(.*\) tracefs.*/\1/p' /proc/mounts` echo nop > $tracefs/tracing/current_tracer
echo nop > $tracefs/tracing/current_tracer echo 0 > $tracefs/tracing/tracing_on
echo 0 > $tracefs/tracing/tracing_on echo $$ > $tracefs/tracing/set_ftrace_pid
echo $$ > $tracefs/tracing/set_ftrace_pid echo function > $tracefs/tracing/current_tracer
echo function > $tracefs/tracing/current_tracer echo 1 > $tracefs/tracing/tracing_on
echo 1 > $tracefs/tracing/tracing_on exec "$@"
exec "$@"
------
function graph tracer function graph tracer
...@@ -2219,10 +2295,12 @@ This tracer is useful in several situations: ...@@ -2219,10 +2295,12 @@ This tracer is useful in several situations:
- you just want to peek inside a working kernel and want to see - you just want to peek inside a working kernel and want to see
what happens there. what happens there.
# tracer: function_graph ::
#
# CPU DURATION FUNCTION CALLS # tracer: function_graph
# | | | | | | | #
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) | sys_open() { 0) | sys_open() {
0) | do_sys_open() { 0) | do_sys_open() {
...@@ -2251,25 +2329,25 @@ want, depending on your needs. ...@@ -2251,25 +2329,25 @@ want, depending on your needs.
tracing_cpu_mask file) or you might sometimes see unordered tracing_cpu_mask file) or you might sometimes see unordered
function calls while cpu tracing switch. function calls while cpu tracing switch.
hide: echo nofuncgraph-cpu > trace_options - hide: echo nofuncgraph-cpu > trace_options
show: echo funcgraph-cpu > trace_options - show: echo funcgraph-cpu > trace_options
- The duration (function's time of execution) is displayed on - The duration (function's time of execution) is displayed on
the closing bracket line of a function or on the same line the closing bracket line of a function or on the same line
than the current function in case of a leaf one. It is default than the current function in case of a leaf one. It is default
enabled. enabled.
hide: echo nofuncgraph-duration > trace_options - hide: echo nofuncgraph-duration > trace_options
show: echo funcgraph-duration > trace_options - show: echo funcgraph-duration > trace_options
- The overhead field precedes the duration field in case of - The overhead field precedes the duration field in case of
reached duration thresholds. reached duration thresholds.
hide: echo nofuncgraph-overhead > trace_options - hide: echo nofuncgraph-overhead > trace_options
show: echo funcgraph-overhead > trace_options - show: echo funcgraph-overhead > trace_options
depends on: funcgraph-duration - depends on: funcgraph-duration
ie: ie::
3) # 1837.709 us | } /* __switch_to */ 3) # 1837.709 us | } /* __switch_to */
3) | finish_task_switch() { 3) | finish_task_switch() {
...@@ -2309,6 +2387,8 @@ want, depending on your needs. ...@@ -2309,6 +2387,8 @@ want, depending on your needs.
1) + 18.542 us | } 1) + 18.542 us | }
2) $ 3594274 us | } 2) $ 3594274 us | }
Flags::
+ means that the function exceeded 10 usecs. + means that the function exceeded 10 usecs.
! means that the function exceeded 100 usecs. ! means that the function exceeded 100 usecs.
# means that the function exceeded 1000 usecs. # means that the function exceeded 1000 usecs.
...@@ -2320,10 +2400,10 @@ want, depending on your needs. ...@@ -2320,10 +2400,10 @@ want, depending on your needs.
- The task/pid field displays the thread cmdline and pid which - The task/pid field displays the thread cmdline and pid which
executed the function. It is default disabled. executed the function. It is default disabled.
hide: echo nofuncgraph-proc > trace_options - hide: echo nofuncgraph-proc > trace_options
show: echo funcgraph-proc > trace_options - show: echo funcgraph-proc > trace_options
ie: ie::
# tracer: function_graph # tracer: function_graph
# #
...@@ -2344,10 +2424,10 @@ want, depending on your needs. ...@@ -2344,10 +2424,10 @@ want, depending on your needs.
system clock since it started. A snapshot of this time is system clock since it started. A snapshot of this time is
given on each entry/exit of functions given on each entry/exit of functions
hide: echo nofuncgraph-abstime > trace_options - hide: echo nofuncgraph-abstime > trace_options
show: echo funcgraph-abstime > trace_options - show: echo funcgraph-abstime > trace_options
ie: ie::
# #
# TIME CPU DURATION FUNCTION CALLS # TIME CPU DURATION FUNCTION CALLS
...@@ -2376,17 +2456,19 @@ enabled for functions whose start is in the trace buffer, ...@@ -2376,17 +2456,19 @@ enabled for functions whose start is in the trace buffer,
allowing easier searching with grep for function durations. allowing easier searching with grep for function durations.
It is default disabled. It is default disabled.
hide: echo nofuncgraph-tail > trace_options - hide: echo nofuncgraph-tail > trace_options
show: echo funcgraph-tail > trace_options - show: echo funcgraph-tail > trace_options
Example with nofuncgraph-tail (default)::
Example with nofuncgraph-tail (default):
0) | putname() { 0) | putname() {
0) | kmem_cache_free() { 0) | kmem_cache_free() {
0) 0.518 us | __phys_addr(); 0) 0.518 us | __phys_addr();
0) 1.757 us | } 0) 1.757 us | }
0) 2.861 us | } 0) 2.861 us | }
Example with funcgraph-tail: Example with funcgraph-tail::
0) | putname() { 0) | putname() {
0) | kmem_cache_free() { 0) | kmem_cache_free() {
0) 0.518 us | __phys_addr(); 0) 0.518 us | __phys_addr();
...@@ -2396,11 +2478,11 @@ It is default disabled. ...@@ -2396,11 +2478,11 @@ It is default disabled.
You can put some comments on specific functions by using You can put some comments on specific functions by using
trace_printk() For example, if you want to put a comment inside trace_printk() For example, if you want to put a comment inside
the __might_sleep() function, you just have to include the __might_sleep() function, you just have to include
<linux/ftrace.h> and call trace_printk() inside __might_sleep() <linux/ftrace.h> and call trace_printk() inside __might_sleep()::
trace_printk("I'm a comment!\n") trace_printk("I'm a comment!\n")
will produce: will produce::
1) | __might_sleep() { 1) | __might_sleep() {
1) | /* I'm a comment! */ 1) | /* I'm a comment! */
...@@ -2487,16 +2569,18 @@ listed in: ...@@ -2487,16 +2569,18 @@ listed in:
available_filter_functions available_filter_functions
::
# cat available_filter_functions # cat available_filter_functions
put_prev_task_idle put_prev_task_idle
kmem_cache_create kmem_cache_create
pick_next_task_rt pick_next_task_rt
get_online_cpus get_online_cpus
pick_next_task_fair pick_next_task_fair
mutex_lock mutex_lock
[...] [...]
If I am only interested in sys_nanosleep and hrtimer_interrupt: If I am only interested in sys_nanosleep and hrtimer_interrupt::
# echo sys_nanosleep hrtimer_interrupt > set_ftrace_filter # echo sys_nanosleep hrtimer_interrupt > set_ftrace_filter
# echo function > current_tracer # echo function > current_tracer
...@@ -2504,17 +2588,17 @@ If I am only interested in sys_nanosleep and hrtimer_interrupt: ...@@ -2504,17 +2588,17 @@ If I am only interested in sys_nanosleep and hrtimer_interrupt:
# usleep 1 # usleep 1
# echo 0 > tracing_on # echo 0 > tracing_on
# cat trace # cat trace
# tracer: function # tracer: function
# #
# entries-in-buffer/entries-written: 5/5 #P:4 # entries-in-buffer/entries-written: 5/5 #P:4
# #
# _-----=> irqs-off # _-----=> irqs-off
# / _----=> need-resched # / _----=> need-resched
# | / _---=> hardirq/softirq # | / _---=> hardirq/softirq
# || / _--=> preempt-depth # || / _--=> preempt-depth
# ||| / delay # ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | | # | | | |||| | |
usleep-2665 [001] .... 4186.475355: sys_nanosleep <-system_call_fastpath usleep-2665 [001] .... 4186.475355: sys_nanosleep <-system_call_fastpath
<idle>-0 [001] d.h1 4186.475409: hrtimer_interrupt <-smp_apic_timer_interrupt <idle>-0 [001] d.h1 4186.475409: hrtimer_interrupt <-smp_apic_timer_interrupt
usleep-2665 [001] d.h1 4186.475426: hrtimer_interrupt <-smp_apic_timer_interrupt usleep-2665 [001] d.h1 4186.475426: hrtimer_interrupt <-smp_apic_timer_interrupt
...@@ -2522,39 +2606,46 @@ If I am only interested in sys_nanosleep and hrtimer_interrupt: ...@@ -2522,39 +2606,46 @@ If I am only interested in sys_nanosleep and hrtimer_interrupt:
<idle>-0 [002] d.h1 4186.475427: hrtimer_interrupt <-smp_apic_timer_interrupt <idle>-0 [002] d.h1 4186.475427: hrtimer_interrupt <-smp_apic_timer_interrupt
To see which functions are being traced, you can cat the file: To see which functions are being traced, you can cat the file:
::
# cat set_ftrace_filter # cat set_ftrace_filter
hrtimer_interrupt hrtimer_interrupt
sys_nanosleep sys_nanosleep
Perhaps this is not enough. The filters also allow glob(7) matching. Perhaps this is not enough. The filters also allow glob(7) matching.
<match>* - will match functions that begin with <match> <match>*
*<match> - will match functions that end with <match> will match functions that begin with <match>
*<match>* - will match functions that have <match> in it *<match>
<match1>*<match2> - will match functions that begin with will match functions that end with <match>
<match1> and end with <match2> *<match>*
will match functions that have <match> in it
Note: It is better to use quotes to enclose the wild cards, <match1>*<match2>
will match functions that begin with <match1> and end with <match2>
.. note::
It is better to use quotes to enclose the wild cards,
otherwise the shell may expand the parameters into names otherwise the shell may expand the parameters into names
of files in the local directory. of files in the local directory.
::
# echo 'hrtimer_*' > set_ftrace_filter # echo 'hrtimer_*' > set_ftrace_filter
Produces: Produces::
# tracer: function # tracer: function
# #
# entries-in-buffer/entries-written: 897/897 #P:4 # entries-in-buffer/entries-written: 897/897 #P:4
# #
# _-----=> irqs-off # _-----=> irqs-off
# / _----=> need-resched # / _----=> need-resched
# | / _---=> hardirq/softirq # | / _---=> hardirq/softirq
# || / _--=> preempt-depth # || / _--=> preempt-depth
# ||| / delay # ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | | # | | | |||| | |
<idle>-0 [003] dN.1 4228.547803: hrtimer_cancel <-tick_nohz_idle_exit <idle>-0 [003] dN.1 4228.547803: hrtimer_cancel <-tick_nohz_idle_exit
<idle>-0 [003] dN.1 4228.547804: hrtimer_try_to_cancel <-hrtimer_cancel <idle>-0 [003] dN.1 4228.547804: hrtimer_try_to_cancel <-hrtimer_cancel
<idle>-0 [003] dN.2 4228.547805: hrtimer_force_reprogram <-__remove_hrtimer <idle>-0 [003] dN.2 4228.547805: hrtimer_force_reprogram <-__remove_hrtimer
...@@ -2565,24 +2656,25 @@ Produces: ...@@ -2565,24 +2656,25 @@ Produces:
<idle>-0 [003] d..2 4228.547860: hrtimer_force_reprogram <-__rem <idle>-0 [003] d..2 4228.547860: hrtimer_force_reprogram <-__rem
Notice that we lost the sys_nanosleep. Notice that we lost the sys_nanosleep.
::
# cat set_ftrace_filter # cat set_ftrace_filter
hrtimer_run_queues hrtimer_run_queues
hrtimer_run_pending hrtimer_run_pending
hrtimer_init hrtimer_init
hrtimer_cancel hrtimer_cancel
hrtimer_try_to_cancel hrtimer_try_to_cancel
hrtimer_forward hrtimer_forward
hrtimer_start hrtimer_start
hrtimer_reprogram hrtimer_reprogram
hrtimer_force_reprogram hrtimer_force_reprogram
hrtimer_get_next_event hrtimer_get_next_event
hrtimer_interrupt hrtimer_interrupt
hrtimer_nanosleep hrtimer_nanosleep
hrtimer_wakeup hrtimer_wakeup
hrtimer_get_remaining hrtimer_get_remaining
hrtimer_get_res hrtimer_get_res
hrtimer_init_sleeper hrtimer_init_sleeper
This is because the '>' and '>>' act just like they do in bash. This is because the '>' and '>>' act just like they do in bash.
...@@ -2590,7 +2682,7 @@ To rewrite the filters, use '>' ...@@ -2590,7 +2682,7 @@ To rewrite the filters, use '>'
To append to the filters, use '>>' To append to the filters, use '>>'
To clear out a filter so that all functions will be recorded To clear out a filter so that all functions will be recorded
again: again::
# echo > set_ftrace_filter # echo > set_ftrace_filter
# cat set_ftrace_filter # cat set_ftrace_filter
...@@ -2598,48 +2690,51 @@ again: ...@@ -2598,48 +2690,51 @@ again:
Again, now we want to append. Again, now we want to append.
::
# echo sys_nanosleep > set_ftrace_filter # echo sys_nanosleep > set_ftrace_filter
# cat set_ftrace_filter # cat set_ftrace_filter
sys_nanosleep sys_nanosleep
# echo 'hrtimer_*' >> set_ftrace_filter # echo 'hrtimer_*' >> set_ftrace_filter
# cat set_ftrace_filter # cat set_ftrace_filter
hrtimer_run_queues hrtimer_run_queues
hrtimer_run_pending hrtimer_run_pending
hrtimer_init hrtimer_init
hrtimer_cancel hrtimer_cancel
hrtimer_try_to_cancel hrtimer_try_to_cancel
hrtimer_forward hrtimer_forward
hrtimer_start hrtimer_start
hrtimer_reprogram hrtimer_reprogram
hrtimer_force_reprogram hrtimer_force_reprogram
hrtimer_get_next_event hrtimer_get_next_event
hrtimer_interrupt hrtimer_interrupt
sys_nanosleep sys_nanosleep
hrtimer_nanosleep hrtimer_nanosleep
hrtimer_wakeup hrtimer_wakeup
hrtimer_get_remaining hrtimer_get_remaining
hrtimer_get_res hrtimer_get_res
hrtimer_init_sleeper hrtimer_init_sleeper
The set_ftrace_notrace prevents those functions from being The set_ftrace_notrace prevents those functions from being
traced. traced.
::
# echo '*preempt*' '*lock*' > set_ftrace_notrace # echo '*preempt*' '*lock*' > set_ftrace_notrace
Produces: Produces::
# tracer: function # tracer: function
# #
# entries-in-buffer/entries-written: 39608/39608 #P:4 # entries-in-buffer/entries-written: 39608/39608 #P:4
# #
# _-----=> irqs-off # _-----=> irqs-off
# / _----=> need-resched # / _----=> need-resched
# | / _---=> hardirq/softirq # | / _---=> hardirq/softirq
# || / _--=> preempt-depth # || / _--=> preempt-depth
# ||| / delay # ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | | # | | | |||| | |
bash-1994 [000] .... 4342.324896: file_ra_state_init <-do_dentry_open bash-1994 [000] .... 4342.324896: file_ra_state_init <-do_dentry_open
bash-1994 [000] .... 4342.324897: open_check_o_direct <-do_last bash-1994 [000] .... 4342.324897: open_check_o_direct <-do_last
bash-1994 [000] .... 4342.324897: ima_file_check <-do_last bash-1994 [000] .... 4342.324897: ima_file_check <-do_last
...@@ -2664,12 +2759,12 @@ function tracer and the function-graph-tracer, there are some ...@@ -2664,12 +2759,12 @@ function tracer and the function-graph-tracer, there are some
special features only available in the function-graph tracer. special features only available in the function-graph tracer.
If you want to trace only one function and all of its children, If you want to trace only one function and all of its children,
you just have to echo its name into set_graph_function: you just have to echo its name into set_graph_function::
echo __do_fault > set_graph_function echo __do_fault > set_graph_function
will produce the following "expanded" trace of the __do_fault() will produce the following "expanded" trace of the __do_fault()
function: function::
0) | __do_fault() { 0) | __do_fault() {
0) | filemap_fault() { 0) | filemap_fault() {
...@@ -2706,13 +2801,13 @@ function: ...@@ -2706,13 +2801,13 @@ function:
0) 2.793 us | } 0) 2.793 us | }
0) + 14.012 us | } 0) + 14.012 us | }
You can also expand several functions at once: You can also expand several functions at once::
echo sys_open > set_graph_function echo sys_open > set_graph_function
echo sys_close >> set_graph_function echo sys_close >> set_graph_function
Now if you want to go back to trace all functions you can clear Now if you want to go back to trace all functions you can clear
this special filter via: this special filter via::
echo > set_graph_function echo > set_graph_function
...@@ -2728,7 +2823,7 @@ also for any other uses (perf, kprobes, stack tracing, profiling, etc). ...@@ -2728,7 +2823,7 @@ also for any other uses (perf, kprobes, stack tracing, profiling, etc).
Please disable this with care. Please disable this with care.
This can be disable (and enabled) with: This can be disable (and enabled) with::
sysctl kernel.ftrace_enabled=0 sysctl kernel.ftrace_enabled=0
sysctl kernel.ftrace_enabled=1 sysctl kernel.ftrace_enabled=1
...@@ -2743,13 +2838,13 @@ Filter commands ...@@ -2743,13 +2838,13 @@ Filter commands
--------------- ---------------
A few commands are supported by the set_ftrace_filter interface. A few commands are supported by the set_ftrace_filter interface.
Trace commands have the following format: Trace commands have the following format::
<function>:<command>:<parameter> <function>:<command>:<parameter>
The following commands are supported: The following commands are supported:
- mod - mod:
This command enables function filtering per module. The This command enables function filtering per module. The
parameter defines the module. For example, if only the write* parameter defines the module. For example, if only the write*
functions in the ext3 module are desired, run: functions in the ext3 module are desired, run:
...@@ -2760,97 +2855,99 @@ The following commands are supported: ...@@ -2760,97 +2855,99 @@ The following commands are supported:
filtering based on function names. Thus, adding more functions filtering based on function names. Thus, adding more functions
in a different module is accomplished by appending (>>) to the in a different module is accomplished by appending (>>) to the
filter file. Remove specific module functions by prepending filter file. Remove specific module functions by prepending
'!': '!'::
echo '!writeback*:mod:ext3' >> set_ftrace_filter echo '!writeback*:mod:ext3' >> set_ftrace_filter
Mod command supports module globbing. Disable tracing for all Mod command supports module globbing. Disable tracing for all
functions except a specific module: functions except a specific module::
echo '!*:mod:!ext3' >> set_ftrace_filter echo '!*:mod:!ext3' >> set_ftrace_filter
Disable tracing for all modules, but still trace kernel: Disable tracing for all modules, but still trace kernel::
echo '!*:mod:*' >> set_ftrace_filter echo '!*:mod:*' >> set_ftrace_filter
Enable filter only for kernel: Enable filter only for kernel::
echo '*write*:mod:!*' >> set_ftrace_filter echo '*write*:mod:!*' >> set_ftrace_filter
Enable filter for module globbing: Enable filter for module globbing::
echo '*write*:mod:*snd*' >> set_ftrace_filter echo '*write*:mod:*snd*' >> set_ftrace_filter
- traceon/traceoff - traceon/traceoff:
These commands turn tracing on and off when the specified These commands turn tracing on and off when the specified
functions are hit. The parameter determines how many times the functions are hit. The parameter determines how many times the
tracing system is turned on and off. If unspecified, there is tracing system is turned on and off. If unspecified, there is
no limit. For example, to disable tracing when a schedule bug no limit. For example, to disable tracing when a schedule bug
is hit the first 5 times, run: is hit the first 5 times, run::
echo '__schedule_bug:traceoff:5' > set_ftrace_filter echo '__schedule_bug:traceoff:5' > set_ftrace_filter
To always disable tracing when __schedule_bug is hit: To always disable tracing when __schedule_bug is hit::
echo '__schedule_bug:traceoff' > set_ftrace_filter echo '__schedule_bug:traceoff' > set_ftrace_filter
These commands are cumulative whether or not they are appended These commands are cumulative whether or not they are appended
to set_ftrace_filter. To remove a command, prepend it by '!' to set_ftrace_filter. To remove a command, prepend it by '!'
and drop the parameter: and drop the parameter::
echo '!__schedule_bug:traceoff:0' > set_ftrace_filter echo '!__schedule_bug:traceoff:0' > set_ftrace_filter
The above removes the traceoff command for __schedule_bug The above removes the traceoff command for __schedule_bug
that have a counter. To remove commands without counters: that have a counter. To remove commands without counters::
echo '!__schedule_bug:traceoff' > set_ftrace_filter echo '!__schedule_bug:traceoff' > set_ftrace_filter
- snapshot - snapshot:
Will cause a snapshot to be triggered when the function is hit. Will cause a snapshot to be triggered when the function is hit.
::
echo 'native_flush_tlb_others:snapshot' > set_ftrace_filter echo 'native_flush_tlb_others:snapshot' > set_ftrace_filter
To only snapshot once: To only snapshot once:
::
echo 'native_flush_tlb_others:snapshot:1' > set_ftrace_filter echo 'native_flush_tlb_others:snapshot:1' > set_ftrace_filter
To remove the above commands: To remove the above commands::
echo '!native_flush_tlb_others:snapshot' > set_ftrace_filter echo '!native_flush_tlb_others:snapshot' > set_ftrace_filter
echo '!native_flush_tlb_others:snapshot:0' > set_ftrace_filter echo '!native_flush_tlb_others:snapshot:0' > set_ftrace_filter
- enable_event/disable_event - enable_event/disable_event:
These commands can enable or disable a trace event. Note, because These commands can enable or disable a trace event. Note, because
function tracing callbacks are very sensitive, when these commands function tracing callbacks are very sensitive, when these commands
are registered, the trace point is activated, but disabled in are registered, the trace point is activated, but disabled in
a "soft" mode. That is, the tracepoint will be called, but a "soft" mode. That is, the tracepoint will be called, but
just will not be traced. The event tracepoint stays in this mode just will not be traced. The event tracepoint stays in this mode
as long as there's a command that triggers it. as long as there's a command that triggers it.
::
echo 'try_to_wake_up:enable_event:sched:sched_switch:2' > \ echo 'try_to_wake_up:enable_event:sched:sched_switch:2' > \
set_ftrace_filter set_ftrace_filter
The format is: The format is::
<function>:enable_event:<system>:<event>[:count] <function>:enable_event:<system>:<event>[:count]
<function>:disable_event:<system>:<event>[:count] <function>:disable_event:<system>:<event>[:count]
To remove the events commands: To remove the events commands::
echo '!try_to_wake_up:enable_event:sched:sched_switch:0' > \ echo '!try_to_wake_up:enable_event:sched:sched_switch:0' > \
set_ftrace_filter set_ftrace_filter
echo '!schedule:disable_event:sched:sched_switch' > \ echo '!schedule:disable_event:sched:sched_switch' > \
set_ftrace_filter set_ftrace_filter
- dump - dump:
When the function is hit, it will dump the contents of the ftrace When the function is hit, it will dump the contents of the ftrace
ring buffer to the console. This is useful if you need to debug ring buffer to the console. This is useful if you need to debug
something, and want to dump the trace when a certain function something, and want to dump the trace when a certain function
is hit. Perhaps its a function that is called before a tripple is hit. Perhaps its a function that is called before a tripple
fault happens and does not allow you to get a regular dump. fault happens and does not allow you to get a regular dump.
- cpudump - cpudump:
When the function is hit, it will dump the contents of the ftrace When the function is hit, it will dump the contents of the ftrace
ring buffer for the current CPU to the console. Unlike the "dump" ring buffer for the current CPU to the console. Unlike the "dump"
command, it only prints out the contents of the ring buffer for the command, it only prints out the contents of the ring buffer for the
...@@ -2863,25 +2960,26 @@ The trace_pipe outputs the same content as the trace file, but ...@@ -2863,25 +2960,26 @@ The trace_pipe outputs the same content as the trace file, but
the effect on the tracing is different. Every read from the effect on the tracing is different. Every read from
trace_pipe is consumed. This means that subsequent reads will be trace_pipe is consumed. This means that subsequent reads will be
different. The trace is live. different. The trace is live.
::
# echo function > current_tracer # echo function > current_tracer
# cat trace_pipe > /tmp/trace.out & # cat trace_pipe > /tmp/trace.out &
[1] 4153 [1] 4153
# echo 1 > tracing_on # echo 1 > tracing_on
# usleep 1 # usleep 1
# echo 0 > tracing_on # echo 0 > tracing_on
# cat trace # cat trace
# tracer: function # tracer: function
# #
# entries-in-buffer/entries-written: 0/0 #P:4 # entries-in-buffer/entries-written: 0/0 #P:4
# #
# _-----=> irqs-off # _-----=> irqs-off
# / _----=> need-resched # / _----=> need-resched
# | / _---=> hardirq/softirq # | / _---=> hardirq/softirq
# || / _--=> preempt-depth # || / _--=> preempt-depth
# ||| / delay # ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | | # | | | |||| | |
# #
# cat /tmp/trace.out # cat /tmp/trace.out
...@@ -2908,44 +3006,51 @@ used to modify the size of the internal trace buffers. The ...@@ -2908,44 +3006,51 @@ used to modify the size of the internal trace buffers. The
number listed is the number of entries that can be recorded per number listed is the number of entries that can be recorded per
CPU. To know the full size, multiply the number of possible CPUs CPU. To know the full size, multiply the number of possible CPUs
with the number of entries. with the number of entries.
::
# cat buffer_size_kb # cat buffer_size_kb
1408 (units kilobytes) 1408 (units kilobytes)
Or simply read buffer_total_size_kb Or simply read buffer_total_size_kb
::
# cat buffer_total_size_kb # cat buffer_total_size_kb
5632 5632
To modify the buffer, simple echo in a number (in 1024 byte segments). To modify the buffer, simple echo in a number (in 1024 byte segments).
::
# echo 10000 > buffer_size_kb # echo 10000 > buffer_size_kb
# cat buffer_size_kb # cat buffer_size_kb
10000 (units kilobytes) 10000 (units kilobytes)
It will try to allocate as much as possible. If you allocate too It will try to allocate as much as possible. If you allocate too
much, it can cause Out-Of-Memory to trigger. much, it can cause Out-Of-Memory to trigger.
::
# echo 1000000000000 > buffer_size_kb # echo 1000000000000 > buffer_size_kb
-bash: echo: write error: Cannot allocate memory -bash: echo: write error: Cannot allocate memory
# cat buffer_size_kb # cat buffer_size_kb
85 85
The per_cpu buffers can be changed individually as well: The per_cpu buffers can be changed individually as well:
::
# echo 10000 > per_cpu/cpu0/buffer_size_kb # echo 10000 > per_cpu/cpu0/buffer_size_kb
# echo 100 > per_cpu/cpu1/buffer_size_kb # echo 100 > per_cpu/cpu1/buffer_size_kb
When the per_cpu buffers are not the same, the buffer_size_kb When the per_cpu buffers are not the same, the buffer_size_kb
at the top level will just show an X at the top level will just show an X
::
# cat buffer_size_kb # cat buffer_size_kb
X X
This is where the buffer_total_size_kb is useful: This is where the buffer_total_size_kb is useful:
::
# cat buffer_total_size_kb # cat buffer_total_size_kb
12916 12916
Writing to the top level buffer_size_kb will reset all the buffers Writing to the top level buffer_size_kb will reset all the buffers
to be the same again. to be the same again.
...@@ -2979,59 +3084,62 @@ feature: ...@@ -2979,59 +3084,62 @@ feature:
snapshot contents. snapshot contents.
More details are shown in the table below. More details are shown in the table below.
status\input | 0 | 1 | else | +--------------+------------+------------+------------+
--------------+------------+------------+------------+ |status\\input | 0 | 1 | else |
not allocated |(do nothing)| alloc+swap |(do nothing)| +==============+============+============+============+
--------------+------------+------------+------------+ |not allocated |(do nothing)| alloc+swap |(do nothing)|
allocated | free | swap | clear | +--------------+------------+------------+------------+
--------------+------------+------------+------------+ |allocated | free | swap | clear |
+--------------+------------+------------+------------+
Here is an example of using the snapshot feature. Here is an example of using the snapshot feature.
::
# echo 1 > events/sched/enable # echo 1 > events/sched/enable
# echo 1 > snapshot # echo 1 > snapshot
# cat snapshot # cat snapshot
# tracer: nop # tracer: nop
# #
# entries-in-buffer/entries-written: 71/71 #P:8 # entries-in-buffer/entries-written: 71/71 #P:8
# #
# _-----=> irqs-off # _-----=> irqs-off
# / _----=> need-resched # / _----=> need-resched
# | / _---=> hardirq/softirq # | / _---=> hardirq/softirq
# || / _--=> preempt-depth # || / _--=> preempt-depth
# ||| / delay # ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | | # | | | |||| | |
<idle>-0 [005] d... 2440.603828: sched_switch: prev_comm=swapper/5 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2242 next_prio=120 <idle>-0 [005] d... 2440.603828: sched_switch: prev_comm=swapper/5 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2242 next_prio=120
sleep-2242 [005] d... 2440.603846: sched_switch: prev_comm=snapshot-test-2 prev_pid=2242 prev_prio=120 prev_state=R ==> next_comm=kworker/5:1 next_pid=60 next_prio=120 sleep-2242 [005] d... 2440.603846: sched_switch: prev_comm=snapshot-test-2 prev_pid=2242 prev_prio=120 prev_state=R ==> next_comm=kworker/5:1 next_pid=60 next_prio=120
[...] [...]
<idle>-0 [002] d... 2440.707230: sched_switch: prev_comm=swapper/2 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2229 next_prio=120 <idle>-0 [002] d... 2440.707230: sched_switch: prev_comm=swapper/2 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2229 next_prio=120
# cat trace # cat trace
# tracer: nop # tracer: nop
# #
# entries-in-buffer/entries-written: 77/77 #P:8 # entries-in-buffer/entries-written: 77/77 #P:8
# #
# _-----=> irqs-off # _-----=> irqs-off
# / _----=> need-resched # / _----=> need-resched
# | / _---=> hardirq/softirq # | / _---=> hardirq/softirq
# || / _--=> preempt-depth # || / _--=> preempt-depth
# ||| / delay # ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | | # | | | |||| | |
<idle>-0 [007] d... 2440.707395: sched_switch: prev_comm=swapper/7 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2243 next_prio=120 <idle>-0 [007] d... 2440.707395: sched_switch: prev_comm=swapper/7 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=snapshot-test-2 next_pid=2243 next_prio=120
snapshot-test-2-2229 [002] d... 2440.707438: sched_switch: prev_comm=snapshot-test-2 prev_pid=2229 prev_prio=120 prev_state=S ==> next_comm=swapper/2 next_pid=0 next_prio=120 snapshot-test-2-2229 [002] d... 2440.707438: sched_switch: prev_comm=snapshot-test-2 prev_pid=2229 prev_prio=120 prev_state=S ==> next_comm=swapper/2 next_pid=0 next_prio=120
[...] [...]
If you try to use this snapshot feature when current tracer is If you try to use this snapshot feature when current tracer is
one of the latency tracers, you will get the following results. one of the latency tracers, you will get the following results.
::
# echo wakeup > current_tracer # echo wakeup > current_tracer
# echo 1 > snapshot # echo 1 > snapshot
bash: echo: write error: Device or resource busy bash: echo: write error: Device or resource busy
# cat snapshot # cat snapshot
cat: snapshot: Device or resource busy cat: snapshot: Device or resource busy
Instances Instances
...@@ -3041,12 +3149,13 @@ This directory can have new directories created inside of it using ...@@ -3041,12 +3149,13 @@ This directory can have new directories created inside of it using
mkdir, and removing directories with rmdir. The directory created mkdir, and removing directories with rmdir. The directory created
with mkdir in this directory will already contain files and other with mkdir in this directory will already contain files and other
directories after it is created. directories after it is created.
::
# mkdir instances/foo # mkdir instances/foo
# ls instances/foo # ls instances/foo
buffer_size_kb buffer_total_size_kb events free_buffer per_cpu buffer_size_kb buffer_total_size_kb events free_buffer per_cpu
set_event snapshot trace trace_clock trace_marker trace_options set_event snapshot trace trace_clock trace_marker trace_options
trace_pipe tracing_on trace_pipe tracing_on
As you can see, the new directory looks similar to the tracing directory As you can see, the new directory looks similar to the tracing directory
itself. In fact, it is very similar, except that the buffer and itself. In fact, it is very similar, except that the buffer and
...@@ -3064,6 +3173,7 @@ may become specific to the instance they reside in. ...@@ -3064,6 +3173,7 @@ may become specific to the instance they reside in.
Notice that none of the function tracer files are there, nor is Notice that none of the function tracer files are there, nor is
current_tracer and available_tracers. This is because the buffers current_tracer and available_tracers. This is because the buffers
can currently only have events enabled for them. can currently only have events enabled for them.
::
# mkdir instances/foo # mkdir instances/foo
# mkdir instances/bar # mkdir instances/bar
...@@ -3078,7 +3188,7 @@ can currently only have events enabled for them. ...@@ -3078,7 +3188,7 @@ can currently only have events enabled for them.
# echo 1 > instances/bar/events/irq/enable # echo 1 > instances/bar/events/irq/enable
# echo 1 > instances/zoot/events/syscalls/enable # echo 1 > instances/zoot/events/syscalls/enable
# cat trace_pipe # cat trace_pipe
CPU:2 [LOST 11745 EVENTS] CPU:2 [LOST 11745 EVENTS]
bash-2044 [002] .... 10594.481032: _raw_spin_lock_irqsave <-get_page_from_freelist bash-2044 [002] .... 10594.481032: _raw_spin_lock_irqsave <-get_page_from_freelist
bash-2044 [002] d... 10594.481032: add_preempt_count <-_raw_spin_lock_irqsave bash-2044 [002] d... 10594.481032: add_preempt_count <-_raw_spin_lock_irqsave
bash-2044 [002] d..1 10594.481032: __rmqueue <-get_page_from_freelist bash-2044 [002] d..1 10594.481032: __rmqueue <-get_page_from_freelist
...@@ -3090,7 +3200,7 @@ CPU:2 [LOST 11745 EVENTS] ...@@ -3090,7 +3200,7 @@ CPU:2 [LOST 11745 EVENTS]
bash-2044 [002] d... 10594.481034: __inc_zone_state <-zone_statistics bash-2044 [002] d... 10594.481034: __inc_zone_state <-zone_statistics
bash-2044 [002] d... 10594.481034: __inc_zone_state <-zone_statistics bash-2044 [002] d... 10594.481034: __inc_zone_state <-zone_statistics
bash-2044 [002] .... 10594.481035: arch_dup_task_struct <-copy_process bash-2044 [002] .... 10594.481035: arch_dup_task_struct <-copy_process
[...] [...]
# cat instances/foo/trace_pipe # cat instances/foo/trace_pipe
bash-1998 [000] d..4 136.676759: sched_wakeup: comm=kworker/0:1 pid=59 prio=120 success=1 target_cpu=000 bash-1998 [000] d..4 136.676759: sched_wakeup: comm=kworker/0:1 pid=59 prio=120 success=1 target_cpu=000
...@@ -3103,7 +3213,7 @@ CPU:2 [LOST 11745 EVENTS] ...@@ -3103,7 +3213,7 @@ CPU:2 [LOST 11745 EVENTS]
bash-1998 [000] d..3 136.677018: sched_switch: prev_comm=bash prev_pid=1998 prev_prio=120 prev_state=R+ ==> next_comm=kworker/0:1 next_pid=59 next_prio=120 bash-1998 [000] d..3 136.677018: sched_switch: prev_comm=bash prev_pid=1998 prev_prio=120 prev_state=R+ ==> next_comm=kworker/0:1 next_pid=59 next_prio=120
kworker/0:1-59 [000] d..4 136.677022: sched_wakeup: comm=sshd pid=1995 prio=120 success=1 target_cpu=001 kworker/0:1-59 [000] d..4 136.677022: sched_wakeup: comm=sshd pid=1995 prio=120 success=1 target_cpu=001
kworker/0:1-59 [000] d..3 136.677025: sched_switch: prev_comm=kworker/0:1 prev_pid=59 prev_prio=120 prev_state=S ==> next_comm=bash next_pid=1998 next_prio=120 kworker/0:1-59 [000] d..3 136.677025: sched_switch: prev_comm=kworker/0:1 prev_pid=59 prev_prio=120 prev_state=S ==> next_comm=bash next_pid=1998 next_prio=120
[...] [...]
# cat instances/bar/trace_pipe # cat instances/bar/trace_pipe
migration/1-14 [001] d.h3 138.732674: softirq_raise: vec=3 [action=NET_RX] migration/1-14 [001] d.h3 138.732674: softirq_raise: vec=3 [action=NET_RX]
...@@ -3118,20 +3228,20 @@ CPU:2 [LOST 11745 EVENTS] ...@@ -3118,20 +3228,20 @@ CPU:2 [LOST 11745 EVENTS]
sshd-1995 [001] d.h1 138.733280: irq_handler_exit: irq=21 ret=unhandled sshd-1995 [001] d.h1 138.733280: irq_handler_exit: irq=21 ret=unhandled
sshd-1995 [001] d.h1 138.733281: irq_handler_entry: irq=21 name=eth0 sshd-1995 [001] d.h1 138.733281: irq_handler_entry: irq=21 name=eth0
sshd-1995 [001] d.h1 138.733283: irq_handler_exit: irq=21 ret=handled sshd-1995 [001] d.h1 138.733283: irq_handler_exit: irq=21 ret=handled
[...] [...]
# cat instances/zoot/trace # cat instances/zoot/trace
# tracer: nop # tracer: nop
# #
# entries-in-buffer/entries-written: 18996/18996 #P:4 # entries-in-buffer/entries-written: 18996/18996 #P:4
# #
# _-----=> irqs-off # _-----=> irqs-off
# / _----=> need-resched # / _----=> need-resched
# | / _---=> hardirq/softirq # | / _---=> hardirq/softirq
# || / _--=> preempt-depth # || / _--=> preempt-depth
# ||| / delay # ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION # TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | | # | | | |||| | |
bash-1998 [000] d... 140.733501: sys_write -> 0x2 bash-1998 [000] d... 140.733501: sys_write -> 0x2
bash-1998 [000] d... 140.733504: sys_dup2(oldfd: a, newfd: 1) bash-1998 [000] d... 140.733504: sys_dup2(oldfd: a, newfd: 1)
bash-1998 [000] d... 140.733506: sys_dup2 -> 0x1 bash-1998 [000] d... 140.733506: sys_dup2 -> 0x1
...@@ -3149,6 +3259,7 @@ the function tracing. The foo instance displays wakeups and task ...@@ -3149,6 +3259,7 @@ the function tracing. The foo instance displays wakeups and task
switches. switches.
To remove the instances, simply delete their directories: To remove the instances, simply delete their directories:
::
# rmdir instances/foo # rmdir instances/foo
# rmdir instances/bar # rmdir instances/bar
...@@ -3174,6 +3285,7 @@ at every function call. This is enabled via the stack tracer. ...@@ -3174,6 +3285,7 @@ at every function call. This is enabled via the stack tracer.
CONFIG_STACK_TRACER enables the ftrace stack tracing functionality. CONFIG_STACK_TRACER enables the ftrace stack tracing functionality.
To enable it, write a '1' into /proc/sys/kernel/stack_tracer_enabled. To enable it, write a '1' into /proc/sys/kernel/stack_tracer_enabled.
::
# echo 1 > /proc/sys/kernel/stack_tracer_enabled # echo 1 > /proc/sys/kernel/stack_tracer_enabled
...@@ -3182,9 +3294,10 @@ the stack size of the kernel during boot up, by adding "stacktrace" ...@@ -3182,9 +3294,10 @@ the stack size of the kernel during boot up, by adding "stacktrace"
to the kernel command line parameter. to the kernel command line parameter.
After running it for a few minutes, the output looks like: After running it for a few minutes, the output looks like:
::
# cat stack_max_size # cat stack_max_size
2928 2928
# cat stack_trace # cat stack_trace
Depth Size Location (18 entries) Depth Size Location (18 entries)
...@@ -3214,7 +3327,6 @@ are not tested by the stack tracer when -mfentry is used. ...@@ -3214,7 +3327,6 @@ are not tested by the stack tracer when -mfentry is used.
Currently, -mfentry is used by gcc 4.6.0 and above on x86 only. Currently, -mfentry is used by gcc 4.6.0 and above on x86 only.
--------- More
----
More details can be found in the source code, in the More details can be found in the source code, in the `kernel/trace/*.c` files.
kernel/trace/*.c files.
Documentation/trace/index.rst
View file @ 1f198e22
...@@ -7,4 +7,5 @@ Linux Tracing Technologies ...@@ -7,4 +7,5 @@ Linux Tracing Technologies
ftrace-design ftrace-design
tracepoint-analysis tracepoint-analysis
ftrace
ftrace-uses ftrace-uses
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7