• Steven Rostedt (Google)'s avatar
    tracing: Allow synthetic events to pass around stacktraces · 00cf3d67
    Steven Rostedt (Google) authored
    Allow a stacktrace from one event to be displayed by the end event of a
    synthetic event. This is very useful when looking for the longest latency
    of a sleep or something blocked on I/O.
    
     # cd /sys/kernel/tracing/
     # echo 's:block_lat pid_t pid; u64 delta; unsigned long[] stack;' > dynamic_events
     # echo 'hist:keys=next_pid:ts=common_timestamp.usecs,st=stacktrace  if prev_state == 1||prev_state == 2' > events/sched/sched_switch/trigger
     # echo 'hist:keys=prev_pid:delta=common_timestamp.usecs-$ts,s=$st:onmax($delta).trace(block_lat,prev_pid,$delta,$s)' >> events/sched/sched_switch/trigger
    
    The above creates a "block_lat" synthetic event that take the stacktrace of
    when a task schedules out in either the interruptible or uninterruptible
    states, and on a new per process max $delta (the time it was scheduled
    out), will print the process id and the stacktrace.
    
      # echo 1 > events/synthetic/block_lat/enable
      # cat trace
     #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
     #              | |         |   |||||     |         |
        kworker/u16:0-767     [006] d..4.   560.645045: block_lat: pid=767 delta=66 stack=STACK:
     => __schedule
     => schedule
     => pipe_read
     => vfs_read
     => ksys_read
     => do_syscall_64
     => 0x966000aa
    
               <idle>-0       [003] d..4.   561.132117: block_lat: pid=0 delta=413787 stack=STACK:
     => __schedule
     => schedule
     => schedule_hrtimeout_range_clock
     => do_sys_poll
     => __x64_sys_poll
     => do_syscall_64
     => 0x966000aa
    
                <...>-153     [006] d..4.   562.068407: block_lat: pid=153 delta=54 stack=STACK:
     => __schedule
     => schedule
     => io_schedule
     => rq_qos_wait
     => wbt_wait
     => __rq_qos_throttle
     => blk_mq_submit_bio
     => submit_bio_noacct_nocheck
     => ext4_bio_write_page
     => mpage_submit_page
     => mpage_process_page_bufs
     => mpage_prepare_extent_to_map
     => ext4_do_writepages
     => ext4_writepages
     => do_writepages
     => __writeback_single_inode
    
    Link: https://lkml.kernel.org/r/20230117152236.010941267@goodmis.org
    
    Cc: Masami Hiramatsu <mhiramat@kernel.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Tom Zanussi <zanussi@kernel.org>
    Cc: Ross Zwisler <zwisler@google.com>
    Cc: Ching-lin Yu <chinglinyu@google.com>
    Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
    00cf3d67
trace_events_hist.c 166 KB