1. 10 May, 2013 9 commits
    • Steven Rostedt (Red Hat)'s avatar
      ftrace: Fix function probe when more than one probe is added · 19dd603e
      Steven Rostedt (Red Hat) authored
      When the first function probe is added and the function tracer
      is updated the functions are modified to call the probe.
      But when a second function is added, it updates the function
      records to have the second function also update, but it fails
      to update the actual function itself.
      
      This prevents the second (or third or forth and so on) probes
      from having their functions called.
      
        # echo vfs_symlink:enable_event:sched:sched_switch > set_ftrace_filter
        # echo vfs_unlink:enable_event:sched:sched_switch > set_ftrace_filter
        # cat trace
       # tracer: nop
       #
       # entries-in-buffer/entries-written: 0/0   #P:4
       #
       #                              _-----=> irqs-off
       #                             / _----=> need-resched
       #                            | / _---=> hardirq/softirq
       #                            || / _--=> preempt-depth
       #                            ||| /     delay
       #           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
       #              | |       |   ||||       |         |
        # touch /tmp/a
        # rm /tmp/a
        # cat trace
       # tracer: nop
       #
       # entries-in-buffer/entries-written: 0/0   #P:4
       #
       #                              _-----=> irqs-off
       #                             / _----=> need-resched
       #                            | / _---=> hardirq/softirq
       #                            || / _--=> preempt-depth
       #                            ||| /     delay
       #           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
       #              | |       |   ||||       |         |
        # ln -s /tmp/a
        # cat trace
       # tracer: nop
       #
       # entries-in-buffer/entries-written: 414/414   #P:4
       #
       #                              _-----=> irqs-off
       #                             / _----=> need-resched
       #                            | / _---=> hardirq/softirq
       #                            || / _--=> preempt-depth
       #                            ||| /     delay
       #           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
       #              | |       |   ||||       |         |
                 <idle>-0     [000] d..3  2847.923031: sched_switch: prev_comm=swapper/0 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=bash next_pid=2786 next_prio=120
                  <...>-3114  [001] d..4  2847.923035: sched_switch: prev_comm=ln prev_pid=3114 prev_prio=120 prev_state=x ==> next_comm=swapper/1 next_pid=0 next_prio=120
                   bash-2786  [000] d..3  2847.923535: sched_switch: prev_comm=bash prev_pid=2786 prev_prio=120 prev_state=S ==> next_comm=kworker/0:1 next_pid=34 next_prio=120
            kworker/0:1-34    [000] d..3  2847.923552: sched_switch: prev_comm=kworker/0:1 prev_pid=34 prev_prio=120 prev_state=S ==> next_comm=swapper/0 next_pid=0 next_prio=120
                 <idle>-0     [002] d..3  2847.923554: sched_switch: prev_comm=swapper/2 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=sshd next_pid=2783 next_prio=120
                   sshd-2783  [002] d..3  2847.923660: sched_switch: prev_comm=sshd prev_pid=2783 prev_prio=120 prev_state=S ==> next_comm=swapper/2 next_pid=0 next_prio=120
      
      Still need to update the functions even though the probe itself
      does not need to be registered again when added a new probe.
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      19dd603e
    • Steven Rostedt (Red Hat)'s avatar
      ftrace: Fix the output of enabled_functions debug file · 23ea9c4d
      Steven Rostedt (Red Hat) authored
      The enabled_functions debugfs file was created to be able to see
      what functions have been modified from nops to calling a tracer.
      
      The current method uses the counter in the function record.
      As when a ftrace_ops is registered to a function, its count
      increases. But that doesn't mean that the function is actively
      being traced. /proc/sys/kernel/ftrace_enabled can be set to zero
      which would disable it, as well as something can go wrong and
      we can think its enabled when only the counter is set.
      
      The record's FTRACE_FL_ENABLED flag is set or cleared when its
      function is modified. That is a much more accurate way of knowing
      what function is enabled or not.
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      23ea9c4d
    • Steven Rostedt (Red Hat)'s avatar
      ftrace: Fix locking in register_ftrace_function_probe() · 5ae0bf59
      Steven Rostedt (Red Hat) authored
      The iteration of the ftrace function list and the call to
      ftrace_match_record() need to be protected by the ftrace_lock.
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      5ae0bf59
    • Steven Rostedt (Red Hat)'s avatar
      tracing: Add helper function trace_create_new_event() to remove duplicate code · da511bf3
      Steven Rostedt (Red Hat) authored
      Both __trace_add_new_event() and __trace_early_add_new_event() do
      basically the same thing, except that __trace_add_new_event() does
      a little more.
      
      Instead of having duplicate code between the two functions, add
      a helper function trace_create_new_event() that both can use.
      This will help against having bugs fixed in one function but not
      the other.
      
      Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      da511bf3
    • Masami Hiramatsu's avatar
      tracing: Modify soft-mode only if there's no other referrer · 1cf4c073
      Masami Hiramatsu authored
      Modify soft-mode flag only if no other soft-mode referrer
      (currently only the ftrace triggers) by using a reference
      counter in each ftrace_event_file.
      
      Without this fix, adding and removing several different
      enable/disable_event triggers on the same event clear
      soft-mode bit from the ftrace_event_file. This also
      happens with a typo of glob on setting triggers.
      
      e.g.
      
       # echo vfs_symlink:enable_event:net:netif_rx > set_ftrace_filter
       # cat events/net/netif_rx/enable
       0*
       # echo typo_func:enable_event:net:netif_rx > set_ftrace_filter
       # cat events/net/netif_rx/enable
       0
       # cat set_ftrace_filter
       #### all functions enabled ####
       vfs_symlink:enable_event:net:netif_rx:unlimited
      
      As above, we still have a trigger, but soft-mode is gone.
      
      Link: http://lkml.kernel.org/r/20130509054429.30398.7464.stgit@mhiramat-M0-7522
      
      Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: David Sharp <dhsharp@google.com>
      Cc: Hiraku Toyooka <hiraku.toyooka.gu@hitachi.com>
      Cc: Tom Zanussi <tom.zanussi@intel.com>
      Signed-off-by: default avatarMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      1cf4c073
    • Masami Hiramatsu's avatar
      tracing: Indicate enabled soft-mode in enable file · 30052170
      Masami Hiramatsu authored
      Indicate enabled soft-mode event as "1*" in "enable" file
      for each event, because it can be soft-disabled when disable_event
      trigger is hit.
      
      Link: http://lkml.kernel.org/r/20130509054426.30398.28202.stgit@mhiramat-M0-7522
      
      Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Tom Zanussi <tom.zanussi@intel.com>
      Signed-off-by: default avatarMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      30052170
    • Masami Hiramatsu's avatar
      tracing/kprobes: Fix to increment return event probe hit-count · cce2c8f2
      Masami Hiramatsu authored
      Fix to increment probe hit-count for function return event.
      
      Link: http://lkml.kernel.org/r/20130509054424.30398.34058.stgit@mhiramat-M0-7522
      
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Tom Zanussi <tom.zanussi@intel.com>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
      Signed-off-by: default avatarMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      cce2c8f2
    • Masami Hiramatsu's avatar
      ftrace: Cleanup regex_lock and ftrace_lock around hash updating · 3f2367ba
      Masami Hiramatsu authored
      Cleanup regex_lock and ftrace_lock locking points around
      ftrace_ops hash update code.
      
      The new rule is that regex_lock protects ops->*_hash
      read-update-write code for each ftrace_ops. Usually,
      hash update is done by following sequence.
      
      1. allocate a new local hash and copy the original hash.
      2. update the local hash.
      3. move(actually, copy) back the local hash to ftrace_ops.
      4. update ftrace entries if needed.
      5. release the local hash.
      
      This makes regex_lock protect #1-#4, and ftrace_lock
      to protect #3, #4 and adding and removing ftrace_ops from the
      ftrace_ops_list. The ftrace_lock protects #3 as well because
      the move functions update the entries too.
      
      Link: http://lkml.kernel.org/r/20130509054421.30398.83411.stgit@mhiramat-M0-7522
      
      Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Tom Zanussi <tom.zanussi@intel.com>
      Signed-off-by: default avatarMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      3f2367ba
    • Masami Hiramatsu's avatar
      ftrace, kprobes: Fix a deadlock on ftrace_regex_lock · f04f24fb
      Masami Hiramatsu authored
      Fix a deadlock on ftrace_regex_lock which happens when setting
      an enable_event trigger on dynamic kprobe event as below.
      
      ----
      sh-2.05b# echo p vfs_symlink > kprobe_events
      sh-2.05b# echo vfs_symlink:enable_event:kprobes:p_vfs_symlink_0 > set_ftrace_filter
      
      =============================================
      [ INFO: possible recursive locking detected ]
      3.9.0+ #35 Not tainted
      ---------------------------------------------
      sh/72 is trying to acquire lock:
       (ftrace_regex_lock){+.+.+.}, at: [<ffffffff810ba6c1>] ftrace_set_hash+0x81/0x1f0
      
      but task is already holding lock:
       (ftrace_regex_lock){+.+.+.}, at: [<ffffffff810b7cbd>] ftrace_regex_write.isra.29.part.30+0x3d/0x220
      
      other info that might help us debug this:
       Possible unsafe locking scenario:
      
             CPU0
             ----
        lock(ftrace_regex_lock);
        lock(ftrace_regex_lock);
      
       *** DEADLOCK ***
      ----
      
      To fix that, this introduces a finer regex_lock for each ftrace_ops.
      ftrace_regex_lock is too big of a lock which protects all
      filter/notrace_hash operations, but it doesn't need to be a global
      lock after supporting multiple ftrace_ops because each ftrace_ops
      has its own filter/notrace_hash.
      
      Link: http://lkml.kernel.org/r/20130509054417.30398.84254.stgit@mhiramat-M0-7522
      
      Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Tom Zanussi <tom.zanussi@intel.com>
      Signed-off-by: default avatarMasami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
      [ Added initialization flag and automate mutex initialization for
        non ftrace.c ftrace_probes. ]
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      f04f24fb
  2. 09 May, 2013 3 commits
  3. 03 May, 2013 1 commit
  4. 30 Apr, 2013 27 commits