Commit 33f13714 authored by Paul E. McKenney's avatar Paul E. McKenney Committed by Steven Rostedt (Google)

ftrace: Use asynchronous grace period for register_ftrace_direct()

When running heavy test workloads with KASAN enabled, RCU Tasks grace
periods can extend for many tens of seconds, significantly slowing
trace registration.  Therefore, make the registration-side RCU Tasks
grace period be asynchronous via call_rcu_tasks().

Link: https://lore.kernel.org/linux-trace-kernel/ac05be77-2972-475b-9b57-56bef15aa00a@paulmck-laptopReported-by: default avatarJakub Kicinski <kuba@kernel.org>
Reported-by: default avatarAlexei Starovoitov <ast@kernel.org>
Reported-by: default avatarChris Mason <clm@fb.com>
Reviewed-by: default avatarMasami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
parent c5963a09
...@@ -5366,6 +5366,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long ...@@ -5366,6 +5366,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long
} }
} }
static void register_ftrace_direct_cb(struct rcu_head *rhp)
{
struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
free_ftrace_hash(fhp);
}
/** /**
* register_ftrace_direct - Call a custom trampoline directly * register_ftrace_direct - Call a custom trampoline directly
* for multiple functions registered in @ops * for multiple functions registered in @ops
...@@ -5464,10 +5471,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) ...@@ -5464,10 +5471,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
out_unlock: out_unlock:
mutex_unlock(&direct_mutex); mutex_unlock(&direct_mutex);
if (free_hash && free_hash != EMPTY_HASH) { if (free_hash && free_hash != EMPTY_HASH)
synchronize_rcu_tasks(); call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
free_ftrace_hash(free_hash);
}
if (new_hash) if (new_hash)
free_ftrace_hash(new_hash); free_ftrace_hash(new_hash);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment