Skip to content

L
linux
Commit f511ce14 authored by Paul E. McKenney's avatar Paul E. McKenney
Browse files
Options

Merge branch 'scftorture.2020.08.24a' into HEAD 

scftorture.2020.08.24a: Torture tests for smp_call_function() and friends.
parents cfb2c107 4e88ec4a
Show whitespace changes
Inline Side-by-side
Showing with 982 additions and 208 deletions
Documentation/admin-guide/kernel-parameters.txt
View file @ f511ce14
......@@ -4157,41 +4157,41 @@
rcu_node tree with an eye towards determining
why a new grace period has not yet started.
rcuperf.gp_async= [KNL]
rcuscale.gp_async= [KNL]
Measure performance of asynchronous
grace-period primitives such as call_rcu().
rcuperf.gp_async_max= [KNL]
rcuscale.gp_async_max= [KNL]
Specify the maximum number of outstanding
callbacks per writer thread. When a writer
thread exceeds this limit, it invokes the
corresponding flavor of rcu_barrier() to allow
previously posted callbacks to drain.
rcuperf.gp_exp= [KNL]
rcuscale.gp_exp= [KNL]
Measure performance of expedited synchronous
grace-period primitives.
rcuperf.holdoff= [KNL]
rcuscale.holdoff= [KNL]
Set test-start holdoff period. The purpose of
this parameter is to delay the start of the
test until boot completes in order to avoid
interference.
rcuperf.kfree_rcu_test= [KNL]
rcuscale.kfree_rcu_test= [KNL]
Set to measure performance of kfree_rcu() flooding.
rcuperf.kfree_nthreads= [KNL]
rcuscale.kfree_nthreads= [KNL]
The number of threads running loops of kfree_rcu().
rcuperf.kfree_alloc_num= [KNL]
rcuscale.kfree_alloc_num= [KNL]
Number of allocations and frees done in an iteration.
rcuperf.kfree_loops= [KNL]
Number of loops doing rcuperf.kfree_alloc_num number
rcuscale.kfree_loops= [KNL]
Number of loops doing rcuscale.kfree_alloc_num number
of allocations and frees.
rcuperf.nreaders= [KNL]
rcuscale.nreaders= [KNL]
Set number of RCU readers. The value -1 selects
N, where N is the number of CPUs. A value
"n" less than -1 selects N-n+1, where N is again
......@@ -4200,23 +4200,23 @@
A value of "n" less than or equal to -N selects
a single reader.
rcuperf.nwriters= [KNL]
rcuscale.nwriters= [KNL]
Set number of RCU writers. The values operate
the same as for rcuperf.nreaders.
the same as for rcuscale.nreaders.
N, where N is the number of CPUs
rcuperf.perf_type= [KNL]
rcuscale.perf_type= [KNL]
Specify the RCU implementation to test.
rcuperf.shutdown= [KNL]
rcuscale.shutdown= [KNL]
Shut the system down after performance tests
complete. This is useful for hands-off automated
testing.
rcuperf.verbose= [KNL]
rcuscale.verbose= [KNL]
Enable additional printk() statements.
rcuperf.writer_holdoff= [KNL]
rcuscale.writer_holdoff= [KNL]
Write-side holdoff between grace periods,
in microseconds. The default of zero says
no holdoff.
......@@ -4502,8 +4502,8 @@
refscale.shutdown= [KNL]
Shut down the system at the end of the performance
test. This defaults to 1 (shut it down) when
rcuperf is built into the kernel and to 0 (leave
it running) when rcuperf is built as a module.
refscale is built into the kernel and to 0 (leave
it running) when refscale is built as a module.
refscale.verbose= [KNL]
Enable additional printk() statements.
......@@ -4649,6 +4649,98 @@
Format: integer between 0 and 10
Default is 0.
scftorture.holdoff= [KNL]
Number of seconds to hold off before starting
test. Defaults to zero for module insertion and
to 10 seconds for built-in smp_call_function()
tests.
scftorture.longwait= [KNL]
Request ridiculously long waits randomly selected
up to the chosen limit in seconds. Zero (the
default) disables this feature. Please note
that requesting even small non-zero numbers of
seconds can result in RCU CPU stall warnings,
softlockup complaints, and so on.
scftorture.nthreads= [KNL]
Number of kthreads to spawn to invoke the
smp_call_function() family of functions.
The default of -1 specifies a number of kthreads
equal to the number of CPUs.
scftorture.onoff_holdoff= [KNL]
Number seconds to wait after the start of the
test before initiating CPU-hotplug operations.
scftorture.onoff_interval= [KNL]
Number seconds to wait between successive
CPU-hotplug operations. Specifying zero (which
is the default) disables CPU-hotplug operations.
scftorture.shutdown_secs= [KNL]
The number of seconds following the start of the
test after which to shut down the system. The
default of zero avoids shutting down the system.
Non-zero values are useful for automated tests.
scftorture.stat_interval= [KNL]
The number of seconds between outputting the
current test statistics to the console. A value
of zero disables statistics output.
scftorture.stutter_cpus= [KNL]
The number of jiffies to wait between each change
to the set of CPUs under test.
scftorture.use_cpus_read_lock= [KNL]
Use use_cpus_read_lock() instead of the default
preempt_disable() to disable CPU hotplug
while invoking one of the smp_call_function*()
functions.
scftorture.verbose= [KNL]
Enable additional printk() statements.
scftorture.weight_single= [KNL]
The probability weighting to use for the
smp_call_function_single() function with a zero
"wait" parameter. A value of -1 selects the
default if all other weights are -1. However,
if at least one weight has some other value, a
value of -1 will instead select a weight of zero.
scftorture.weight_single_wait= [KNL]
The probability weighting to use for the
smp_call_function_single() function with a
non-zero "wait" parameter. See weight_single.
scftorture.weight_many= [KNL]
The probability weighting to use for the
smp_call_function_many() function with a zero
"wait" parameter. See weight_single.
Note well that setting a high probability for
this weighting can place serious IPI load
on the system.
scftorture.weight_many_wait= [KNL]
The probability weighting to use for the
smp_call_function_many() function with a
non-zero "wait" parameter. See weight_single
and weight_many.
scftorture.weight_all= [KNL]
The probability weighting to use for the
smp_call_function_all() function with a zero
"wait" parameter. See weight_single and
weight_many.
scftorture.weight_all_wait= [KNL]
The probability weighting to use for the
smp_call_function_all() function with a
non-zero "wait" parameter. See weight_single
and weight_many.
skew_tick= [KNL] Offset the periodic timer tick per cpu to mitigate
xtime_lock contention on larger systems, and/or RCU lock
contention on all systems with CONFIG_MAXSMP set.
......
MAINTAINERS
View file @ f511ce14
......@@ -17510,8 +17510,9 @@ S: Supported
T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
F: Documentation/RCU/torture.rst
F: kernel/locking/locktorture.c
F: kernel/rcu/rcuperf.c
F: kernel/rcu/rcuscale.c
F: kernel/rcu/rcutorture.c
F: kernel/rcu/refscale.c
F: kernel/torture.c
TOSHIBA ACPI EXTRAS DRIVER
......
kernel/Makefile
View file @ f511ce14
......@@ -133,6 +133,8 @@ KASAN_SANITIZE_stackleak.o := n
KCSAN_SANITIZE_stackleak.o := n
KCOV_INSTRUMENT_stackleak.o := n
obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o
$(obj)/configs.o: $(obj)/config_data.gz
targets += config_data.gz
......
kernel/rcu/Kconfig.debug
View file @ f511ce14
......@@ -23,7 +23,7 @@ config TORTURE_TEST
tristate
default n
config RCU_PERF_TEST
config RCU_SCALE_TEST
tristate "performance tests for RCU"
depends on DEBUG_KERNEL
select TORTURE_TEST
......
kernel/rcu/Makefile
View file @ f511ce14
......@@ -11,7 +11,7 @@ obj-y += update.o sync.o
obj-$(CONFIG_TREE_SRCU) += srcutree.o
obj-$(CONFIG_TINY_SRCU) += srcutiny.o
obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o
obj-$(CONFIG_RCU_PERF_TEST) += rcuperf.o
obj-$(CONFIG_RCU_SCALE_TEST) += rcuscale.o
obj-$(CONFIG_RCU_REF_SCALE_TEST) += refscale.o
obj-$(CONFIG_TREE_RCU) += tree.o
obj-$(CONFIG_TINY_RCU) += tiny.o
......
kernel/rcu/rcuperf.c → kernel/rcu/rcuscale.c
View file @ f511ce14
// SPDX-License-Identifier: GPL-2.0+
/*
* Read-Copy Update module-based performance-test facility
* Read-Copy Update module-based scalability-test facility
*
* Copyright (C) IBM Corporation, 2015
*
......@@ -44,13 +44,13 @@
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Paul E. McKenney <paulmck@linux.ibm.com>");
#define PERF_FLAG "-perf:"
#define PERFOUT_STRING(s) \
pr_alert("%s" PERF_FLAG " %s\n", perf_type, s)
#define VERBOSE_PERFOUT_STRING(s) \
do { if (verbose) pr_alert("%s" PERF_FLAG " %s\n", perf_type, s); } while (0)
#define VERBOSE_PERFOUT_ERRSTRING(s) \
do { if (verbose) pr_alert("%s" PERF_FLAG "!!! %s\n", perf_type, s); } while (0)
#define SCALE_FLAG "-scale:"
#define SCALEOUT_STRING(s) \
pr_alert("%s" SCALE_FLAG " %s\n", scale_type, s)
#define VERBOSE_SCALEOUT_STRING(s) \
do { if (verbose) pr_alert("%s" SCALE_FLAG " %s\n", scale_type, s); } while (0)
#define VERBOSE_SCALEOUT_ERRSTRING(s) \
do { if (verbose) pr_alert("%s" SCALE_FLAG "!!! %s\n", scale_type, s); } while (0)
/*
* The intended use cases for the nreaders and nwriters module parameters
......@@ -61,25 +61,25 @@ MODULE_AUTHOR("Paul E. McKenney <paulmck@linux.ibm.com>");
* nr_cpus for a mixed reader/writer test.
*
* 2. Specify the nr_cpus kernel boot parameter, but set
* rcuperf.nreaders to zero. This will set nwriters to the
* rcuscale.nreaders to zero. This will set nwriters to the
* value specified by nr_cpus for an update-only test.
*
* 3. Specify the nr_cpus kernel boot parameter, but set
* rcuperf.nwriters to zero. This will set nreaders to the
* rcuscale.nwriters to zero. This will set nreaders to the
* value specified by nr_cpus for a read-only test.
*
* Various other use cases may of course be specified.
*
* Note that this test's readers are intended only as a test load for
* the writers. The reader performance statistics will be overly
* the writers. The reader scalability statistics will be overly
* pessimistic due to the per-critical-section interrupt disabling,
* test-end checks, and the pair of calls through pointers.
*/
#ifdef MODULE
# define RCUPERF_SHUTDOWN 0
# define RCUSCALE_SHUTDOWN 0
#else
# define RCUPERF_SHUTDOWN 1
# define RCUSCALE_SHUTDOWN 1
#endif
torture_param(bool, gp_async, false, "Use asynchronous GP wait primitives");
......@@ -88,16 +88,16 @@ torture_param(bool, gp_exp, false, "Use expedited GP wait primitives");
torture_param(int, holdoff, 10, "Holdoff time before test start (s)");
torture_param(int, nreaders, -1, "Number of RCU reader threads");
torture_param(int, nwriters, -1, "Number of RCU updater threads");
torture_param(bool, shutdown, RCUPERF_SHUTDOWN,
"Shutdown at end of performance tests.");
torture_param(bool, shutdown, RCUSCALE_SHUTDOWN,
"Shutdown at end of scalability tests.");
torture_param(int, verbose, 1, "Enable verbose debugging printk()s");
torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable");
torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() perf test?");
torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() scale test?");
torture_param(int, kfree_mult, 1, "Multiple of kfree_obj size to allocate.");
static char *perf_type = "rcu";
module_param(perf_type, charp, 0444);
MODULE_PARM_DESC(perf_type, "Type of RCU to performance-test (rcu, srcu, ...)");
static char *scale_type = "rcu";
module_param(scale_type, charp, 0444);
MODULE_PARM_DESC(scale_type, "Type of RCU to scalability-test (rcu, srcu, ...)");
static int nrealreaders;
static int nrealwriters;
......@@ -107,12 +107,12 @@ static struct task_struct *shutdown_task;
static u64 **writer_durations;
static int *writer_n_durations;
static atomic_t n_rcu_perf_reader_started;
static atomic_t n_rcu_perf_writer_started;
static atomic_t n_rcu_perf_writer_finished;
static atomic_t n_rcu_scale_reader_started;
static atomic_t n_rcu_scale_writer_started;
static atomic_t n_rcu_scale_writer_finished;
static wait_queue_head_t shutdown_wq;
static u64 t_rcu_perf_writer_started;
static u64 t_rcu_perf_writer_finished;
static u64 t_rcu_scale_writer_started;
static u64 t_rcu_scale_writer_finished;
static unsigned long b_rcu_gp_test_started;
static unsigned long b_rcu_gp_test_finished;
static DEFINE_PER_CPU(atomic_t, n_async_inflight);
......@@ -124,7 +124,7 @@ static DEFINE_PER_CPU(atomic_t, n_async_inflight);
* Operations vector for selecting different types of tests.
*/
struct rcu_perf_ops {
struct rcu_scale_ops {
int ptype;
void (*init)(void);
void (*cleanup)(void);
......@@ -140,19 +140,19 @@ struct rcu_perf_ops {
const char *name;
};
static struct rcu_perf_ops *cur_ops;
static struct rcu_scale_ops *cur_ops;
/*
* Definitions for rcu perf testing.
* Definitions for rcu scalability testing.
*/
static int rcu_perf_read_lock(void) __acquires(RCU)
static int rcu_scale_read_lock(void) __acquires(RCU)
{
rcu_read_lock();
return 0;
}
static void rcu_perf_read_unlock(int idx) __releases(RCU)
static void rcu_scale_read_unlock(int idx) __releases(RCU)
{
rcu_read_unlock();
}
......@@ -162,15 +162,15 @@ static unsigned long __maybe_unused rcu_no_completed(void)
return 0;
}
static void rcu_sync_perf_init(void)
static void rcu_sync_scale_init(void)
{
}
static struct rcu_perf_ops rcu_ops = {
static struct rcu_scale_ops rcu_ops = {
.ptype = RCU_FLAVOR,
.init = rcu_sync_perf_init,
.readlock = rcu_perf_read_lock,
.readunlock = rcu_perf_read_unlock,
.init = rcu_sync_scale_init,
.readlock = rcu_scale_read_lock,
.readunlock = rcu_scale_read_unlock,
.get_gp_seq = rcu_get_gp_seq,
.gp_diff = rcu_seq_diff,
.exp_completed = rcu_exp_batches_completed,
......@@ -182,23 +182,23 @@ static struct rcu_perf_ops rcu_ops = {
};
/*
* Definitions for srcu perf testing.
* Definitions for srcu scalability testing.
*/
DEFINE_STATIC_SRCU(srcu_ctl_perf);
static struct srcu_struct *srcu_ctlp = &srcu_ctl_perf;
DEFINE_STATIC_SRCU(srcu_ctl_scale);
static struct srcu_struct *srcu_ctlp = &srcu_ctl_scale;
static int srcu_perf_read_lock(void) __acquires(srcu_ctlp)
static int srcu_scale_read_lock(void) __acquires(srcu_ctlp)
{
return srcu_read_lock(srcu_ctlp);
}
static void srcu_perf_read_unlock(int idx) __releases(srcu_ctlp)
static void srcu_scale_read_unlock(int idx) __releases(srcu_ctlp)
{
srcu_read_unlock(srcu_ctlp, idx);
}
static unsigned long srcu_perf_completed(void)
static unsigned long srcu_scale_completed(void)
{
return srcu_batches_completed(srcu_ctlp);
}
......@@ -213,78 +213,78 @@ static void srcu_rcu_barrier(void)
srcu_barrier(srcu_ctlp);
}
static void srcu_perf_synchronize(void)
static void srcu_scale_synchronize(void)
{
synchronize_srcu(srcu_ctlp);
}
static void srcu_perf_synchronize_expedited(void)
static void srcu_scale_synchronize_expedited(void)
{
synchronize_srcu_expedited(srcu_ctlp);
}
static struct rcu_perf_ops srcu_ops = {
static struct rcu_scale_ops srcu_ops = {
.ptype = SRCU_FLAVOR,
.init = rcu_sync_perf_init,
.readlock = srcu_perf_read_lock,
.readunlock = srcu_perf_read_unlock,
.get_gp_seq = srcu_perf_completed,
.init = rcu_sync_scale_init,
.readlock = srcu_scale_read_lock,
.readunlock = srcu_scale_read_unlock,
.get_gp_seq = srcu_scale_completed,
.gp_diff = rcu_seq_diff,
.exp_completed = srcu_perf_completed,
.exp_completed = srcu_scale_completed,
.async = srcu_call_rcu,
.gp_barrier = srcu_rcu_barrier,
.sync = srcu_perf_synchronize,
.exp_sync = srcu_perf_synchronize_expedited,
.sync = srcu_scale_synchronize,
.exp_sync = srcu_scale_synchronize_expedited,
.name = "srcu"
};
static struct srcu_struct srcud;
static void srcu_sync_perf_init(void)
static void srcu_sync_scale_init(void)
{
srcu_ctlp = &srcud;
init_srcu_struct(srcu_ctlp);
}
static void srcu_sync_perf_cleanup(void)
static void srcu_sync_scale_cleanup(void)
{
cleanup_srcu_struct(srcu_ctlp);
}
static struct rcu_perf_ops srcud_ops = {
static struct rcu_scale_ops srcud_ops = {
.ptype = SRCU_FLAVOR,
.init = srcu_sync_perf_init,
.cleanup = srcu_sync_perf_cleanup,
.readlock = srcu_perf_read_lock,
.readunlock = srcu_perf_read_unlock,
.get_gp_seq = srcu_perf_completed,
.init = srcu_sync_scale_init,
.cleanup = srcu_sync_scale_cleanup,
.readlock = srcu_scale_read_lock,
.readunlock = srcu_scale_read_unlock,
.get_gp_seq = srcu_scale_completed,
.gp_diff = rcu_seq_diff,
.exp_completed = srcu_perf_completed,
.exp_completed = srcu_scale_completed,
.async = srcu_call_rcu,
.gp_barrier = srcu_rcu_barrier,
.sync = srcu_perf_synchronize,
.exp_sync = srcu_perf_synchronize_expedited,
.sync = srcu_scale_synchronize,
.exp_sync = srcu_scale_synchronize_expedited,
.name = "srcud"
};
/*
* Definitions for RCU-tasks perf testing.
* Definitions for RCU-tasks scalability testing.
*/
static int tasks_perf_read_lock(void)
static int tasks_scale_read_lock(void)
{
return 0;
}
static void tasks_perf_read_unlock(int idx)
static void tasks_scale_read_unlock(int idx)
{
}
static struct rcu_perf_ops tasks_ops = {
static struct rcu_scale_ops tasks_ops = {
.ptype = RCU_TASKS_FLAVOR,
.init = rcu_sync_perf_init,
.readlock = tasks_perf_read_lock,
.readunlock = tasks_perf_read_unlock,
.init = rcu_sync_scale_init,
.readlock = tasks_scale_read_lock,
.readunlock = tasks_scale_read_unlock,
.get_gp_seq = rcu_no_completed,
.gp_diff = rcu_seq_diff,
.async = call_rcu_tasks,
......@@ -294,7 +294,7 @@ static struct rcu_perf_ops tasks_ops = {
.name = "tasks"
};
static unsigned long rcuperf_seq_diff(unsigned long new, unsigned long old)
static unsigned long rcuscale_seq_diff(unsigned long new, unsigned long old)
{
if (!cur_ops->gp_diff)
return new - old;
......@@ -302,60 +302,60 @@ static unsigned long rcuperf_seq_diff(unsigned long new, unsigned long old)
}
/*
* If performance tests complete, wait for shutdown to commence.
* If scalability tests complete, wait for shutdown to commence.
*/
static void rcu_perf_wait_shutdown(void)
static void rcu_scale_wait_shutdown(void)
{
cond_resched_tasks_rcu_qs();
if (atomic_read(&n_rcu_perf_writer_finished) < nrealwriters)
if (atomic_read(&n_rcu_scale_writer_finished) < nrealwriters)
return;
while (!torture_must_stop())
schedule_timeout_uninterruptible(1);
}
/*
* RCU perf reader kthread. Repeatedly does empty RCU read-side critical
* section, minimizing update-side interference. However, the point of
* this test is not to evaluate reader performance, but instead to serve
* as a test load for update-side performance testing.
* RCU scalability reader kthread. Repeatedly does empty RCU read-side
* critical section, minimizing update-side interference. However, the
* point of this test is not to evaluate reader scalability, but instead
* to serve as a test load for update-side scalability testing.
*/
static int
rcu_perf_reader(void *arg)
rcu_scale_reader(void *arg)
{
unsigned long flags;
int idx;
long me = (long)arg;
VERBOSE_PERFOUT_STRING("rcu_perf_reader task started");
VERBOSE_SCALEOUT_STRING("rcu_scale_reader task started");
set_cpus_allowed_ptr(current, cpumask_of(me % nr_cpu_ids));
set_user_nice(current, MAX_NICE);
atomic_inc(&n_rcu_perf_reader_started);
atomic_inc(&n_rcu_scale_reader_started);
do {
local_irq_save(flags);
idx = cur_ops->readlock();
cur_ops->readunlock(idx);
local_irq_restore(flags);
rcu_perf_wait_shutdown();
rcu_scale_wait_shutdown();
} while (!torture_must_stop());
torture_kthread_stopping("rcu_perf_reader");
torture_kthread_stopping("rcu_scale_reader");
return 0;
}
/*
* Callback function for asynchronous grace periods from rcu_perf_writer().
* Callback function for asynchronous grace periods from rcu_scale_writer().
*/
static void rcu_perf_async_cb(struct rcu_head *rhp)
static void rcu_scale_async_cb(struct rcu_head *rhp)
{
atomic_dec(this_cpu_ptr(&n_async_inflight));
kfree(rhp);
}
/*
* RCU perf writer kthread. Repeatedly does a grace period.
* RCU scale writer kthread. Repeatedly does a grace period.
*/
static int
rcu_perf_writer(void *arg)
rcu_scale_writer(void *arg)
{
int i = 0;
int i_max;
......@@ -366,7 +366,7 @@ rcu_perf_writer(void *arg)
u64 *wdp;
u64 *wdpp = writer_durations[me];
VERBOSE_PERFOUT_STRING("rcu_perf_writer task started");
VERBOSE_SCALEOUT_STRING("rcu_scale_writer task started");
WARN_ON(!wdpp);
set_cpus_allowed_ptr(current, cpumask_of(me % nr_cpu_ids));
sched_set_fifo_low(current);
......@@ -383,8 +383,8 @@ rcu_perf_writer(void *arg)
schedule_timeout_uninterruptible(1);
t = ktime_get_mono_fast_ns();
if (atomic_inc_return(&n_rcu_perf_writer_started) >= nrealwriters) {
t_rcu_perf_writer_started = t;
if (atomic_inc_return(&n_rcu_scale_writer_started) >= nrealwriters) {
t_rcu_scale_writer_started = t;
if (gp_exp) {
b_rcu_gp_test_started =
cur_ops->exp_completed() / 2;
......@@ -404,7 +404,7 @@ rcu_perf_writer(void *arg)
rhp = kmalloc(sizeof(*rhp), GFP_KERNEL);
if (rhp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) {
atomic_inc(this_cpu_ptr(&n_async_inflight));
cur_ops->async(rhp, rcu_perf_async_cb);
cur_ops->async(rhp, rcu_scale_async_cb);
rhp = NULL;
} else if (!kthread_should_stop()) {
cur_ops->gp_barrier();
......@@ -421,19 +421,19 @@ rcu_perf_writer(void *arg)
*wdp = t - *wdp;
i_max = i;
if (!started &&
atomic_read(&n_rcu_perf_writer_started) >= nrealwriters)
atomic_read(&n_rcu_scale_writer_started) >= nrealwriters)
started = true;
if (!done && i >= MIN_MEAS) {
done = true;
sched_set_normal(current, 0);
pr_alert("%s%s rcu_perf_writer %ld has %d measurements\n",
perf_type, PERF_FLAG, me, MIN_MEAS);
if (atomic_inc_return(&n_rcu_perf_writer_finished) >=
pr_alert("%s%s rcu_scale_writer %ld has %d measurements\n",
scale_type, SCALE_FLAG, me, MIN_MEAS);
if (atomic_inc_return(&n_rcu_scale_writer_finished) >=
nrealwriters) {
schedule_timeout_interruptible(10);
rcu_ftrace_dump(DUMP_ALL);
PERFOUT_STRING("Test complete");
t_rcu_perf_writer_finished = t;
SCALEOUT_STRING("Test complete");
t_rcu_scale_writer_finished = t;
if (gp_exp) {
b_rcu_gp_test_finished =
cur_ops->exp_completed() / 2;
......@@ -448,30 +448,30 @@ rcu_perf_writer(void *arg)
}
}
if (done && !alldone &&
atomic_read(&n_rcu_perf_writer_finished) >= nrealwriters)
atomic_read(&n_rcu_scale_writer_finished) >= nrealwriters)
alldone = true;
if (started && !alldone && i < MAX_MEAS - 1)
i++;
rcu_perf_wait_shutdown();
rcu_scale_wait_shutdown();
} while (!torture_must_stop());
if (gp_async) {
cur_ops->gp_barrier();
}
writer_n_durations[me] = i_max;
torture_kthread_stopping("rcu_perf_writer");
torture_kthread_stopping("rcu_scale_writer");
return 0;
}
static void
rcu_perf_print_module_parms(struct rcu_perf_ops *cur_ops, const char *tag)
rcu_scale_print_module_parms(struct rcu_scale_ops *cur_ops, const char *tag)
{
pr_alert("%s" PERF_FLAG
pr_alert("%s" SCALE_FLAG
"--- %s: nreaders=%d nwriters=%d verbose=%d shutdown=%d\n",
perf_type, tag, nrealreaders, nrealwriters, verbose, shutdown);
scale_type, tag, nrealreaders, nrealwriters, verbose, shutdown);
}
static void
rcu_perf_cleanup(void)
rcu_scale_cleanup(void)
{
int i;
int j;
......@@ -484,11 +484,11 @@ rcu_perf_cleanup(void)
* during the mid-boot phase, so have to wait till the end.
*/
if (rcu_gp_is_expedited() && !rcu_gp_is_normal() && !gp_exp)
VERBOSE_PERFOUT_ERRSTRING("All grace periods expedited, no normal ones to measure!");
VERBOSE_SCALEOUT_ERRSTRING("All grace periods expedited, no normal ones to measure!");
if (rcu_gp_is_normal() && gp_exp)
VERBOSE_PERFOUT_ERRSTRING("All grace periods normal, no expedited ones to measure!");
VERBOSE_SCALEOUT_ERRSTRING("All grace periods normal, no expedited ones to measure!");
if (gp_exp && gp_async)
VERBOSE_PERFOUT_ERRSTRING("No expedited async GPs, so went with async!");
VERBOSE_SCALEOUT_ERRSTRING("No expedited async GPs, so went with async!");
if (torture_cleanup_begin())
return;
......@@ -499,29 +499,29 @@ rcu_perf_cleanup(void)
if (reader_tasks) {
for (i = 0; i < nrealreaders; i++)
torture_stop_kthread(rcu_perf_reader,
torture_stop_kthread(rcu_scale_reader,
reader_tasks[i]);
kfree(reader_tasks);
}
if (writer_tasks) {
for (i = 0; i < nrealwriters; i++) {
torture_stop_kthread(rcu_perf_writer,
torture_stop_kthread(rcu_scale_writer,
writer_tasks[i]);
if (!writer_n_durations)
continue;
j = writer_n_durations[i];
pr_alert("%s%s writer %d gps: %d\n",
perf_type, PERF_FLAG, i, j);
scale_type, SCALE_FLAG, i, j);
ngps += j;
}
pr_alert("%s%s start: %llu end: %llu duration: %llu gps: %d batches: %ld\n",
perf_type, PERF_FLAG,
t_rcu_perf_writer_started, t_rcu_perf_writer_finished,
t_rcu_perf_writer_finished -
t_rcu_perf_writer_started,
scale_type, SCALE_FLAG,
t_rcu_scale_writer_started, t_rcu_scale_writer_finished,
t_rcu_scale_writer_finished -
t_rcu_scale_writer_started,
ngps,
rcuperf_seq_diff(b_rcu_gp_test_finished,
rcuscale_seq_diff(b_rcu_gp_test_finished,
b_rcu_gp_test_started));
for (i = 0; i < nrealwriters; i++) {
if (!writer_durations)
......@@ -534,7 +534,7 @@ rcu_perf_cleanup(void)
for (j = 0; j <= writer_n_durations[i]; j++) {
wdp = &wdpp[j];
pr_alert("%s%s %4d writer-duration: %5d %llu\n",
perf_type, PERF_FLAG,
scale_type, SCALE_FLAG,
i, j, *wdp);
if (j % 100 == 0)
schedule_timeout_uninterruptible(1);
......@@ -573,22 +573,22 @@ static int compute_real(int n)
}
/*
* RCU perf shutdown kthread. Just waits to be awakened, then shuts
* RCU scalability shutdown kthread. Just waits to be awakened, then shuts
* down system.
*/
static int
rcu_perf_shutdown(void *arg)
rcu_scale_shutdown(void *arg)
{
wait_event(shutdown_wq,
atomic_read(&n_rcu_perf_writer_finished) >= nrealwriters);
atomic_read(&n_rcu_scale_writer_finished) >= nrealwriters);
smp_mb(); /* Wake before output. */
rcu_perf_cleanup();
rcu_scale_cleanup();
kernel_power_off();
return -EINVAL;
}
/*
* kfree_rcu() performance tests: Start a kfree_rcu() loop on all CPUs for number
* kfree_rcu() scalability tests: Start a kfree_rcu() loop on all CPUs for number
* of iterations and measure total time and number of GP for all iterations to complete.
*/
......@@ -598,8 +598,8 @@ torture_param(int, kfree_loops, 10, "Number of loops doing kfree_alloc_num alloc
static struct task_struct **kfree_reader_tasks;
static int kfree_nrealthreads;
static atomic_t n_kfree_perf_thread_started;
static atomic_t n_kfree_perf_thread_ended;
static atomic_t n_kfree_scale_thread_started;
static atomic_t n_kfree_scale_thread_ended;
struct kfree_obj {
char kfree_obj[8];
......@@ -607,7 +607,7 @@ struct kfree_obj {
};
static int
kfree_perf_thread(void *arg)
kfree_scale_thread(void *arg)
{
int i, loop = 0;
long me = (long)arg;
......@@ -615,13 +615,13 @@ kfree_perf_thread(void *arg)
u64 start_time, end_time;
long long mem_begin, mem_during = 0;
VERBOSE_PERFOUT_STRING("kfree_perf_thread task started");
VERBOSE_SCALEOUT_STRING("kfree_scale_thread task started");
set_cpus_allowed_ptr(current, cpumask_of(me % nr_cpu_ids));
set_user_nice(current, MAX_NICE);
start_time = ktime_get_mono_fast_ns();
if (atomic_inc_return(&n_kfree_perf_thread_started) >= kfree_nrealthreads) {
if (atomic_inc_return(&n_kfree_scale_thread_started) >= kfree_nrealthreads) {
if (gp_exp)
b_rcu_gp_test_started = cur_ops->exp_completed() / 2;
else
......@@ -646,7 +646,7 @@ kfree_perf_thread(void *arg)
cond_resched();
} while (!torture_must_stop() && ++loop < kfree_loops);
if (atomic_inc_return(&n_kfree_perf_thread_ended) >= kfree_nrealthreads) {
if (atomic_inc_return(&n_kfree_scale_thread_ended) >= kfree_nrealthreads) {
end_time = ktime_get_mono_fast_ns();
if (gp_exp)
......@@ -656,7 +656,7 @@ kfree_perf_thread(void *arg)
pr_alert("Total time taken by all kfree'ers: %llu ns, loops: %d, batches: %ld, memory footprint: %lldMB\n",
(unsigned long long)(end_time - start_time), kfree_loops,
rcuperf_seq_diff(b_rcu_gp_test_finished, b_rcu_gp_test_started),
rcuscale_seq_diff(b_rcu_gp_test_finished, b_rcu_gp_test_started),
(mem_begin - mem_during) >> (20 - PAGE_SHIFT));
if (shutdown) {
......@@ -665,12 +665,12 @@ kfree_perf_thread(void *arg)
}
}
torture_kthread_stopping("kfree_perf_thread");
torture_kthread_stopping("kfree_scale_thread");
return 0;
}
static void
kfree_perf_cleanup(void)
kfree_scale_cleanup(void)
{
int i;
......@@ -679,7 +679,7 @@ kfree_perf_cleanup(void)
if (kfree_reader_tasks) {
for (i = 0; i < kfree_nrealthreads; i++)
torture_stop_kthread(kfree_perf_thread,
torture_stop_kthread(kfree_scale_thread,
kfree_reader_tasks[i]);
kfree(kfree_reader_tasks);
}
......@@ -691,20 +691,20 @@ kfree_perf_cleanup(void)
* shutdown kthread. Just waits to be awakened, then shuts down system.
*/
static int
kfree_perf_shutdown(void *arg)
kfree_scale_shutdown(void *arg)
{
wait_event(shutdown_wq,
atomic_read(&n_kfree_perf_thread_ended) >= kfree_nrealthreads);
atomic_read(&n_kfree_scale_thread_ended) >= kfree_nrealthreads);
smp_mb(); /* Wake before output. */
kfree_perf_cleanup();
kfree_scale_cleanup();
kernel_power_off();
return -EINVAL;
}
static int __init
kfree_perf_init(void)
kfree_scale_init(void)
{
long i;
int firsterr = 0;
......@@ -713,7 +713,7 @@ kfree_perf_init(void)
/* Start up the kthreads. */
if (shutdown) {
init_waitqueue_head(&shutdown_wq);
firsterr = torture_create_kthread(kfree_perf_shutdown, NULL,
firsterr = torture_create_kthread(kfree_scale_shutdown, NULL,
shutdown_task);
if (firsterr)
goto unwind;
......@@ -730,13 +730,13 @@ kfree_perf_init(void)
}
for (i = 0; i < kfree_nrealthreads; i++) {
firsterr = torture_create_kthread(kfree_perf_thread, (void *)i,
firsterr = torture_create_kthread(kfree_scale_thread, (void *)i,
kfree_reader_tasks[i]);
if (firsterr)
goto unwind;
}
while (atomic_read(&n_kfree_perf_thread_started) < kfree_nrealthreads)
while (atomic_read(&n_kfree_scale_thread_started) < kfree_nrealthreads)
schedule_timeout_uninterruptible(1);
torture_init_end();
......@@ -744,35 +744,35 @@ kfree_perf_init(void)
unwind:
torture_init_end();
kfree_perf_cleanup();
kfree_scale_cleanup();
return firsterr;
}
static int __init
rcu_perf_init(void)
rcu_scale_init(void)
{
long i;
int firsterr = 0;
static struct rcu_perf_ops *perf_ops[] = {
static struct rcu_scale_ops *scale_ops[] = {
&rcu_ops, &srcu_ops, &srcud_ops, &tasks_ops,
};
if (!torture_init_begin(perf_type, verbose))
if (!torture_init_begin(scale_type, verbose))
return -EBUSY;
/* Process args and tell the world that the perf'er is on the job. */
for (i = 0; i < ARRAY_SIZE(perf_ops); i++) {
cur_ops = perf_ops[i];
if (strcmp(perf_type, cur_ops->name) == 0)
/* Process args and announce that the scalability'er is on the job. */
for (i = 0; i < ARRAY_SIZE(scale_ops); i++) {
cur_ops = scale_ops[i];
if (strcmp(scale_type, cur_ops->name) == 0)
break;
}
if (i == ARRAY_SIZE(perf_ops)) {
pr_alert("rcu-perf: invalid perf type: \"%s\"\n", perf_type);
pr_alert("rcu-perf types:");
for (i = 0; i < ARRAY_SIZE(perf_ops); i++)
pr_cont(" %s", perf_ops[i]->name);
if (i == ARRAY_SIZE(scale_ops)) {
pr_alert("rcu-scale: invalid scale type: \"%s\"\n", scale_type);
pr_alert("rcu-scale types:");
for (i = 0; i < ARRAY_SIZE(scale_ops); i++)
pr_cont(" %s", scale_ops[i]->name);
pr_cont("\n");
WARN_ON(!IS_MODULE(CONFIG_RCU_PERF_TEST));
WARN_ON(!IS_MODULE(CONFIG_RCU_SCALE_TEST));
firsterr = -EINVAL;
cur_ops = NULL;
goto unwind;
......@@ -781,20 +781,20 @@ rcu_perf_init(void)
cur_ops->init();
if (kfree_rcu_test)
return kfree_perf_init();
return kfree_scale_init();
nrealwriters = compute_real(nwriters);
nrealreaders = compute_real(nreaders);
atomic_set(&n_rcu_perf_reader_started, 0);
atomic_set(&n_rcu_perf_writer_started, 0);
atomic_set(&n_rcu_perf_writer_finished, 0);
rcu_perf_print_module_parms(cur_ops, "Start of test");
atomic_set(&n_rcu_scale_reader_started, 0);
atomic_set(&n_rcu_scale_writer_started, 0);
atomic_set(&n_rcu_scale_writer_finished, 0);
rcu_scale_print_module_parms(cur_ops, "Start of test");
/* Start up the kthreads. */
if (shutdown) {
init_waitqueue_head(&shutdown_wq);
firsterr = torture_create_kthread(rcu_perf_shutdown, NULL,
firsterr = torture_create_kthread(rcu_scale_shutdown, NULL,
shutdown_task);
if (firsterr)
goto unwind;
......@@ -803,17 +803,17 @@ rcu_perf_init(void)
reader_tasks = kcalloc(nrealreaders, sizeof(reader_tasks[0]),
GFP_KERNEL);
if (reader_tasks == NULL) {
VERBOSE_PERFOUT_ERRSTRING("out of memory");
VERBOSE_SCALEOUT_ERRSTRING("out of memory");
firsterr = -ENOMEM;
goto unwind;
}
for (i = 0; i < nrealreaders; i++) {
firsterr = torture_create_kthread(rcu_perf_reader, (void *)i,
firsterr = torture_create_kthread(rcu_scale_reader, (void *)i,
reader_tasks[i]);
if (firsterr)
goto unwind;
}
while (atomic_read(&n_rcu_perf_reader_started) < nrealreaders)
while (atomic_read(&n_rcu_scale_reader_started) < nrealreaders)
schedule_timeout_uninterruptible(1);
writer_tasks = kcalloc(nrealwriters, sizeof(reader_tasks[0]),
GFP_KERNEL);
......@@ -823,7 +823,7 @@ rcu_perf_init(void)
kcalloc(nrealwriters, sizeof(*writer_n_durations),
GFP_KERNEL);
if (!writer_tasks || !writer_durations || !writer_n_durations) {
VERBOSE_PERFOUT_ERRSTRING("out of memory");
VERBOSE_SCALEOUT_ERRSTRING("out of memory");
firsterr = -ENOMEM;
goto unwind;
}
......@@ -835,7 +835,7 @@ rcu_perf_init(void)
firsterr = -ENOMEM;
goto unwind;
}
firsterr = torture_create_kthread(rcu_perf_writer, (void *)i,
firsterr = torture_create_kthread(rcu_scale_writer, (void *)i,
writer_tasks[i]);
if (firsterr)
goto unwind;
......@@ -845,9 +845,9 @@ rcu_perf_init(void)
unwind:
torture_init_end();
rcu_perf_cleanup();
rcu_scale_cleanup();
return firsterr;
}
module_init(rcu_perf_init);
module_exit(rcu_perf_cleanup);
module_init(rcu_scale_init);
module_exit(rcu_scale_cleanup);
kernel/scftorture.c 0 → 100644
View file @ f511ce14
// SPDX-License-Identifier: GPL-2.0+
//
// Torture test for smp_call_function() and friends.
//
// Copyright (C) Facebook, 2020.
//
// Author: Paul E. McKenney <paulmck@kernel.org>
#define pr_fmt(fmt) fmt
#include <linux/atomic.h>
#include <linux/bitops.h>
#include <linux/completion.h>
#include <linux/cpu.h>
#include <linux/delay.h>
#include <linux/err.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/kthread.h>
#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/notifier.h>
#include <linux/percpu.h>
#include <linux/rcupdate.h>
#include <linux/rcupdate_trace.h>
#include <linux/reboot.h>
#include <linux/sched.h>
#include <linux/spinlock.h>
#include <linux/smp.h>
#include <linux/stat.h>
#include <linux/srcu.h>
#include <linux/slab.h>
#include <linux/torture.h>
#include <linux/types.h>
#define SCFTORT_STRING "scftorture"
#define SCFTORT_FLAG SCFTORT_STRING ": "
#define SCFTORTOUT(s, x...) \
pr_alert(SCFTORT_FLAG s, ## x)
#define VERBOSE_SCFTORTOUT(s, x...) \
do { if (verbose) pr_alert(SCFTORT_FLAG s, ## x); } while (0)
#define VERBOSE_SCFTORTOUT_ERRSTRING(s, x...) \
do { if (verbose) pr_alert(SCFTORT_FLAG "!!! " s, ## x); } while (0)
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Paul E. McKenney <paulmck@kernel.org>");
// Wait until there are multiple CPUs before starting test.
torture_param(int, holdoff, IS_BUILTIN(CONFIG_SCF_TORTURE_TEST) ? 10 : 0,
"Holdoff time before test start (s)");
torture_param(int, longwait, 0, "Include ridiculously long waits? (seconds)");
torture_param(int, nthreads, -1, "# threads, defaults to -1 for all CPUs.");
torture_param(int, onoff_holdoff, 0, "Time after boot before CPU hotplugs (s)");
torture_param(int, onoff_interval, 0, "Time between CPU hotplugs (s), 0=disable");
torture_param(int, shutdown_secs, 0, "Shutdown time (ms), <= zero to disable.");
torture_param(int, stat_interval, 60, "Number of seconds between stats printk()s.");
torture_param(int, stutter_cpus, 5, "Number of jiffies to change CPUs under test, 0=disable");
torture_param(bool, use_cpus_read_lock, 0, "Use cpus_read_lock() to exclude CPU hotplug.");
torture_param(int, verbose, 0, "Enable verbose debugging printk()s");
torture_param(int, weight_single, -1, "Testing weight for single-CPU no-wait operations.");
torture_param(int, weight_single_wait, -1, "Testing weight for single-CPU operations.");
torture_param(int, weight_many, -1, "Testing weight for multi-CPU no-wait operations.");
torture_param(int, weight_many_wait, -1, "Testing weight for multi-CPU operations.");
torture_param(int, weight_all, -1, "Testing weight for all-CPU no-wait operations.");
torture_param(int, weight_all_wait, -1, "Testing weight for all-CPU operations.");
char *torture_type = "";
#ifdef MODULE
# define SCFTORT_SHUTDOWN 0
#else
# define SCFTORT_SHUTDOWN 1
#endif
torture_param(bool, shutdown, SCFTORT_SHUTDOWN, "Shutdown at end of torture test.");
struct scf_statistics {
struct task_struct *task;
int cpu;
long long n_single;
long long n_single_ofl;
long long n_single_wait;
long long n_single_wait_ofl;
long long n_many;
long long n_many_wait;
long long n_all;
long long n_all_wait;
};
static struct scf_statistics *scf_stats_p;
static struct task_struct *scf_torture_stats_task;
static DEFINE_PER_CPU(long long, scf_invoked_count);
// Data for random primitive selection
#define SCF_PRIM_SINGLE 0
#define SCF_PRIM_MANY 1
#define SCF_PRIM_ALL 2
#define SCF_NPRIMS (2 * 3) // Need wait and no-wait versions of each.
static char *scf_prim_name[] = {
"smp_call_function_single",
"smp_call_function_many",
"smp_call_function",
};
struct scf_selector {
unsigned long scfs_weight;
int scfs_prim;
bool scfs_wait;
};
static struct scf_selector scf_sel_array[SCF_NPRIMS];
static int scf_sel_array_len;
static unsigned long scf_sel_totweight;
// Communicate between caller and handler.
struct scf_check {
bool scfc_in;
bool scfc_out;
int scfc_cpu; // -1 for not _single().
bool scfc_wait;
};
// Use to wait for all threads to start.
static atomic_t n_started;
static atomic_t n_errs;
static atomic_t n_mb_in_errs;
static atomic_t n_mb_out_errs;
static atomic_t n_alloc_errs;
static bool scfdone;
static char *bangstr = "";
static DEFINE_TORTURE_RANDOM_PERCPU(scf_torture_rand);
// Print torture statistics. Caller must ensure serialization.
static void scf_torture_stats_print(void)
{
int cpu;
int i;
long long invoked_count = 0;
bool isdone = READ_ONCE(scfdone);
struct scf_statistics scfs = {};
for_each_possible_cpu(cpu)
invoked_count += data_race(per_cpu(scf_invoked_count, cpu));
for (i = 0; i < nthreads; i++) {
scfs.n_single += scf_stats_p[i].n_single;
scfs.n_single_ofl += scf_stats_p[i].n_single_ofl;
scfs.n_single_wait += scf_stats_p[i].n_single_wait;
scfs.n_single_wait_ofl += scf_stats_p[i].n_single_wait_ofl;
scfs.n_many += scf_stats_p[i].n_many;
scfs.n_many_wait += scf_stats_p[i].n_many_wait;
scfs.n_all += scf_stats_p[i].n_all;
scfs.n_all_wait += scf_stats_p[i].n_all_wait;
}
if (atomic_read(&n_errs) || atomic_read(&n_mb_in_errs) ||
atomic_read(&n_mb_out_errs) || atomic_read(&n_alloc_errs))
bangstr = "!!! ";
pr_alert("%s %sscf_invoked_count %s: %lld single: %lld/%lld single_ofl: %lld/%lld many: %lld/%lld all: %lld/%lld ",
SCFTORT_FLAG, bangstr, isdone ? "VER" : "ver", invoked_count,
scfs.n_single, scfs.n_single_wait, scfs.n_single_ofl, scfs.n_single_wait_ofl,
scfs.n_many, scfs.n_many_wait, scfs.n_all, scfs.n_all_wait);
torture_onoff_stats();
pr_cont("ste: %d stnmie: %d stnmoe: %d staf: %d\n", atomic_read(&n_errs),
atomic_read(&n_mb_in_errs), atomic_read(&n_mb_out_errs),
atomic_read(&n_alloc_errs));
}
// Periodically prints torture statistics, if periodic statistics printing
// was specified via the stat_interval module parameter.
static int
scf_torture_stats(void *arg)
{
VERBOSE_TOROUT_STRING("scf_torture_stats task started");
do {
schedule_timeout_interruptible(stat_interval * HZ);
scf_torture_stats_print();
torture_shutdown_absorb("scf_torture_stats");
} while (!torture_must_stop());
torture_kthread_stopping("scf_torture_stats");
return 0;
}
// Add a primitive to the scf_sel_array[].
static void scf_sel_add(unsigned long weight, int prim, bool wait)
{
struct scf_selector *scfsp = &scf_sel_array[scf_sel_array_len];
// If no weight, if array would overflow, if computing three-place
// percentages would overflow, or if the scf_prim_name[] array would
// overflow, don't bother. In the last three two cases, complain.
if (!weight ||
WARN_ON_ONCE(scf_sel_array_len >= ARRAY_SIZE(scf_sel_array)) ||
WARN_ON_ONCE(0 - 100000 * weight <= 100000 * scf_sel_totweight) ||
WARN_ON_ONCE(prim >= ARRAY_SIZE(scf_prim_name)))
return;
scf_sel_totweight += weight;
scfsp->scfs_weight = scf_sel_totweight;
scfsp->scfs_prim = prim;
scfsp->scfs_wait = wait;
scf_sel_array_len++;
}
// Dump out weighting percentages for scf_prim_name[] array.
static void scf_sel_dump(void)
{
int i;
unsigned long oldw = 0;
struct scf_selector *scfsp;
unsigned long w;
for (i = 0; i < scf_sel_array_len; i++) {
scfsp = &scf_sel_array[i];
w = (scfsp->scfs_weight - oldw) * 100000 / scf_sel_totweight;
pr_info("%s: %3lu.%03lu %s(%s)\n", __func__, w / 1000, w % 1000,
scf_prim_name[scfsp->scfs_prim],
scfsp->scfs_wait ? "wait" : "nowait");
oldw = scfsp->scfs_weight;
}
}
// Randomly pick a primitive and wait/nowait, based on weightings.
static struct scf_selector *scf_sel_rand(struct torture_random_state *trsp)
{
int i;
unsigned long w = torture_random(trsp) % (scf_sel_totweight + 1);
for (i = 0; i < scf_sel_array_len; i++)
if (scf_sel_array[i].scfs_weight >= w)
return &scf_sel_array[i];
WARN_ON_ONCE(1);
return &scf_sel_array[0];
}
// Update statistics and occasionally burn up mass quantities of CPU time,
// if told to do so via scftorture.longwait. Otherwise, occasionally burn
// a little bit.
static void scf_handler(void *scfc_in)
{
int i;
int j;
unsigned long r = torture_random(this_cpu_ptr(&scf_torture_rand));
struct scf_check *scfcp = scfc_in;
if (likely(scfcp)) {
WRITE_ONCE(scfcp->scfc_out, false); // For multiple receivers.
if (WARN_ON_ONCE(unlikely(!READ_ONCE(scfcp->scfc_in))))
atomic_inc(&n_mb_in_errs);
}
this_cpu_inc(scf_invoked_count);
if (longwait <= 0) {
if (!(r & 0xffc0))
udelay(r & 0x3f);
goto out;
}
if (r & 0xfff)
goto out;
r = (r >> 12);
if (longwait <= 0) {
udelay((r & 0xff) + 1);
goto out;
}
r = r % longwait + 1;
for (i = 0; i < r; i++) {
for (j = 0; j < 1000; j++) {
udelay(1000);
cpu_relax();
}
}
out:
if (unlikely(!scfcp))
return;
if (scfcp->scfc_wait)
WRITE_ONCE(scfcp->scfc_out, true);
else
kfree(scfcp);
}
// As above, but check for correct CPU.
static void scf_handler_1(void *scfc_in)
{
struct scf_check *scfcp = scfc_in;
if (likely(scfcp) && WARN_ONCE(smp_processor_id() != scfcp->scfc_cpu, "%s: Wanted CPU %d got CPU %d\n", __func__, scfcp->scfc_cpu, smp_processor_id())) {
atomic_inc(&n_errs);
}
scf_handler(scfcp);
}
// Randomly do an smp_call_function*() invocation.
static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_random_state *trsp)
{
uintptr_t cpu;
int ret = 0;
struct scf_check *scfcp = NULL;
struct scf_selector *scfsp = scf_sel_rand(trsp);
if (use_cpus_read_lock)
cpus_read_lock();
else
preempt_disable();
if (scfsp->scfs_prim == SCF_PRIM_SINGLE || scfsp->scfs_wait) {
scfcp = kmalloc(sizeof(*scfcp), GFP_ATOMIC);
if (WARN_ON_ONCE(!scfcp)) {
atomic_inc(&n_alloc_errs);
} else {
scfcp->scfc_cpu = -1;
scfcp->scfc_wait = scfsp->scfs_wait;
scfcp->scfc_out = false;
}
}
switch (scfsp->scfs_prim) {
case SCF_PRIM_SINGLE:
cpu = torture_random(trsp) % nr_cpu_ids;
if (scfsp->scfs_wait)
scfp->n_single_wait++;
else
scfp->n_single++;
if (scfcp) {
scfcp->scfc_cpu = cpu;
barrier(); // Prevent race-reduction compiler optimizations.
scfcp->scfc_in = true;
}
ret = smp_call_function_single(cpu, scf_handler_1, (void *)scfcp, scfsp->scfs_wait);
if (ret) {
if (scfsp->scfs_wait)
scfp->n_single_wait_ofl++;
else
scfp->n_single_ofl++;
kfree(scfcp);
scfcp = NULL;
}
break;
case SCF_PRIM_MANY:
if (scfsp->scfs_wait)
scfp->n_many_wait++;
else
scfp->n_many++;
if (scfcp) {
barrier(); // Prevent race-reduction compiler optimizations.
scfcp->scfc_in = true;
}
smp_call_function_many(cpu_online_mask, scf_handler, scfcp, scfsp->scfs_wait);
break;
case SCF_PRIM_ALL:
if (scfsp->scfs_wait)
scfp->n_all_wait++;
else
scfp->n_all++;
if (scfcp) {
barrier(); // Prevent race-reduction compiler optimizations.
scfcp->scfc_in = true;
}
smp_call_function(scf_handler, scfcp, scfsp->scfs_wait);
break;
default:
WARN_ON_ONCE(1);
if (scfcp)
scfcp->scfc_out = true;
}
if (scfcp && scfsp->scfs_wait) {
if (WARN_ON_ONCE((num_online_cpus() > 1 || scfsp->scfs_prim == SCF_PRIM_SINGLE) &&
!scfcp->scfc_out))
atomic_inc(&n_mb_out_errs); // Leak rather than trash!
else
kfree(scfcp);
barrier(); // Prevent race-reduction compiler optimizations.
}
if (use_cpus_read_lock)
cpus_read_unlock();
else
preempt_enable();
if (!(torture_random(trsp) & 0xfff))
schedule_timeout_uninterruptible(1);
}
// SCF test kthread. Repeatedly does calls to members of the
// smp_call_function() family of functions.
static int scftorture_invoker(void *arg)
{
int cpu;
DEFINE_TORTURE_RANDOM(rand);
struct scf_statistics *scfp = (struct scf_statistics *)arg;
bool was_offline = false;
VERBOSE_SCFTORTOUT("scftorture_invoker %d: task started", scfp->cpu);
cpu = scfp->cpu % nr_cpu_ids;
set_cpus_allowed_ptr(current, cpumask_of(cpu));
set_user_nice(current, MAX_NICE);
if (holdoff)
schedule_timeout_interruptible(holdoff * HZ);
VERBOSE_SCFTORTOUT("scftorture_invoker %d: Waiting for all SCF torturers from cpu %d", scfp->cpu, smp_processor_id());
// Make sure that the CPU is affinitized appropriately during testing.
WARN_ON_ONCE(smp_processor_id() != scfp->cpu);
if (!atomic_dec_return(&n_started))
while (atomic_read_acquire(&n_started)) {
if (torture_must_stop()) {
VERBOSE_SCFTORTOUT("scftorture_invoker %d ended before starting", scfp->cpu);
goto end;
}
schedule_timeout_uninterruptible(1);
}
VERBOSE_SCFTORTOUT("scftorture_invoker %d started", scfp->cpu);
do {
scftorture_invoke_one(scfp, &rand);
while (cpu_is_offline(cpu) && !torture_must_stop()) {
schedule_timeout_interruptible(HZ / 5);
was_offline = true;
}
if (was_offline) {
set_cpus_allowed_ptr(current, cpumask_of(cpu));
was_offline = false;
}
cond_resched();
} while (!torture_must_stop());
VERBOSE_SCFTORTOUT("scftorture_invoker %d ended", scfp->cpu);
end:
torture_kthread_stopping("scftorture_invoker");
return 0;
}
static void
scftorture_print_module_parms(const char *tag)
{
pr_alert(SCFTORT_FLAG
"--- %s: verbose=%d holdoff=%d longwait=%d nthreads=%d onoff_holdoff=%d onoff_interval=%d shutdown_secs=%d stat_interval=%d stutter_cpus=%d use_cpus_read_lock=%d, weight_single=%d, weight_single_wait=%d, weight_many=%d, weight_many_wait=%d, weight_all=%d, weight_all_wait=%d\n", tag,
verbose, holdoff, longwait, nthreads, onoff_holdoff, onoff_interval, shutdown, stat_interval, stutter_cpus, use_cpus_read_lock, weight_single, weight_single_wait, weight_many, weight_many_wait, weight_all, weight_all_wait);
}
static void scf_cleanup_handler(void *unused)
{
}
static void scf_torture_cleanup(void)
{
int i;
if (torture_cleanup_begin())
return;
WRITE_ONCE(scfdone, true);
if (nthreads)
for (i = 0; i < nthreads; i++)
torture_stop_kthread("scftorture_invoker", scf_stats_p[i].task);
else
goto end;
smp_call_function(scf_cleanup_handler, NULL, 0);
torture_stop_kthread(scf_torture_stats, scf_torture_stats_task);
scf_torture_stats_print(); // -After- the stats thread is stopped!
kfree(scf_stats_p); // -After- the last stats print has completed!
scf_stats_p = NULL;
if (atomic_read(&n_errs) || atomic_read(&n_mb_in_errs) || atomic_read(&n_mb_out_errs))
scftorture_print_module_parms("End of test: FAILURE");
else if (torture_onoff_failures())
scftorture_print_module_parms("End of test: LOCK_HOTPLUG");
else
scftorture_print_module_parms("End of test: SUCCESS");
end:
torture_cleanup_end();
}
static int __init scf_torture_init(void)
{
long i;
int firsterr = 0;
unsigned long weight_single1 = weight_single;
unsigned long weight_single_wait1 = weight_single_wait;
unsigned long weight_many1 = weight_many;
unsigned long weight_many_wait1 = weight_many_wait;
unsigned long weight_all1 = weight_all;
unsigned long weight_all_wait1 = weight_all_wait;
if (!torture_init_begin(SCFTORT_STRING, verbose))
return -EBUSY;
scftorture_print_module_parms("Start of test");
if (weight_single == -1 && weight_single_wait == -1 &&
weight_many == -1 && weight_many_wait == -1 &&
weight_all == -1 && weight_all_wait == -1) {
weight_single1 = 2 * nr_cpu_ids;
weight_single_wait1 = 2 * nr_cpu_ids;
weight_many1 = 2;
weight_many_wait1 = 2;
weight_all1 = 1;
weight_all_wait1 = 1;
} else {
if (weight_single == -1)
weight_single1 = 0;
if (weight_single_wait == -1)
weight_single_wait1 = 0;
if (weight_many == -1)
weight_many1 = 0;
if (weight_many_wait == -1)
weight_many_wait1 = 0;
if (weight_all == -1)
weight_all1 = 0;
if (weight_all_wait == -1)
weight_all_wait1 = 0;
}
if (weight_single1 == 0 && weight_single_wait1 == 0 &&
weight_many1 == 0 && weight_many_wait1 == 0 &&
weight_all1 == 0 && weight_all_wait1 == 0) {
VERBOSE_SCFTORTOUT_ERRSTRING("all zero weights makes no sense");
firsterr = -EINVAL;
goto unwind;
}
scf_sel_add(weight_single1, SCF_PRIM_SINGLE, false);
scf_sel_add(weight_single_wait1, SCF_PRIM_SINGLE, true);
scf_sel_add(weight_many1, SCF_PRIM_MANY, false);
scf_sel_add(weight_many_wait1, SCF_PRIM_MANY, true);
scf_sel_add(weight_all1, SCF_PRIM_ALL, false);
scf_sel_add(weight_all_wait1, SCF_PRIM_ALL, true);
scf_sel_dump();
if (onoff_interval > 0) {
firsterr = torture_onoff_init(onoff_holdoff * HZ, onoff_interval, NULL);
if (firsterr)
goto unwind;
}
if (shutdown_secs > 0) {
firsterr = torture_shutdown_init(shutdown_secs, scf_torture_cleanup);
if (firsterr)
goto unwind;
}
// Worker tasks invoking smp_call_function().
if (nthreads < 0)
nthreads = num_online_cpus();
scf_stats_p = kcalloc(nthreads, sizeof(scf_stats_p[0]), GFP_KERNEL);
if (!scf_stats_p) {
VERBOSE_SCFTORTOUT_ERRSTRING("out of memory");
firsterr = -ENOMEM;
goto unwind;
}
VERBOSE_SCFTORTOUT("Starting %d smp_call_function() threads\n", nthreads);
atomic_set(&n_started, nthreads);
for (i = 0; i < nthreads; i++) {
scf_stats_p[i].cpu = i;
firsterr = torture_create_kthread(scftorture_invoker, (void *)&scf_stats_p[i],
scf_stats_p[i].task);
if (firsterr)
goto unwind;
}
if (stat_interval > 0) {
firsterr = torture_create_kthread(scf_torture_stats, NULL, scf_torture_stats_task);
if (firsterr)
goto unwind;
}
torture_init_end();
return 0;
unwind:
torture_init_end();
scf_torture_cleanup();
return firsterr;
}
module_init(scf_torture_init);
module_exit(scf_torture_cleanup);
kernel/time/tick-sched.c
View file @ f511ce14
......@@ -927,7 +927,7 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
if (ratelimit < 10 &&
(local_softirq_pending() & SOFTIRQ_STOP_IDLE_MASK)) {
pr_warn("NOHZ: local_softirq_pending %02x\n",
pr_warn("NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #%02x!!!\n",
(unsigned int) local_softirq_pending());
ratelimit++;
}
......
lib/Kconfig.debug
View file @ f511ce14
......@@ -1367,6 +1367,16 @@ config WW_MUTEX_SELFTEST
Say M if you want these self tests to build as a module.
Say N if you are unsure.
config SCF_TORTURE_TEST
tristate "torture tests for smp_call_function*()"
depends on DEBUG_KERNEL
select TORTURE_TEST
help
This option provides a kernel module that runs torture tests
on the smp_call_function() family of primitives. The kernel
module may be built after the fact on the running kernel to
be tested, if desired.
endmenu # lock debugging
config TRACE_IRQFLAGS
......
tools/testing/selftests/rcutorture/bin/kvm-recheck-rcuperf-ftrace.sh → tools/testing/selftests/rcutorture/bin/kvm-recheck-rcuscale-ftrace.sh
View file @ f511ce14
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
# Analyze a given results directory for rcuperf performance measurements,
# Analyze a given results directory for rcuscale performance measurements,
# looking for ftrace data. Exits with 0 if data was found, analyzed, and
# printed. Intended to be invoked from kvm-recheck-rcuperf.sh after
# printed. Intended to be invoked from kvm-recheck-rcuscale.sh after
# argument checking.
#
# Usage: kvm-recheck-rcuperf-ftrace.sh resdir
# Usage: kvm-recheck-rcuscale-ftrace.sh resdir
#
# Copyright (C) IBM Corporation, 2016
#
......
tools/testing/selftests/rcutorture/bin/kvm-recheck-rcuperf.sh → tools/testing/selftests/rcutorture/bin/kvm-recheck-rcuscale.sh
View file @ f511ce14
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
# Analyze a given results directory for rcuperf performance measurements.
# Analyze a given results directory for rcuscale scalability measurements.
#
# Usage: kvm-recheck-rcuperf.sh resdir
# Usage: kvm-recheck-rcuscale.sh resdir
#
# Copyright (C) IBM Corporation, 2016
#
......@@ -20,7 +20,7 @@ fi
PATH=`pwd`/tools/testing/selftests/rcutorture/bin:$PATH; export PATH
. functions.sh
if kvm-recheck-rcuperf-ftrace.sh $i
if kvm-recheck-rcuscale-ftrace.sh $i
then
# ftrace data was successfully analyzed, call it good!
exit 0
......@@ -30,12 +30,12 @@ configfile=`echo $i | sed -e 's/^.*\///'`
sed -e 's/^\[[^]]*]//' < $i/console.log |
awk '
/-perf: .* gps: .* batches:/ {
/-scale: .* gps: .* batches:/ {
ngps = $9;
nbatches = $11;
}
/-perf: .*writer-duration/ {
/-scale: .*writer-duration/ {
gptimes[++n] = $5 / 1000.;
sum += $5 / 1000.;
}
......@@ -43,7 +43,7 @@ awk '
END {
newNR = asort(gptimes);
if (newNR <= 0) {
print "No rcuperf records found???"
print "No rcuscale records found???"
exit;
}
pct50 = int(newNR * 50 / 100);
......@@ -79,5 +79,5 @@ END {
print "99th percentile grace-period duration: " gptimes[pct99];
print "Maximum grace-period duration: " gptimes[newNR];
print "Grace periods: " ngps + 0 " Batches: " nbatches + 0 " Ratio: " ngps / nbatches;
print "Computed from rcuperf printk output.";
print "Computed from rcuscale printk output.";
}'
tools/testing/selftests/rcutorture/bin/kvm-recheck-scf.sh 0 → 100755
View file @ f511ce14
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
# Analyze a given results directory for rcutorture progress.
#
# Usage: kvm-recheck-rcu.sh resdir
#
# Copyright (C) Facebook, 2020
#
# Authors: Paul E. McKenney <paulmck@kernel.org>
i="$1"
if test -d "$i" -a -r "$i"
then
:
else
echo Unreadable results directory: $i
exit 1
fi
. functions.sh
configfile=`echo $i | sed -e 's/^.*\///'`
nscfs="`grep 'scf_invoked_count ver:' $i/console.log 2> /dev/null | tail -1 | sed -e 's/^.* scf_invoked_count ver: //' -e 's/ .*$//' | tr -d '\015'`"
if test -z "$nscfs"
then
echo "$configfile ------- "
else
dur="`sed -e 's/^.* scftorture.shutdown_secs=//' -e 's/ .*$//' < $i/qemu-cmd 2> /dev/null`"
if test -z "$dur"
then
rate=""
else
nscfss=`awk -v nscfs=$nscfs -v dur=$dur '
BEGIN { print nscfs / dur }' < /dev/null`
rate=" ($nscfss/s)"
fi
echo "${configfile} ------- ${nscfs} SCF handler invocations$rate"
fi
tools/testing/selftests/rcutorture/bin/kvm.sh
View file @ f511ce14
......@@ -71,7 +71,7 @@ usage () {
echo " --qemu-args qemu-arguments"
echo " --qemu-cmd qemu-system-..."
echo " --results absolute-pathname"
echo " --torture lock|rcu|rcuperf|refscale|scf"
echo " --torture lock|rcu|rcuscale|refscale|scf"
echo " --trust-make"
exit 1
}
......@@ -198,13 +198,13 @@ do
shift
;;
--torture)
checkarg --torture "(suite name)" "$#" "$2" '^\(lock\|rcu\|rcuperf\|refscale\)$' '^--'
checkarg --torture "(suite name)" "$#" "$2" '^\(lock\|rcu\|rcuscale\|refscale\|scf\)$' '^--'
TORTURE_SUITE=$2
shift
if test "$TORTURE_SUITE" = rcuperf || test "$TORTURE_SUITE" = refscale
if test "$TORTURE_SUITE" = rcuscale || test "$TORTURE_SUITE" = refscale
then
# If you really want jitter for refscale or
# rcuperf, specify it after specifying the rcuperf
# rcuscale, specify it after specifying the rcuscale
# or the refscale. (But why jitter in these cases?)
jitter=0
fi
......
tools/testing/selftests/rcutorture/bin/parse-console.sh
View file @ f511ce14
......@@ -33,8 +33,8 @@ then
fi
cat /dev/null > $file.diags
# Check for proper termination, except for rcuperf and refscale.
if test "$TORTURE_SUITE" != rcuperf && test "$TORTURE_SUITE" != refscale
# Check for proper termination, except for rcuscale and refscale.
if test "$TORTURE_SUITE" != rcuscale && test "$TORTURE_SUITE" != refscale
then
# check for abject failure
......@@ -67,6 +67,7 @@ then
grep --binary-files=text 'torture:.*ver:' $file |
egrep --binary-files=text -v '\(null\)|rtc: 000000000* ' |
sed -e 's/^(initramfs)[^]]*] //' -e 's/^\[[^]]*] //' |
sed -e 's/^.*ver: //' |
awk '
BEGIN {
ver = 0;
......@@ -74,13 +75,13 @@ then
}
{
if (!badseq && ($5 + 0 != $5 || $5 <= ver)) {
if (!badseq && ($1 + 0 != $1 || $1 <= ver)) {
badseqno1 = ver;
badseqno2 = $5;
badseqno2 = $1;
badseqnr = NR;
badseq = 1;
}
ver = $5
ver = $1
}
END {
......
tools/testing/selftests/rcutorture/configs/rcuperf/ver_functions.sh → tools/testing/selftests/rcutorture/configs/rcuscale/ver_functions.sh
View file @ f511ce14
......@@ -11,6 +11,6 @@
#
# Adds per-version torture-module parameters to kernels supporting them.
per_version_boot_params () {
echo $1 rcuperf.shutdown=1 \
rcuperf.verbose=1
echo $1 rcuscale.shutdown=1 \
rcuscale.verbose=1
}
tools/testing/selftests/rcutorture/configs/scf/CFcommon 0 → 100644
View file @ f511ce14
CONFIG_SCF_TORTURE_TEST=y
CONFIG_PRINTK_TIME=y
tools/testing/selftests/rcutorture/configs/scf/NOPREEMPT 0 → 100644
View file @ f511ce14
CONFIG_SMP=y
CONFIG_PREEMPT_NONE=y
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=n
CONFIG_HZ_PERIODIC=n
CONFIG_NO_HZ_IDLE=n
CONFIG_NO_HZ_FULL=y
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_PROVE_LOCKING=n
tools/testing/selftests/rcutorture/configs/scf/PREEMPT 0 → 100644
View file @ f511ce14
CONFIG_SMP=y
CONFIG_PREEMPT_NONE=n
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=y
CONFIG_HZ_PERIODIC=n
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_FULL=n
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
tools/testing/selftests/rcutorture/configs/scf/ver_functions.sh 0 → 100644
View file @ f511ce14
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
# Torture-suite-dependent shell functions for the rest of the scripts.
#
# Copyright (C) Facebook, 2020
#
# Authors: Paul E. McKenney <paulmck@kernel.org>
# scftorture_param_onoff bootparam-string config-file
#
# Adds onoff scftorture module parameters to kernels having it.
scftorture_param_onoff () {
if ! bootparam_hotplug_cpu "$1" && configfrag_hotplug_cpu "$2"
then
echo CPU-hotplug kernel, adding scftorture onoff. 1>&2
echo scftorture.onoff_interval=1000 scftorture.onoff_holdoff=30
fi
}
# per_version_boot_params bootparam-string config-file seconds
#
# Adds per-version torture-module parameters to kernels supporting them.
per_version_boot_params () {
echo $1 `scftorture_param_onoff "$1" "$2"` \
scftorture.stat_interval=15 \
scftorture.shutdown_secs=$3 \
scftorture.verbose=1 \
scf
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7