Commit bda4ec9f authored by Paul E. McKenney's avatar Paul E. McKenney

Merge branches 'bigrt.2012.09.23a', 'doctorture.2012.09.23a',...

Merge branches 'bigrt.2012.09.23a', 'doctorture.2012.09.23a', 'fixes.2012.09.23a', 'hotplug.2012.09.23a' and 'idlechop.2012.09.23a' into HEAD

bigrt.2012.09.23a contains additional commits to reduce scheduling latency
	from RCU on huge systems (many hundrends or thousands of CPUs).

doctorture.2012.09.23a contains documentation changes and rcutorture fixes.

fixes.2012.09.23a contains miscellaneous fixes.

hotplug.2012.09.23a contains CPU-hotplug-related changes.

idle.2012.09.23a fixes architectures for which RCU no longer considered
	the idle loop to be a quiescent state due to earlier
	adaptive-dynticks changes.  Affected architectures are alpha,
	cris, frv, h8300, m32r, m68k, mn10300, parisc, score, xtensa,
	and ia64.
...@@ -310,6 +310,12 @@ over a rather long period of time, but improvements are always welcome! ...@@ -310,6 +310,12 @@ over a rather long period of time, but improvements are always welcome!
code under the influence of preempt_disable(), you instead code under the influence of preempt_disable(), you instead
need to use synchronize_irq() or synchronize_sched(). need to use synchronize_irq() or synchronize_sched().
This same limitation also applies to synchronize_rcu_bh()
and synchronize_srcu(), as well as to the asynchronous and
expedited forms of the three primitives, namely call_rcu(),
call_rcu_bh(), call_srcu(), synchronize_rcu_expedited(),
synchronize_rcu_bh_expedited(), and synchronize_srcu_expedited().
12. Any lock acquired by an RCU callback must be acquired elsewhere 12. Any lock acquired by an RCU callback must be acquired elsewhere
with softirq disabled, e.g., via spin_lock_irqsave(), with softirq disabled, e.g., via spin_lock_irqsave(),
spin_lock_bh(), etc. Failing to disable irq on a given spin_lock_bh(), etc. Failing to disable irq on a given
......
...@@ -99,7 +99,7 @@ In kernels with CONFIG_RCU_FAST_NO_HZ, even more information is ...@@ -99,7 +99,7 @@ In kernels with CONFIG_RCU_FAST_NO_HZ, even more information is
printed: printed:
INFO: rcu_preempt detected stall on CPU INFO: rcu_preempt detected stall on CPU
0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 drain=0 . timer=-1 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 drain=0 . timer not pending
(t=65000 jiffies) (t=65000 jiffies)
The "(64628 ticks this GP)" indicates that this CPU has taken more The "(64628 ticks this GP)" indicates that this CPU has taken more
...@@ -116,13 +116,13 @@ number between the two "/"s is the value of the nesting, which will ...@@ -116,13 +116,13 @@ number between the two "/"s is the value of the nesting, which will
be a small positive number if in the idle loop and a very large positive be a small positive number if in the idle loop and a very large positive
number (as shown above) otherwise. number (as shown above) otherwise.
For CONFIG_RCU_FAST_NO_HZ kernels, the "drain=0" indicates that the For CONFIG_RCU_FAST_NO_HZ kernels, the "drain=0" indicates that the CPU is
CPU is not in the process of trying to force itself into dyntick-idle not in the process of trying to force itself into dyntick-idle state, the
state, the "." indicates that the CPU has not given up forcing RCU "." indicates that the CPU has not given up forcing RCU into dyntick-idle
into dyntick-idle mode (it would be "H" otherwise), and the "timer=-1" mode (it would be "H" otherwise), and the "timer not pending" indicates
indicates that the CPU has not recented forced RCU into dyntick-idle that the CPU has not recently forced RCU into dyntick-idle mode (it
mode (it would otherwise indicate the number of microseconds remaining would otherwise indicate the number of microseconds remaining in this
in this forced state). forced state).
Multiple Warnings From One Stall Multiple Warnings From One Stall
......
...@@ -873,7 +873,7 @@ d. Do you need to treat NMI handlers, hardirq handlers, ...@@ -873,7 +873,7 @@ d. Do you need to treat NMI handlers, hardirq handlers,
and code segments with preemption disabled (whether and code segments with preemption disabled (whether
via preempt_disable(), local_irq_save(), local_bh_disable(), via preempt_disable(), local_irq_save(), local_bh_disable(),
or some other mechanism) as if they were explicit RCU readers? or some other mechanism) as if they were explicit RCU readers?
If so, you need RCU-sched. If so, RCU-sched is the only choice that will work for you.
e. Do you need RCU grace periods to complete even in the face e. Do you need RCU grace periods to complete even in the face
of softirq monopolization of one or more of the CPUs? For of softirq monopolization of one or more of the CPUs? For
...@@ -884,7 +884,12 @@ f. Is your workload too update-intensive for normal use of ...@@ -884,7 +884,12 @@ f. Is your workload too update-intensive for normal use of
RCU, but inappropriate for other synchronization mechanisms? RCU, but inappropriate for other synchronization mechanisms?
If so, consider SLAB_DESTROY_BY_RCU. But please be careful! If so, consider SLAB_DESTROY_BY_RCU. But please be careful!
g. Otherwise, use RCU. g. Do you need read-side critical sections that are respected
even though they are in the middle of the idle loop, during
user-mode execution, or on an offlined CPU? If so, SRCU is the
only choice that will work for you.
h. Otherwise, use RCU.
Of course, this all assumes that you have determined that RCU is in fact Of course, this all assumes that you have determined that RCU is in fact
the right tool for your job. the right tool for your job.
......
...@@ -28,6 +28,7 @@ ...@@ -28,6 +28,7 @@
#include <linux/tty.h> #include <linux/tty.h>
#include <linux/console.h> #include <linux/console.h>
#include <linux/slab.h> #include <linux/slab.h>
#include <linux/rcupdate.h>
#include <asm/reg.h> #include <asm/reg.h>
#include <asm/uaccess.h> #include <asm/uaccess.h>
...@@ -54,9 +55,12 @@ cpu_idle(void) ...@@ -54,9 +55,12 @@ cpu_idle(void)
/* FIXME -- EV6 and LCA45 know how to power down /* FIXME -- EV6 and LCA45 know how to power down
the CPU. */ the CPU. */
rcu_idle_enter();
while (!need_resched()) while (!need_resched())
cpu_relax(); cpu_relax();
schedule();
rcu_idle_exit();
schedule_preempt_disabled();
} }
} }
......
...@@ -166,6 +166,7 @@ smp_callin(void) ...@@ -166,6 +166,7 @@ smp_callin(void)
DBGS(("smp_callin: commencing CPU %d current %p active_mm %p\n", DBGS(("smp_callin: commencing CPU %d current %p active_mm %p\n",
cpuid, current, current->active_mm)); cpuid, current, current->active_mm));
preempt_disable();
/* Do nothing. */ /* Do nothing. */
cpu_idle(); cpu_idle();
} }
......
...@@ -25,6 +25,7 @@ ...@@ -25,6 +25,7 @@
#include <linux/elfcore.h> #include <linux/elfcore.h>
#include <linux/mqueue.h> #include <linux/mqueue.h>
#include <linux/reboot.h> #include <linux/reboot.h>
#include <linux/rcupdate.h>
//#define DEBUG //#define DEBUG
...@@ -74,6 +75,7 @@ void cpu_idle (void) ...@@ -74,6 +75,7 @@ void cpu_idle (void)
{ {
/* endless idle loop with no priority at all */ /* endless idle loop with no priority at all */
while (1) { while (1) {
rcu_idle_enter();
while (!need_resched()) { while (!need_resched()) {
void (*idle)(void); void (*idle)(void);
/* /*
...@@ -86,6 +88,7 @@ void cpu_idle (void) ...@@ -86,6 +88,7 @@ void cpu_idle (void)
idle = default_idle; idle = default_idle;
idle(); idle();
} }
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
} }
} }
......
...@@ -25,6 +25,7 @@ ...@@ -25,6 +25,7 @@
#include <linux/reboot.h> #include <linux/reboot.h>
#include <linux/interrupt.h> #include <linux/interrupt.h>
#include <linux/pagemap.h> #include <linux/pagemap.h>
#include <linux/rcupdate.h>
#include <asm/asm-offsets.h> #include <asm/asm-offsets.h>
#include <asm/uaccess.h> #include <asm/uaccess.h>
...@@ -69,12 +70,14 @@ void cpu_idle(void) ...@@ -69,12 +70,14 @@ void cpu_idle(void)
{ {
/* endless idle loop with no priority at all */ /* endless idle loop with no priority at all */
while (1) { while (1) {
rcu_idle_enter();
while (!need_resched()) { while (!need_resched()) {
check_pgt_cache(); check_pgt_cache();
if (!frv_dma_inprogress && idle) if (!frv_dma_inprogress && idle)
idle(); idle();
} }
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
} }
......
...@@ -36,6 +36,7 @@ ...@@ -36,6 +36,7 @@
#include <linux/reboot.h> #include <linux/reboot.h>
#include <linux/fs.h> #include <linux/fs.h>
#include <linux/slab.h> #include <linux/slab.h>
#include <linux/rcupdate.h>
#include <asm/uaccess.h> #include <asm/uaccess.h>
#include <asm/traps.h> #include <asm/traps.h>
...@@ -78,8 +79,10 @@ void (*idle)(void) = default_idle; ...@@ -78,8 +79,10 @@ void (*idle)(void) = default_idle;
void cpu_idle(void) void cpu_idle(void)
{ {
while (1) { while (1) {
rcu_idle_enter();
while (!need_resched()) while (!need_resched())
idle(); idle();
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
} }
} }
......
...@@ -29,6 +29,7 @@ ...@@ -29,6 +29,7 @@
#include <linux/kdebug.h> #include <linux/kdebug.h>
#include <linux/utsname.h> #include <linux/utsname.h>
#include <linux/tracehook.h> #include <linux/tracehook.h>
#include <linux/rcupdate.h>
#include <asm/cpu.h> #include <asm/cpu.h>
#include <asm/delay.h> #include <asm/delay.h>
...@@ -279,6 +280,7 @@ cpu_idle (void) ...@@ -279,6 +280,7 @@ cpu_idle (void)
/* endless idle loop with no priority at all */ /* endless idle loop with no priority at all */
while (1) { while (1) {
rcu_idle_enter();
if (can_do_pal_halt) { if (can_do_pal_halt) {
current_thread_info()->status &= ~TS_POLLING; current_thread_info()->status &= ~TS_POLLING;
/* /*
...@@ -309,6 +311,7 @@ cpu_idle (void) ...@@ -309,6 +311,7 @@ cpu_idle (void)
normal_xtp(); normal_xtp();
#endif #endif
} }
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
check_pgt_cache(); check_pgt_cache();
if (cpu_is_offline(cpu)) if (cpu_is_offline(cpu))
......
...@@ -26,6 +26,7 @@ ...@@ -26,6 +26,7 @@
#include <linux/ptrace.h> #include <linux/ptrace.h>
#include <linux/unistd.h> #include <linux/unistd.h>
#include <linux/hardirq.h> #include <linux/hardirq.h>
#include <linux/rcupdate.h>
#include <asm/io.h> #include <asm/io.h>
#include <asm/uaccess.h> #include <asm/uaccess.h>
...@@ -82,6 +83,7 @@ void cpu_idle (void) ...@@ -82,6 +83,7 @@ void cpu_idle (void)
{ {
/* endless idle loop with no priority at all */ /* endless idle loop with no priority at all */
while (1) { while (1) {
rcu_idle_enter();
while (!need_resched()) { while (!need_resched()) {
void (*idle)(void) = pm_idle; void (*idle)(void) = pm_idle;
...@@ -90,6 +92,7 @@ void cpu_idle (void) ...@@ -90,6 +92,7 @@ void cpu_idle (void)
idle(); idle();
} }
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
} }
} }
......
...@@ -25,6 +25,7 @@ ...@@ -25,6 +25,7 @@
#include <linux/reboot.h> #include <linux/reboot.h>
#include <linux/init_task.h> #include <linux/init_task.h>
#include <linux/mqueue.h> #include <linux/mqueue.h>
#include <linux/rcupdate.h>
#include <asm/uaccess.h> #include <asm/uaccess.h>
#include <asm/traps.h> #include <asm/traps.h>
...@@ -75,8 +76,10 @@ void cpu_idle(void) ...@@ -75,8 +76,10 @@ void cpu_idle(void)
{ {
/* endless idle loop with no priority at all */ /* endless idle loop with no priority at all */
while (1) { while (1) {
rcu_idle_enter();
while (!need_resched()) while (!need_resched())
idle(); idle();
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
} }
} }
......
...@@ -25,6 +25,7 @@ ...@@ -25,6 +25,7 @@
#include <linux/err.h> #include <linux/err.h>
#include <linux/fs.h> #include <linux/fs.h>
#include <linux/slab.h> #include <linux/slab.h>
#include <linux/rcupdate.h>
#include <asm/uaccess.h> #include <asm/uaccess.h>
#include <asm/pgtable.h> #include <asm/pgtable.h>
#include <asm/io.h> #include <asm/io.h>
...@@ -107,6 +108,7 @@ void cpu_idle(void) ...@@ -107,6 +108,7 @@ void cpu_idle(void)
{ {
/* endless idle loop with no priority at all */ /* endless idle loop with no priority at all */
for (;;) { for (;;) {
rcu_idle_enter();
while (!need_resched()) { while (!need_resched()) {
void (*idle)(void); void (*idle)(void);
...@@ -121,6 +123,7 @@ void cpu_idle(void) ...@@ -121,6 +123,7 @@ void cpu_idle(void)
} }
idle(); idle();
} }
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
} }
......
...@@ -48,6 +48,7 @@ ...@@ -48,6 +48,7 @@
#include <linux/unistd.h> #include <linux/unistd.h>
#include <linux/kallsyms.h> #include <linux/kallsyms.h>
#include <linux/uaccess.h> #include <linux/uaccess.h>
#include <linux/rcupdate.h>
#include <asm/io.h> #include <asm/io.h>
#include <asm/asm-offsets.h> #include <asm/asm-offsets.h>
...@@ -69,8 +70,10 @@ void cpu_idle(void) ...@@ -69,8 +70,10 @@ void cpu_idle(void)
/* endless idle loop with no priority at all */ /* endless idle loop with no priority at all */
while (1) { while (1) {
rcu_idle_enter();
while (!need_resched()) while (!need_resched())
barrier(); barrier();
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
check_pgt_cache(); check_pgt_cache();
} }
......
...@@ -27,6 +27,7 @@ ...@@ -27,6 +27,7 @@
#include <linux/reboot.h> #include <linux/reboot.h>
#include <linux/elfcore.h> #include <linux/elfcore.h>
#include <linux/pm.h> #include <linux/pm.h>
#include <linux/rcupdate.h>
void (*pm_power_off)(void); void (*pm_power_off)(void);
EXPORT_SYMBOL(pm_power_off); EXPORT_SYMBOL(pm_power_off);
...@@ -50,9 +51,10 @@ void __noreturn cpu_idle(void) ...@@ -50,9 +51,10 @@ void __noreturn cpu_idle(void)
{ {
/* endless idle loop with no priority at all */ /* endless idle loop with no priority at all */
while (1) { while (1) {
rcu_idle_enter();
while (!need_resched()) while (!need_resched())
barrier(); barrier();
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
} }
} }
......
...@@ -199,12 +199,14 @@ static int __init cpuid_init(void) ...@@ -199,12 +199,14 @@ static int __init cpuid_init(void)
goto out_chrdev; goto out_chrdev;
} }
cpuid_class->devnode = cpuid_devnode; cpuid_class->devnode = cpuid_devnode;
get_online_cpus();
for_each_online_cpu(i) { for_each_online_cpu(i) {
err = cpuid_device_create(i); err = cpuid_device_create(i);
if (err != 0) if (err != 0)
goto out_class; goto out_class;
} }
register_hotcpu_notifier(&cpuid_class_cpu_notifier); register_hotcpu_notifier(&cpuid_class_cpu_notifier);
put_online_cpus();
err = 0; err = 0;
goto out; goto out;
...@@ -214,6 +216,7 @@ static int __init cpuid_init(void) ...@@ -214,6 +216,7 @@ static int __init cpuid_init(void)
for_each_online_cpu(i) { for_each_online_cpu(i) {
cpuid_device_destroy(i); cpuid_device_destroy(i);
} }
put_online_cpus();
class_destroy(cpuid_class); class_destroy(cpuid_class);
out_chrdev: out_chrdev:
__unregister_chrdev(CPUID_MAJOR, 0, NR_CPUS, "cpu/cpuid"); __unregister_chrdev(CPUID_MAJOR, 0, NR_CPUS, "cpu/cpuid");
...@@ -225,11 +228,13 @@ static void __exit cpuid_exit(void) ...@@ -225,11 +228,13 @@ static void __exit cpuid_exit(void)
{ {
int cpu = 0; int cpu = 0;
get_online_cpus();
for_each_online_cpu(cpu) for_each_online_cpu(cpu)
cpuid_device_destroy(cpu); cpuid_device_destroy(cpu);
class_destroy(cpuid_class); class_destroy(cpuid_class);
__unregister_chrdev(CPUID_MAJOR, 0, NR_CPUS, "cpu/cpuid"); __unregister_chrdev(CPUID_MAJOR, 0, NR_CPUS, "cpu/cpuid");
unregister_hotcpu_notifier(&cpuid_class_cpu_notifier); unregister_hotcpu_notifier(&cpuid_class_cpu_notifier);
put_online_cpus();
} }
module_init(cpuid_init); module_init(cpuid_init);
......
...@@ -257,12 +257,14 @@ static int __init msr_init(void) ...@@ -257,12 +257,14 @@ static int __init msr_init(void)
goto out_chrdev; goto out_chrdev;
} }
msr_class->devnode = msr_devnode; msr_class->devnode = msr_devnode;
get_online_cpus();
for_each_online_cpu(i) { for_each_online_cpu(i) {
err = msr_device_create(i); err = msr_device_create(i);
if (err != 0) if (err != 0)
goto out_class; goto out_class;
} }
register_hotcpu_notifier(&msr_class_cpu_notifier); register_hotcpu_notifier(&msr_class_cpu_notifier);
put_online_cpus();
err = 0; err = 0;
goto out; goto out;
...@@ -271,6 +273,7 @@ static int __init msr_init(void) ...@@ -271,6 +273,7 @@ static int __init msr_init(void)
i = 0; i = 0;
for_each_online_cpu(i) for_each_online_cpu(i)
msr_device_destroy(i); msr_device_destroy(i);
put_online_cpus();
class_destroy(msr_class); class_destroy(msr_class);
out_chrdev: out_chrdev:
__unregister_chrdev(MSR_MAJOR, 0, NR_CPUS, "cpu/msr"); __unregister_chrdev(MSR_MAJOR, 0, NR_CPUS, "cpu/msr");
...@@ -281,11 +284,13 @@ static int __init msr_init(void) ...@@ -281,11 +284,13 @@ static int __init msr_init(void)
static void __exit msr_exit(void) static void __exit msr_exit(void)
{ {
int cpu = 0; int cpu = 0;
get_online_cpus();
for_each_online_cpu(cpu) for_each_online_cpu(cpu)
msr_device_destroy(cpu); msr_device_destroy(cpu);
class_destroy(msr_class); class_destroy(msr_class);
__unregister_chrdev(MSR_MAJOR, 0, NR_CPUS, "cpu/msr"); __unregister_chrdev(MSR_MAJOR, 0, NR_CPUS, "cpu/msr");
unregister_hotcpu_notifier(&msr_class_cpu_notifier); unregister_hotcpu_notifier(&msr_class_cpu_notifier);
put_online_cpus();
} }
module_init(msr_init); module_init(msr_init);
......
...@@ -31,6 +31,7 @@ ...@@ -31,6 +31,7 @@
#include <linux/mqueue.h> #include <linux/mqueue.h>
#include <linux/fs.h> #include <linux/fs.h>
#include <linux/slab.h> #include <linux/slab.h>
#include <linux/rcupdate.h>
#include <asm/pgtable.h> #include <asm/pgtable.h>
#include <asm/uaccess.h> #include <asm/uaccess.h>
...@@ -110,8 +111,10 @@ void cpu_idle(void) ...@@ -110,8 +111,10 @@ void cpu_idle(void)
/* endless idle loop with no priority at all */ /* endless idle loop with no priority at all */
while (1) { while (1) {
rcu_idle_enter();
while (!need_resched()) while (!need_resched())
platform_idle(); platform_idle();
rcu_idle_exit();
schedule_preempt_disabled(); schedule_preempt_disabled();
} }
} }
......
...@@ -430,6 +430,8 @@ enum ...@@ -430,6 +430,8 @@ enum
NR_SOFTIRQS NR_SOFTIRQS
}; };
#define SOFTIRQ_STOP_IDLE_MASK (~(1 << RCU_SOFTIRQ))
/* map softirq index to softirq name. update 'softirq_to_name' in /* map softirq index to softirq name. update 'softirq_to_name' in
* kernel/softirq.c when adding a new softirq. * kernel/softirq.c when adding a new softirq.
*/ */
......
...@@ -210,14 +210,12 @@ extern void exit_rcu(void); ...@@ -210,14 +210,12 @@ extern void exit_rcu(void);
* to nest RCU_NONIDLE() wrappers, but the nesting level is currently * to nest RCU_NONIDLE() wrappers, but the nesting level is currently
* quite limited. If deeper nesting is required, it will be necessary * quite limited. If deeper nesting is required, it will be necessary
* to adjust DYNTICK_TASK_NESTING_VALUE accordingly. * to adjust DYNTICK_TASK_NESTING_VALUE accordingly.
*
* This macro may be used from process-level code only.
*/ */
#define RCU_NONIDLE(a) \ #define RCU_NONIDLE(a) \
do { \ do { \
rcu_idle_exit(); \ rcu_irq_enter(); \
do { a; } while (0); \ do { a; } while (0); \
rcu_idle_enter(); \ rcu_irq_exit(); \
} while (0) } while (0)
/* /*
......
...@@ -45,6 +45,7 @@ ...@@ -45,6 +45,7 @@
#include <linux/mutex.h> #include <linux/mutex.h>
#include <linux/export.h> #include <linux/export.h>
#include <linux/hardirq.h> #include <linux/hardirq.h>
#include <linux/delay.h>
#define CREATE_TRACE_POINTS #define CREATE_TRACE_POINTS
#include <trace/events/rcu.h> #include <trace/events/rcu.h>
...@@ -81,6 +82,9 @@ void __rcu_read_unlock(void) ...@@ -81,6 +82,9 @@ void __rcu_read_unlock(void)
} else { } else {
barrier(); /* critical section before exit code. */ barrier(); /* critical section before exit code. */
t->rcu_read_lock_nesting = INT_MIN; t->rcu_read_lock_nesting = INT_MIN;
#ifdef CONFIG_PROVE_RCU_DELAY
udelay(10); /* Make preemption more probable. */
#endif /* #ifdef CONFIG_PROVE_RCU_DELAY */
barrier(); /* assign before ->rcu_read_unlock_special load */ barrier(); /* assign before ->rcu_read_unlock_special load */
if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special))) if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
rcu_read_unlock_special(t); rcu_read_unlock_special(t);
......
...@@ -56,25 +56,28 @@ static void __call_rcu(struct rcu_head *head, ...@@ -56,25 +56,28 @@ static void __call_rcu(struct rcu_head *head,
static long long rcu_dynticks_nesting = DYNTICK_TASK_EXIT_IDLE; static long long rcu_dynticks_nesting = DYNTICK_TASK_EXIT_IDLE;
/* Common code for rcu_idle_enter() and rcu_irq_exit(), see kernel/rcutree.c. */ /* Common code for rcu_idle_enter() and rcu_irq_exit(), see kernel/rcutree.c. */
static void rcu_idle_enter_common(long long oldval) static void rcu_idle_enter_common(long long newval)
{ {
if (rcu_dynticks_nesting) { if (newval) {
RCU_TRACE(trace_rcu_dyntick("--=", RCU_TRACE(trace_rcu_dyntick("--=",
oldval, rcu_dynticks_nesting)); rcu_dynticks_nesting, newval));
rcu_dynticks_nesting = newval;
return; return;
} }
RCU_TRACE(trace_rcu_dyntick("Start", oldval, rcu_dynticks_nesting)); RCU_TRACE(trace_rcu_dyntick("Start", rcu_dynticks_nesting, newval));
if (!is_idle_task(current)) { if (!is_idle_task(current)) {
struct task_struct *idle = idle_task(smp_processor_id()); struct task_struct *idle = idle_task(smp_processor_id());
RCU_TRACE(trace_rcu_dyntick("Error on entry: not idle task", RCU_TRACE(trace_rcu_dyntick("Error on entry: not idle task",
oldval, rcu_dynticks_nesting)); rcu_dynticks_nesting, newval));
ftrace_dump(DUMP_ALL); ftrace_dump(DUMP_ALL);
WARN_ONCE(1, "Current pid: %d comm: %s / Idle pid: %d comm: %s", WARN_ONCE(1, "Current pid: %d comm: %s / Idle pid: %d comm: %s",
current->pid, current->comm, current->pid, current->comm,
idle->pid, idle->comm); /* must be idle task! */ idle->pid, idle->comm); /* must be idle task! */
} }
rcu_sched_qs(0); /* implies rcu_bh_qsctr_inc(0) */ rcu_sched_qs(0); /* implies rcu_bh_qsctr_inc(0) */
barrier();
rcu_dynticks_nesting = newval;
} }
/* /*
...@@ -84,17 +87,16 @@ static void rcu_idle_enter_common(long long oldval) ...@@ -84,17 +87,16 @@ static void rcu_idle_enter_common(long long oldval)
void rcu_idle_enter(void) void rcu_idle_enter(void)
{ {
unsigned long flags; unsigned long flags;
long long oldval; long long newval;
local_irq_save(flags); local_irq_save(flags);
oldval = rcu_dynticks_nesting;
WARN_ON_ONCE((rcu_dynticks_nesting & DYNTICK_TASK_NEST_MASK) == 0); WARN_ON_ONCE((rcu_dynticks_nesting & DYNTICK_TASK_NEST_MASK) == 0);
if ((rcu_dynticks_nesting & DYNTICK_TASK_NEST_MASK) == if ((rcu_dynticks_nesting & DYNTICK_TASK_NEST_MASK) ==
DYNTICK_TASK_NEST_VALUE) DYNTICK_TASK_NEST_VALUE)
rcu_dynticks_nesting = 0; newval = 0;
else else
rcu_dynticks_nesting -= DYNTICK_TASK_NEST_VALUE; newval = rcu_dynticks_nesting - DYNTICK_TASK_NEST_VALUE;
rcu_idle_enter_common(oldval); rcu_idle_enter_common(newval);
local_irq_restore(flags); local_irq_restore(flags);
} }
EXPORT_SYMBOL_GPL(rcu_idle_enter); EXPORT_SYMBOL_GPL(rcu_idle_enter);
...@@ -105,15 +107,15 @@ EXPORT_SYMBOL_GPL(rcu_idle_enter); ...@@ -105,15 +107,15 @@ EXPORT_SYMBOL_GPL(rcu_idle_enter);
void rcu_irq_exit(void) void rcu_irq_exit(void)
{ {
unsigned long flags; unsigned long flags;
long long oldval; long long newval;
local_irq_save(flags); local_irq_save(flags);
oldval = rcu_dynticks_nesting; newval = rcu_dynticks_nesting - 1;
rcu_dynticks_nesting--; WARN_ON_ONCE(newval < 0);
WARN_ON_ONCE(rcu_dynticks_nesting < 0); rcu_idle_enter_common(newval);
rcu_idle_enter_common(oldval);
local_irq_restore(flags); local_irq_restore(flags);
} }
EXPORT_SYMBOL_GPL(rcu_irq_exit);
/* Common code for rcu_idle_exit() and rcu_irq_enter(), see kernel/rcutree.c. */ /* Common code for rcu_idle_exit() and rcu_irq_enter(), see kernel/rcutree.c. */
static void rcu_idle_exit_common(long long oldval) static void rcu_idle_exit_common(long long oldval)
...@@ -171,6 +173,7 @@ void rcu_irq_enter(void) ...@@ -171,6 +173,7 @@ void rcu_irq_enter(void)
rcu_idle_exit_common(oldval); rcu_idle_exit_common(oldval);
local_irq_restore(flags); local_irq_restore(flags);
} }
EXPORT_SYMBOL_GPL(rcu_irq_enter);
#ifdef CONFIG_DEBUG_LOCK_ALLOC #ifdef CONFIG_DEBUG_LOCK_ALLOC
......
...@@ -278,7 +278,7 @@ static int rcu_boost(void) ...@@ -278,7 +278,7 @@ static int rcu_boost(void)
rcu_preempt_ctrlblk.exp_tasks == NULL) rcu_preempt_ctrlblk.exp_tasks == NULL)
return 0; /* Nothing to boost. */ return 0; /* Nothing to boost. */
raw_local_irq_save(flags); local_irq_save(flags);
/* /*
* Recheck with irqs disabled: all tasks in need of boosting * Recheck with irqs disabled: all tasks in need of boosting
...@@ -287,7 +287,7 @@ static int rcu_boost(void) ...@@ -287,7 +287,7 @@ static int rcu_boost(void)
*/ */
if (rcu_preempt_ctrlblk.boost_tasks == NULL && if (rcu_preempt_ctrlblk.boost_tasks == NULL &&
rcu_preempt_ctrlblk.exp_tasks == NULL) { rcu_preempt_ctrlblk.exp_tasks == NULL) {
raw_local_irq_restore(flags); local_irq_restore(flags);
return 0; return 0;
} }
...@@ -317,7 +317,7 @@ static int rcu_boost(void) ...@@ -317,7 +317,7 @@ static int rcu_boost(void)
t = container_of(tb, struct task_struct, rcu_node_entry); t = container_of(tb, struct task_struct, rcu_node_entry);
rt_mutex_init_proxy_locked(&mtx, t); rt_mutex_init_proxy_locked(&mtx, t);
t->rcu_boost_mutex = &mtx; t->rcu_boost_mutex = &mtx;
raw_local_irq_restore(flags); local_irq_restore(flags);
rt_mutex_lock(&mtx); rt_mutex_lock(&mtx);
rt_mutex_unlock(&mtx); /* Keep lockdep happy. */ rt_mutex_unlock(&mtx); /* Keep lockdep happy. */
...@@ -991,9 +991,9 @@ static void rcu_trace_sub_qlen(struct rcu_ctrlblk *rcp, int n) ...@@ -991,9 +991,9 @@ static void rcu_trace_sub_qlen(struct rcu_ctrlblk *rcp, int n)
{ {
unsigned long flags; unsigned long flags;
raw_local_irq_save(flags); local_irq_save(flags);
rcp->qlen -= n; rcp->qlen -= n;
raw_local_irq_restore(flags); local_irq_restore(flags);
} }
/* /*
......
...@@ -53,10 +53,11 @@ MODULE_AUTHOR("Paul E. McKenney <paulmck@us.ibm.com> and Josh Triplett <josh@fre ...@@ -53,10 +53,11 @@ MODULE_AUTHOR("Paul E. McKenney <paulmck@us.ibm.com> and Josh Triplett <josh@fre
static int nreaders = -1; /* # reader threads, defaults to 2*ncpus */ static int nreaders = -1; /* # reader threads, defaults to 2*ncpus */
static int nfakewriters = 4; /* # fake writer threads */ static int nfakewriters = 4; /* # fake writer threads */
static int stat_interval; /* Interval between stats, in seconds. */ static int stat_interval = 60; /* Interval between stats, in seconds. */
/* Defaults to "only at end of test". */ /* Zero means "only at end of test". */
static bool verbose; /* Print more debug info. */ static bool verbose; /* Print more debug info. */
static bool test_no_idle_hz; /* Test RCU's support for tickless idle CPUs. */ static bool test_no_idle_hz = true;
/* Test RCU support for tickless idle CPUs. */
static int shuffle_interval = 3; /* Interval between shuffles (in sec)*/ static int shuffle_interval = 3; /* Interval between shuffles (in sec)*/
static int stutter = 5; /* Start/stop testing interval (in sec) */ static int stutter = 5; /* Start/stop testing interval (in sec) */
static int irqreader = 1; /* RCU readers from irq (timers). */ static int irqreader = 1; /* RCU readers from irq (timers). */
...@@ -119,11 +120,11 @@ MODULE_PARM_DESC(torture_type, "Type of RCU to torture (rcu, rcu_bh, srcu)"); ...@@ -119,11 +120,11 @@ MODULE_PARM_DESC(torture_type, "Type of RCU to torture (rcu, rcu_bh, srcu)");
#define TORTURE_FLAG "-torture:" #define TORTURE_FLAG "-torture:"
#define PRINTK_STRING(s) \ #define PRINTK_STRING(s) \
do { printk(KERN_ALERT "%s" TORTURE_FLAG s "\n", torture_type); } while (0) do { pr_alert("%s" TORTURE_FLAG s "\n", torture_type); } while (0)
#define VERBOSE_PRINTK_STRING(s) \ #define VERBOSE_PRINTK_STRING(s) \
do { if (verbose) printk(KERN_ALERT "%s" TORTURE_FLAG s "\n", torture_type); } while (0) do { if (verbose) pr_alert("%s" TORTURE_FLAG s "\n", torture_type); } while (0)
#define VERBOSE_PRINTK_ERRSTRING(s) \ #define VERBOSE_PRINTK_ERRSTRING(s) \
do { if (verbose) printk(KERN_ALERT "%s" TORTURE_FLAG "!!! " s "\n", torture_type); } while (0) do { if (verbose) pr_alert("%s" TORTURE_FLAG "!!! " s "\n", torture_type); } while (0)
static char printk_buf[4096]; static char printk_buf[4096];
...@@ -176,8 +177,14 @@ static long n_rcu_torture_boosts; ...@@ -176,8 +177,14 @@ static long n_rcu_torture_boosts;
static long n_rcu_torture_timers; static long n_rcu_torture_timers;
static long n_offline_attempts; static long n_offline_attempts;
static long n_offline_successes; static long n_offline_successes;
static unsigned long sum_offline;
static int min_offline = -1;
static int max_offline;
static long n_online_attempts; static long n_online_attempts;
static long n_online_successes; static long n_online_successes;
static unsigned long sum_online;
static int min_online = -1;
static int max_online;
static long n_barrier_attempts; static long n_barrier_attempts;
static long n_barrier_successes; static long n_barrier_successes;
static struct list_head rcu_torture_removed; static struct list_head rcu_torture_removed;
...@@ -235,7 +242,7 @@ rcutorture_shutdown_notify(struct notifier_block *unused1, ...@@ -235,7 +242,7 @@ rcutorture_shutdown_notify(struct notifier_block *unused1,
if (fullstop == FULLSTOP_DONTSTOP) if (fullstop == FULLSTOP_DONTSTOP)
fullstop = FULLSTOP_SHUTDOWN; fullstop = FULLSTOP_SHUTDOWN;
else else
printk(KERN_WARNING /* but going down anyway, so... */ pr_warn(/* but going down anyway, so... */
"Concurrent 'rmmod rcutorture' and shutdown illegal!\n"); "Concurrent 'rmmod rcutorture' and shutdown illegal!\n");
mutex_unlock(&fullstop_mutex); mutex_unlock(&fullstop_mutex);
return NOTIFY_DONE; return NOTIFY_DONE;
...@@ -248,7 +255,7 @@ rcutorture_shutdown_notify(struct notifier_block *unused1, ...@@ -248,7 +255,7 @@ rcutorture_shutdown_notify(struct notifier_block *unused1,
static void rcutorture_shutdown_absorb(char *title) static void rcutorture_shutdown_absorb(char *title)
{ {
if (ACCESS_ONCE(fullstop) == FULLSTOP_SHUTDOWN) { if (ACCESS_ONCE(fullstop) == FULLSTOP_SHUTDOWN) {
printk(KERN_NOTICE pr_notice(
"rcutorture thread %s parking due to system shutdown\n", "rcutorture thread %s parking due to system shutdown\n",
title); title);
schedule_timeout_uninterruptible(MAX_SCHEDULE_TIMEOUT); schedule_timeout_uninterruptible(MAX_SCHEDULE_TIMEOUT);
...@@ -1214,11 +1221,13 @@ rcu_torture_printk(char *page) ...@@ -1214,11 +1221,13 @@ rcu_torture_printk(char *page)
n_rcu_torture_boost_failure, n_rcu_torture_boost_failure,
n_rcu_torture_boosts, n_rcu_torture_boosts,
n_rcu_torture_timers); n_rcu_torture_timers);
cnt += sprintf(&page[cnt], "onoff: %ld/%ld:%ld/%ld ", cnt += sprintf(&page[cnt],
n_online_successes, "onoff: %ld/%ld:%ld/%ld %d,%d:%d,%d %lu:%lu (HZ=%d) ",
n_online_attempts, n_online_successes, n_online_attempts,
n_offline_successes, n_offline_successes, n_offline_attempts,
n_offline_attempts); min_online, max_online,
min_offline, max_offline,
sum_online, sum_offline, HZ);
cnt += sprintf(&page[cnt], "barrier: %ld/%ld:%ld", cnt += sprintf(&page[cnt], "barrier: %ld/%ld:%ld",
n_barrier_successes, n_barrier_successes,
n_barrier_attempts, n_barrier_attempts,
...@@ -1267,7 +1276,7 @@ rcu_torture_stats_print(void) ...@@ -1267,7 +1276,7 @@ rcu_torture_stats_print(void)
int cnt; int cnt;
cnt = rcu_torture_printk(printk_buf); cnt = rcu_torture_printk(printk_buf);
printk(KERN_ALERT "%s", printk_buf); pr_alert("%s", printk_buf);
} }
/* /*
...@@ -1380,7 +1389,7 @@ rcu_torture_stutter(void *arg) ...@@ -1380,7 +1389,7 @@ rcu_torture_stutter(void *arg)
static inline void static inline void
rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, char *tag) rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, char *tag)
{ {
printk(KERN_ALERT "%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"--- %s: nreaders=%d nfakewriters=%d " "--- %s: nreaders=%d nfakewriters=%d "
"stat_interval=%d verbose=%d test_no_idle_hz=%d " "stat_interval=%d verbose=%d test_no_idle_hz=%d "
"shuffle_interval=%d stutter=%d irqreader=%d " "shuffle_interval=%d stutter=%d irqreader=%d "
...@@ -1460,7 +1469,7 @@ rcu_torture_shutdown(void *arg) ...@@ -1460,7 +1469,7 @@ rcu_torture_shutdown(void *arg)
!kthread_should_stop()) { !kthread_should_stop()) {
delta = shutdown_time - jiffies_snap; delta = shutdown_time - jiffies_snap;
if (verbose) if (verbose)
printk(KERN_ALERT "%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"rcu_torture_shutdown task: %lu jiffies remaining\n", "rcu_torture_shutdown task: %lu jiffies remaining\n",
torture_type, delta); torture_type, delta);
schedule_timeout_interruptible(delta); schedule_timeout_interruptible(delta);
...@@ -1490,8 +1499,10 @@ static int __cpuinit ...@@ -1490,8 +1499,10 @@ static int __cpuinit
rcu_torture_onoff(void *arg) rcu_torture_onoff(void *arg)
{ {
int cpu; int cpu;
unsigned long delta;
int maxcpu = -1; int maxcpu = -1;
DEFINE_RCU_RANDOM(rand); DEFINE_RCU_RANDOM(rand);
unsigned long starttime;
VERBOSE_PRINTK_STRING("rcu_torture_onoff task started"); VERBOSE_PRINTK_STRING("rcu_torture_onoff task started");
for_each_online_cpu(cpu) for_each_online_cpu(cpu)
...@@ -1506,29 +1517,51 @@ rcu_torture_onoff(void *arg) ...@@ -1506,29 +1517,51 @@ rcu_torture_onoff(void *arg)
cpu = (rcu_random(&rand) >> 4) % (maxcpu + 1); cpu = (rcu_random(&rand) >> 4) % (maxcpu + 1);
if (cpu_online(cpu) && cpu_is_hotpluggable(cpu)) { if (cpu_online(cpu) && cpu_is_hotpluggable(cpu)) {
if (verbose) if (verbose)
printk(KERN_ALERT "%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"rcu_torture_onoff task: offlining %d\n", "rcu_torture_onoff task: offlining %d\n",
torture_type, cpu); torture_type, cpu);
starttime = jiffies;
n_offline_attempts++; n_offline_attempts++;
if (cpu_down(cpu) == 0) { if (cpu_down(cpu) == 0) {
if (verbose) if (verbose)
printk(KERN_ALERT "%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"rcu_torture_onoff task: offlined %d\n", "rcu_torture_onoff task: offlined %d\n",
torture_type, cpu); torture_type, cpu);
n_offline_successes++; n_offline_successes++;
delta = jiffies - starttime;
sum_offline += delta;
if (min_offline < 0) {
min_offline = delta;
max_offline = delta;
}
if (min_offline > delta)
min_offline = delta;
if (max_offline < delta)
max_offline = delta;
} }
} else if (cpu_is_hotpluggable(cpu)) { } else if (cpu_is_hotpluggable(cpu)) {
if (verbose) if (verbose)
printk(KERN_ALERT "%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"rcu_torture_onoff task: onlining %d\n", "rcu_torture_onoff task: onlining %d\n",
torture_type, cpu); torture_type, cpu);
starttime = jiffies;
n_online_attempts++; n_online_attempts++;
if (cpu_up(cpu) == 0) { if (cpu_up(cpu) == 0) {
if (verbose) if (verbose)
printk(KERN_ALERT "%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"rcu_torture_onoff task: onlined %d\n", "rcu_torture_onoff task: onlined %d\n",
torture_type, cpu); torture_type, cpu);
n_online_successes++; n_online_successes++;
delta = jiffies - starttime;
sum_online += delta;
if (min_online < 0) {
min_online = delta;
max_online = delta;
}
if (min_online > delta)
min_online = delta;
if (max_online < delta)
max_online = delta;
} }
} }
schedule_timeout_interruptible(onoff_interval * HZ); schedule_timeout_interruptible(onoff_interval * HZ);
...@@ -1593,14 +1626,14 @@ static int __cpuinit rcu_torture_stall(void *args) ...@@ -1593,14 +1626,14 @@ static int __cpuinit rcu_torture_stall(void *args)
if (!kthread_should_stop()) { if (!kthread_should_stop()) {
stop_at = get_seconds() + stall_cpu; stop_at = get_seconds() + stall_cpu;
/* RCU CPU stall is expected behavior in following code. */ /* RCU CPU stall is expected behavior in following code. */
printk(KERN_ALERT "rcu_torture_stall start.\n"); pr_alert("rcu_torture_stall start.\n");
rcu_read_lock(); rcu_read_lock();
preempt_disable(); preempt_disable();
while (ULONG_CMP_LT(get_seconds(), stop_at)) while (ULONG_CMP_LT(get_seconds(), stop_at))
continue; /* Induce RCU CPU stall warning. */ continue; /* Induce RCU CPU stall warning. */
preempt_enable(); preempt_enable();
rcu_read_unlock(); rcu_read_unlock();
printk(KERN_ALERT "rcu_torture_stall end.\n"); pr_alert("rcu_torture_stall end.\n");
} }
rcutorture_shutdown_absorb("rcu_torture_stall"); rcutorture_shutdown_absorb("rcu_torture_stall");
while (!kthread_should_stop()) while (!kthread_should_stop())
...@@ -1716,10 +1749,10 @@ static int rcu_torture_barrier_init(void) ...@@ -1716,10 +1749,10 @@ static int rcu_torture_barrier_init(void)
if (n_barrier_cbs == 0) if (n_barrier_cbs == 0)
return 0; return 0;
if (cur_ops->call == NULL || cur_ops->cb_barrier == NULL) { if (cur_ops->call == NULL || cur_ops->cb_barrier == NULL) {
printk(KERN_ALERT "%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
" Call or barrier ops missing for %s,\n", " Call or barrier ops missing for %s,\n",
torture_type, cur_ops->name); torture_type, cur_ops->name);
printk(KERN_ALERT "%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
" RCU barrier testing omitted from run.\n", " RCU barrier testing omitted from run.\n",
torture_type); torture_type);
return 0; return 0;
...@@ -1814,7 +1847,7 @@ rcu_torture_cleanup(void) ...@@ -1814,7 +1847,7 @@ rcu_torture_cleanup(void)
mutex_lock(&fullstop_mutex); mutex_lock(&fullstop_mutex);
rcutorture_record_test_transition(); rcutorture_record_test_transition();
if (fullstop == FULLSTOP_SHUTDOWN) { if (fullstop == FULLSTOP_SHUTDOWN) {
printk(KERN_WARNING /* but going down anyway, so... */ pr_warn(/* but going down anyway, so... */
"Concurrent 'rmmod rcutorture' and shutdown illegal!\n"); "Concurrent 'rmmod rcutorture' and shutdown illegal!\n");
mutex_unlock(&fullstop_mutex); mutex_unlock(&fullstop_mutex);
schedule_timeout_uninterruptible(10); schedule_timeout_uninterruptible(10);
...@@ -1938,17 +1971,17 @@ rcu_torture_init(void) ...@@ -1938,17 +1971,17 @@ rcu_torture_init(void)
break; break;
} }
if (i == ARRAY_SIZE(torture_ops)) { if (i == ARRAY_SIZE(torture_ops)) {
printk(KERN_ALERT "rcu-torture: invalid torture type: \"%s\"\n", pr_alert("rcu-torture: invalid torture type: \"%s\"\n",
torture_type); torture_type);
printk(KERN_ALERT "rcu-torture types:"); pr_alert("rcu-torture types:");
for (i = 0; i < ARRAY_SIZE(torture_ops); i++) for (i = 0; i < ARRAY_SIZE(torture_ops); i++)
printk(KERN_ALERT " %s", torture_ops[i]->name); pr_alert(" %s", torture_ops[i]->name);
printk(KERN_ALERT "\n"); pr_alert("\n");
mutex_unlock(&fullstop_mutex); mutex_unlock(&fullstop_mutex);
return -EINVAL; return -EINVAL;
} }
if (cur_ops->fqs == NULL && fqs_duration != 0) { if (cur_ops->fqs == NULL && fqs_duration != 0) {
printk(KERN_ALERT "rcu-torture: ->fqs NULL and non-zero fqs_duration, fqs disabled.\n"); pr_alert("rcu-torture: ->fqs NULL and non-zero fqs_duration, fqs disabled.\n");
fqs_duration = 0; fqs_duration = 0;
} }
if (cur_ops->init) if (cur_ops->init)
...@@ -1996,7 +2029,7 @@ rcu_torture_init(void) ...@@ -1996,7 +2029,7 @@ rcu_torture_init(void)
/* Start up the kthreads. */ /* Start up the kthreads. */
VERBOSE_PRINTK_STRING("Creating rcu_torture_writer task"); VERBOSE_PRINTK_STRING("Creating rcu_torture_writer task");
writer_task = kthread_run(rcu_torture_writer, NULL, writer_task = kthread_create(rcu_torture_writer, NULL,
"rcu_torture_writer"); "rcu_torture_writer");
if (IS_ERR(writer_task)) { if (IS_ERR(writer_task)) {
firsterr = PTR_ERR(writer_task); firsterr = PTR_ERR(writer_task);
...@@ -2004,6 +2037,7 @@ rcu_torture_init(void) ...@@ -2004,6 +2037,7 @@ rcu_torture_init(void)
writer_task = NULL; writer_task = NULL;
goto unwind; goto unwind;
} }
wake_up_process(writer_task);
fakewriter_tasks = kzalloc(nfakewriters * sizeof(fakewriter_tasks[0]), fakewriter_tasks = kzalloc(nfakewriters * sizeof(fakewriter_tasks[0]),
GFP_KERNEL); GFP_KERNEL);
if (fakewriter_tasks == NULL) { if (fakewriter_tasks == NULL) {
...@@ -2118,7 +2152,7 @@ rcu_torture_init(void) ...@@ -2118,7 +2152,7 @@ rcu_torture_init(void)
} }
if (shutdown_secs > 0) { if (shutdown_secs > 0) {
shutdown_time = jiffies + shutdown_secs * HZ; shutdown_time = jiffies + shutdown_secs * HZ;
shutdown_task = kthread_run(rcu_torture_shutdown, NULL, shutdown_task = kthread_create(rcu_torture_shutdown, NULL,
"rcu_torture_shutdown"); "rcu_torture_shutdown");
if (IS_ERR(shutdown_task)) { if (IS_ERR(shutdown_task)) {
firsterr = PTR_ERR(shutdown_task); firsterr = PTR_ERR(shutdown_task);
...@@ -2126,6 +2160,7 @@ rcu_torture_init(void) ...@@ -2126,6 +2160,7 @@ rcu_torture_init(void)
shutdown_task = NULL; shutdown_task = NULL;
goto unwind; goto unwind;
} }
wake_up_process(shutdown_task);
} }
i = rcu_torture_onoff_init(); i = rcu_torture_onoff_init();
if (i != 0) { if (i != 0) {
......
This diff is collapsed.
...@@ -396,9 +396,6 @@ struct rcu_state { ...@@ -396,9 +396,6 @@ struct rcu_state {
struct rcu_head **orphan_donetail; /* Tail of above. */ struct rcu_head **orphan_donetail; /* Tail of above. */
long qlen_lazy; /* Number of lazy callbacks. */ long qlen_lazy; /* Number of lazy callbacks. */
long qlen; /* Total number of callbacks. */ long qlen; /* Total number of callbacks. */
struct task_struct *rcu_barrier_in_progress;
/* Task doing rcu_barrier(), */
/* or NULL if no barrier. */
struct mutex barrier_mutex; /* Guards barrier fields. */ struct mutex barrier_mutex; /* Guards barrier fields. */
atomic_t barrier_cpu_count; /* # CPUs waiting on. */ atomic_t barrier_cpu_count; /* # CPUs waiting on. */
struct completion barrier_completion; /* Wake at barrier end. */ struct completion barrier_completion; /* Wake at barrier end. */
......
...@@ -421,9 +421,11 @@ static void rcu_print_detail_task_stall_rnp(struct rcu_node *rnp) ...@@ -421,9 +421,11 @@ static void rcu_print_detail_task_stall_rnp(struct rcu_node *rnp)
unsigned long flags; unsigned long flags;
struct task_struct *t; struct task_struct *t;
if (!rcu_preempt_blocked_readers_cgp(rnp))
return;
raw_spin_lock_irqsave(&rnp->lock, flags); raw_spin_lock_irqsave(&rnp->lock, flags);
if (!rcu_preempt_blocked_readers_cgp(rnp)) {
raw_spin_unlock_irqrestore(&rnp->lock, flags);
return;
}
t = list_entry(rnp->gp_tasks, t = list_entry(rnp->gp_tasks,
struct task_struct, rcu_node_entry); struct task_struct, rcu_node_entry);
list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry)
...@@ -583,17 +585,23 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp, ...@@ -583,17 +585,23 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
raw_spin_unlock(&rnp_root->lock); /* irqs still disabled */ raw_spin_unlock(&rnp_root->lock); /* irqs still disabled */
} }
rnp->gp_tasks = NULL;
rnp->exp_tasks = NULL;
#ifdef CONFIG_RCU_BOOST #ifdef CONFIG_RCU_BOOST
/* In case root is being boosted and leaf is not. */ rnp->boost_tasks = NULL;
/*
* In case root is being boosted and leaf was not. Make sure
* that we boost the tasks blocking the current grace period
* in this case.
*/
raw_spin_lock(&rnp_root->lock); /* irqs already disabled */ raw_spin_lock(&rnp_root->lock); /* irqs already disabled */
if (rnp_root->boost_tasks != NULL && if (rnp_root->boost_tasks != NULL &&
rnp_root->boost_tasks != rnp_root->gp_tasks) rnp_root->boost_tasks != rnp_root->gp_tasks &&
rnp_root->boost_tasks != rnp_root->exp_tasks)
rnp_root->boost_tasks = rnp_root->gp_tasks; rnp_root->boost_tasks = rnp_root->gp_tasks;
raw_spin_unlock(&rnp_root->lock); /* irqs still disabled */ raw_spin_unlock(&rnp_root->lock); /* irqs still disabled */
#endif /* #ifdef CONFIG_RCU_BOOST */ #endif /* #ifdef CONFIG_RCU_BOOST */
rnp->gp_tasks = NULL;
rnp->exp_tasks = NULL;
return retval; return retval;
} }
...@@ -1204,9 +1212,9 @@ static int rcu_boost_kthread(void *arg) ...@@ -1204,9 +1212,9 @@ static int rcu_boost_kthread(void *arg)
* kthread to start boosting them. If there is an expedited grace * kthread to start boosting them. If there is an expedited grace
* period in progress, it is always time to boost. * period in progress, it is always time to boost.
* *
* The caller must hold rnp->lock, which this function releases, * The caller must hold rnp->lock, which this function releases.
* but irqs remain disabled. The ->boost_kthread_task is immortal, * The ->boost_kthread_task is immortal, so we don't need to worry
* so we don't need to worry about it going away. * about it going away.
*/ */
static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags) static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
{ {
...@@ -2217,11 +2225,15 @@ static void print_cpu_stall_fast_no_hz(char *cp, int cpu) ...@@ -2217,11 +2225,15 @@ static void print_cpu_stall_fast_no_hz(char *cp, int cpu)
{ {
struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu); struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
struct timer_list *tltp = &rdtp->idle_gp_timer; struct timer_list *tltp = &rdtp->idle_gp_timer;
char c;
c = rdtp->dyntick_holdoff == jiffies ? 'H' : '.';
if (timer_pending(tltp))
sprintf(cp, "drain=%d %c timer=%lu", sprintf(cp, "drain=%d %c timer=%lu",
rdtp->dyntick_drain, rdtp->dyntick_drain, c, tltp->expires - jiffies);
rdtp->dyntick_holdoff == jiffies ? 'H' : '.', else
timer_pending(tltp) ? tltp->expires - jiffies : -1); sprintf(cp, "drain=%d %c timer not pending",
rdtp->dyntick_drain, c);
} }
#else /* #ifdef CONFIG_RCU_FAST_NO_HZ */ #else /* #ifdef CONFIG_RCU_FAST_NO_HZ */
...@@ -2289,11 +2301,10 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp) ...@@ -2289,11 +2301,10 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp)
/* Increment ->ticks_this_gp for all flavors of RCU. */ /* Increment ->ticks_this_gp for all flavors of RCU. */
static void increment_cpu_stall_ticks(void) static void increment_cpu_stall_ticks(void)
{ {
__get_cpu_var(rcu_sched_data).ticks_this_gp++; struct rcu_state *rsp;
__get_cpu_var(rcu_bh_data).ticks_this_gp++;
#ifdef CONFIG_TREE_PREEMPT_RCU for_each_rcu_flavor(rsp)
__get_cpu_var(rcu_preempt_data).ticks_this_gp++; __this_cpu_ptr(rsp->rda)->ticks_this_gp++;
#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
} }
#else /* #ifdef CONFIG_RCU_CPU_STALL_INFO */ #else /* #ifdef CONFIG_RCU_CPU_STALL_INFO */
......
...@@ -51,8 +51,8 @@ static int show_rcubarrier(struct seq_file *m, void *unused) ...@@ -51,8 +51,8 @@ static int show_rcubarrier(struct seq_file *m, void *unused)
struct rcu_state *rsp; struct rcu_state *rsp;
for_each_rcu_flavor(rsp) for_each_rcu_flavor(rsp)
seq_printf(m, "%s: %c bcc: %d nbd: %lu\n", seq_printf(m, "%s: bcc: %d nbd: %lu\n",
rsp->name, rsp->rcu_barrier_in_progress ? 'B' : '.', rsp->name,
atomic_read(&rsp->barrier_cpu_count), atomic_read(&rsp->barrier_cpu_count),
rsp->n_barrier_done); rsp->n_barrier_done);
return 0; return 0;
......
...@@ -5304,27 +5304,17 @@ void idle_task_exit(void) ...@@ -5304,27 +5304,17 @@ void idle_task_exit(void)
} }
/* /*
* While a dead CPU has no uninterruptible tasks queued at this point, * Since this CPU is going 'away' for a while, fold any nr_active delta
* it might still have a nonzero ->nr_uninterruptible counter, because * we might have. Assumes we're called after migrate_tasks() so that the
* for performance reasons the counter is not stricly tracking tasks to * nr_active count is stable.
* their home CPUs. So we just add the counter to another CPU's counter, *
* to keep the global sum constant after CPU-down: * Also see the comment "Global load-average calculations".
*/
static void migrate_nr_uninterruptible(struct rq *rq_src)
{
struct rq *rq_dest = cpu_rq(cpumask_any(cpu_active_mask));
rq_dest->nr_uninterruptible += rq_src->nr_uninterruptible;
rq_src->nr_uninterruptible = 0;
}
/*
* remove the tasks which were accounted by rq from calc_load_tasks.
*/ */
static void calc_global_load_remove(struct rq *rq) static void calc_load_migrate(struct rq *rq)
{ {
atomic_long_sub(rq->calc_load_active, &calc_load_tasks); long delta = calc_load_fold_active(rq);
rq->calc_load_active = 0; if (delta)
atomic_long_add(delta, &calc_load_tasks);
} }
/* /*
...@@ -5617,9 +5607,18 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu) ...@@ -5617,9 +5607,18 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)
migrate_tasks(cpu); migrate_tasks(cpu);
BUG_ON(rq->nr_running != 1); /* the migration thread */ BUG_ON(rq->nr_running != 1); /* the migration thread */
raw_spin_unlock_irqrestore(&rq->lock, flags); raw_spin_unlock_irqrestore(&rq->lock, flags);
break;
case CPU_DEAD:
{
struct rq *dest_rq;
migrate_nr_uninterruptible(rq); local_irq_save(flags);
calc_global_load_remove(rq); dest_rq = cpu_rq(smp_processor_id());
raw_spin_lock(&dest_rq->lock);
calc_load_migrate(rq);
raw_spin_unlock_irqrestore(&dest_rq->lock, flags);
}
break; break;
#endif #endif
} }
......
...@@ -436,7 +436,8 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts) ...@@ -436,7 +436,8 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
if (unlikely(local_softirq_pending() && cpu_online(cpu))) { if (unlikely(local_softirq_pending() && cpu_online(cpu))) {
static int ratelimit; static int ratelimit;
if (ratelimit < 10) { if (ratelimit < 10 &&
(local_softirq_pending() & SOFTIRQ_STOP_IDLE_MASK)) {
printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n", printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
(unsigned int) local_softirq_pending()); (unsigned int) local_softirq_pending());
ratelimit++; ratelimit++;
......
...@@ -629,6 +629,20 @@ config PROVE_RCU_REPEATEDLY ...@@ -629,6 +629,20 @@ config PROVE_RCU_REPEATEDLY
Say N if you are unsure. Say N if you are unsure.
config PROVE_RCU_DELAY
bool "RCU debugging: preemptible RCU race provocation"
depends on DEBUG_KERNEL && PREEMPT_RCU
default n
help
There is a class of races that involve an unlikely preemption
of __rcu_read_unlock() just after ->rcu_read_lock_nesting has
been set to INT_MIN. This feature inserts a delay at that
point to increase the probability of these races.
Say Y to increase probability of preemption of __rcu_read_unlock().
Say N if you are unsure.
config SPARSE_RCU_POINTER config SPARSE_RCU_POINTER
bool "RCU debugging: sparse-based checks for pointer usage" bool "RCU debugging: sparse-based checks for pointer usage"
default n default n
......
...@@ -1483,13 +1483,11 @@ static void *kmemleak_seq_next(struct seq_file *seq, void *v, loff_t *pos) ...@@ -1483,13 +1483,11 @@ static void *kmemleak_seq_next(struct seq_file *seq, void *v, loff_t *pos)
{ {
struct kmemleak_object *prev_obj = v; struct kmemleak_object *prev_obj = v;
struct kmemleak_object *next_obj = NULL; struct kmemleak_object *next_obj = NULL;
struct list_head *n = &prev_obj->object_list; struct kmemleak_object *obj = prev_obj;
++(*pos); ++(*pos);
list_for_each_continue_rcu(n, &object_list) { list_for_each_entry_continue_rcu(obj, &object_list, object_list) {
struct kmemleak_object *obj =
list_entry(n, struct kmemleak_object, object_list);
if (get_object(obj)) { if (get_object(obj)) {
next_obj = obj; next_obj = obj;
break; break;
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment