Commit b99cdfdf authored by Linus Torvalds's avatar Linus Torvalds

Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull RCU updates from Thomas Gleixner:
 "A large update to RCU:

  Preparatory work for consolidating the RCU flavors:

   - Introduce grace-period sequence numbers to the RCU-bh, RCU-preempt,
     and RCU-sched flavors, replacing the old ->gpnum and ->completed
     pair of fields.

     This change allows lockless code to obtain the complete
     grace-period state with a single READ_ONCE(), which is needed to
     maintain tolerable lock contention during the upcoming
     consolidation of the three RCU flavors.

     Note that grace-period sequence numbers are already used by
     rcu_barrier(), expedited RCU grace periods, and SRCU, and are thus
     already heavily used and well-tested. Joel Fernandes contributed a
     number of excellent fixes and improvements.

   - Clean up some grace-period-reporting loose ends, including
     improving the handling of quiescent states from offline CPUs and
     fixing some false-positive WARN_ON_ONCE() invocations.

     (Strictly speaking, the WARN_ON_ONCE() invocations were quite
     correct, but their invariants were (harmlessly) violated by the
     earlier sloppy handling of quiescent states from offline CPUs.)

     In addition, improve grace-period forward-progress guarantees so as
     to allow removal of fail-safe checks that required otherwise
     needless lock acquisitions. Finally, add more diagnostics to help
     debug the upcoming consolidation of the RCU-bh, RCU-preempt, and
     RCU-sched flavors.

  The rest:

   - SRCU updates

   - Updates to rcutorture and associated scripting.

   - The usual pile of miscellaneous fixes"

* 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (118 commits)
  rcutorture: Fix rcu_barrier successes counter
  rcutorture: Add support to detect if boost kthread prio is too low
  rcutorture: Use monotonic timestamp for stall detection
  rcutorture: Make boost test more robust
  rcutorture: Disable RT throttling for boost tests
  rcutorture: Emphasize testing of single reader protection type
  rcutorture: Handle extended read-side critical sections
  rcutorture: Make rcu_torture_timer() use rcu_torture_one_read()
  rcutorture: Use per-CPU random state for rcu_torture_timer()
  rcutorture: Use atomic increment for n_rcu_torture_timers
  rcutorture: Extract common code from rcu_torture_reader()
  rcuperf: Remove unused torturing_tasks() function
  rcu: Remove rcutorture test version and sequence number
  rcutorture: Change units of onoff_interval to jiffies
  rcu: Assign higher prio to RCU threads if rcutorture is built-in
  rculist: Improve documentation for list_for_each_entry_from_rcu()
  srcu: Add grace-period number to rcutorture statistics printout
  rcu: Print stall-warning NMI dyntick state in hexadecimal
  MAINTAINERS: Update RCU, SRCU, and TORTURE-TEST entries
  rcu: Make rcu_seq_diff() more exact
  ...
parents d0daaeaf ea73a5c6
...@@ -380,31 +380,26 @@ and therefore need no protection. ...@@ -380,31 +380,26 @@ and therefore need no protection.
as follows: as follows:
<pre> <pre>
1 unsigned long gpnum; 1 unsigned long gp_seq;
2 unsigned long completed;
</pre> </pre>
<p>RCU grace periods are numbered, and <p>RCU grace periods are numbered, and
the <tt>-&gt;gpnum</tt> field contains the number of the grace the <tt>-&gt;gp_seq</tt> field contains the current grace-period
period that started most recently. sequence number.
The <tt>-&gt;completed</tt> field contains the number of the The bottom two bits are the state of the current grace period,
grace period that completed most recently. which can be zero for not yet started or one for in progress.
If the two fields are equal, the RCU grace period that most recently In other words, if the bottom two bits of <tt>-&gt;gp_seq</tt> are
started has already completed, and therefore the corresponding zero, the corresponding flavor of RCU is idle.
flavor of RCU is idle. Any other value in the bottom two bits indicates that something is broken.
If <tt>-&gt;gpnum</tt> is one greater than <tt>-&gt;completed</tt>, This field is protected by the root <tt>rcu_node</tt> structure's
then <tt>-&gt;gpnum</tt> gives the number of the current RCU
grace period, which has not yet completed.
Any other combination of values indicates that something is broken.
These two fields are protected by the root <tt>rcu_node</tt>'s
<tt>-&gt;lock</tt> field. <tt>-&gt;lock</tt> field.
</p><p>There are <tt>-&gt;gpnum</tt> and <tt>-&gt;completed</tt> fields </p><p>There are <tt>-&gt;gp_seq</tt> fields
in the <tt>rcu_node</tt> and <tt>rcu_data</tt> structures in the <tt>rcu_node</tt> and <tt>rcu_data</tt> structures
as well. as well.
The fields in the <tt>rcu_state</tt> structure represent the The fields in the <tt>rcu_state</tt> structure represent the
most current values, and those of the other structures are compared most current value, and those of the other structures are compared
in order to detect the start of a new grace period in a distributed in order to detect the beginnings and ends of grace periods in a distributed
fashion. fashion.
The values flow from <tt>rcu_state</tt> to <tt>rcu_node</tt> The values flow from <tt>rcu_state</tt> to <tt>rcu_node</tt>
(down the tree from the root to the leaves) to <tt>rcu_data</tt>. (down the tree from the root to the leaves) to <tt>rcu_data</tt>.
...@@ -512,27 +507,47 @@ than to be heisenbugged out of existence. ...@@ -512,27 +507,47 @@ than to be heisenbugged out of existence.
as follows: as follows:
<pre> <pre>
1 unsigned long gpnum; 1 unsigned long gp_seq;
2 unsigned long completed; 2 unsigned long gp_seq_needed;
</pre> </pre>
<p>These fields are the counterparts of the fields of the same name in <p>The <tt>rcu_node</tt> structures' <tt>-&gt;gp_seq</tt> fields are
the <tt>rcu_state</tt> structure. the counterparts of the field of the same name in the <tt>rcu_state</tt>
They each may lag up to one behind their <tt>rcu_state</tt> structure.
counterparts. They each may lag up to one step behind their <tt>rcu_state</tt>
If a given <tt>rcu_node</tt> structure's <tt>-&gt;gpnum</tt> and counterpart.
<tt>-&gt;complete</tt> fields are equal, then this <tt>rcu_node</tt> If the bottom two bits of a given <tt>rcu_node</tt> structure's
<tt>-&gt;gp_seq</tt> field is zero, then this <tt>rcu_node</tt>
structure believes that RCU is idle. structure believes that RCU is idle.
Otherwise, as with the <tt>rcu_state</tt> structure, </p><p>The <tt>&gt;gp_seq</tt> field of each <tt>rcu_node</tt>
the <tt>-&gt;gpnum</tt> field will be one greater than the structure is updated at the beginning and the end
<tt>-&gt;complete</tt> fields, with <tt>-&gt;gpnum</tt> of each grace period.
indicating which grace period this <tt>rcu_node</tt> believes
is still being waited for. <p>The <tt>-&gt;gp_seq_needed</tt> fields record the
furthest-in-the-future grace period request seen by the corresponding
<tt>rcu_node</tt> structure. The request is considered fulfilled when
the value of the <tt>-&gt;gp_seq</tt> field equals or exceeds that of
the <tt>-&gt;gp_seq_needed</tt> field.
</p><p>The <tt>&gt;gpnum</tt> field of each <tt>rcu_node</tt> <table>
structure is updated at the beginning <tr><th>&nbsp;</th></tr>
of each grace period, and the <tt>-&gt;completed</tt> fields are <tr><th align="left">Quick Quiz:</th></tr>
updated at the end of each grace period. <tr><td>
Suppose that this <tt>rcu_node</tt> structure doesn't see
a request for a very long time.
Won't wrapping of the <tt>-&gt;gp_seq</tt> field cause
problems?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
No, because if the <tt>-&gt;gp_seq_needed</tt> field lags behind the
<tt>-&gt;gp_seq</tt> field, the <tt>-&gt;gp_seq_needed</tt> field
will be updated at the end of the grace period.
Modulo-arithmetic comparisons therefore will always get the
correct answer, even with wrapping.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h5>Quiescent-State Tracking</h5> <h5>Quiescent-State Tracking</h5>
...@@ -626,9 +641,8 @@ normal and expedited grace periods, respectively. ...@@ -626,9 +641,8 @@ normal and expedited grace periods, respectively.
</ol> </ol>
<p><font color="ffffff">So the locking is absolutely required in <p><font color="ffffff">So the locking is absolutely required in
order to coordinate order to coordinate clearing of the bits with updating of the
clearing of the bits with the grace-period numbers in grace-period sequence number in <tt>-&gt;gp_seq</tt>.
<tt>-&gt;gpnum</tt> and <tt>-&gt;completed</tt>.
</font></td></tr> </font></td></tr>
<tr><td>&nbsp;</td></tr> <tr><td>&nbsp;</td></tr>
</table> </table>
...@@ -1038,15 +1052,15 @@ out any <tt>rcu_data</tt> structure for which this flag is not set. ...@@ -1038,15 +1052,15 @@ out any <tt>rcu_data</tt> structure for which this flag is not set.
as follows: as follows:
<pre> <pre>
1 unsigned long completed; 1 unsigned long gp_seq;
2 unsigned long gpnum; 2 unsigned long gp_seq_needed;
3 bool cpu_no_qs; 3 bool cpu_no_qs;
4 bool core_needs_qs; 4 bool core_needs_qs;
5 bool gpwrap; 5 bool gpwrap;
6 unsigned long rcu_qs_ctr_snap; 6 unsigned long rcu_qs_ctr_snap;
</pre> </pre>
<p>The <tt>completed</tt> and <tt>gpnum</tt> <p>The <tt>-&gt;gp_seq</tt> and <tt>-&gt;gp_seq_needed</tt>
fields are the counterparts of the fields of the same name fields are the counterparts of the fields of the same name
in the <tt>rcu_state</tt> and <tt>rcu_node</tt> structures. in the <tt>rcu_state</tt> and <tt>rcu_node</tt> structures.
They may each lag up to one behind their <tt>rcu_node</tt> They may each lag up to one behind their <tt>rcu_node</tt>
...@@ -1054,15 +1068,9 @@ counterparts, but in <tt>CONFIG_NO_HZ_IDLE</tt> and ...@@ -1054,15 +1068,9 @@ counterparts, but in <tt>CONFIG_NO_HZ_IDLE</tt> and
<tt>CONFIG_NO_HZ_FULL</tt> kernels can lag <tt>CONFIG_NO_HZ_FULL</tt> kernels can lag
arbitrarily far behind for CPUs in dyntick-idle mode (but these counters arbitrarily far behind for CPUs in dyntick-idle mode (but these counters
will catch up upon exit from dyntick-idle mode). will catch up upon exit from dyntick-idle mode).
If a given <tt>rcu_data</tt> structure's <tt>-&gt;gpnum</tt> and If the lower two bits of a given <tt>rcu_data</tt> structure's
<tt>-&gt;complete</tt> fields are equal, then this <tt>rcu_data</tt> <tt>-&gt;gp_seq</tt> are zero, then this <tt>rcu_data</tt>
structure believes that RCU is idle. structure believes that RCU is idle.
Otherwise, as with the <tt>rcu_state</tt> and <tt>rcu_node</tt>
structure,
the <tt>-&gt;gpnum</tt> field will be one greater than the
<tt>-&gt;complete</tt> fields, with <tt>-&gt;gpnum</tt>
indicating which grace period this <tt>rcu_data</tt> believes
is still being waited for.
<table> <table>
<tr><th>&nbsp;</th></tr> <tr><th>&nbsp;</th></tr>
...@@ -1070,13 +1078,13 @@ is still being waited for. ...@@ -1070,13 +1078,13 @@ is still being waited for.
<tr><td> <tr><td>
All this replication of the grace period numbers can only cause All this replication of the grace period numbers can only cause
massive confusion. massive confusion.
Why not just keep a global pair of counters and be done with it??? Why not just keep a global sequence number and be done with it???
</td></tr> </td></tr>
<tr><th align="left">Answer:</th></tr> <tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff"> <tr><td bgcolor="#ffffff"><font color="ffffff">
Because if there was only a single global pair of grace-period Because if there was only a single global sequence
numbers, there would need to be a single global lock to allow numbers, there would need to be a single global lock to allow
safely accessing and updating them. safely accessing and updating it.
And if we are not going to have a single global lock, we need And if we are not going to have a single global lock, we need
to carefully manage the numbers on a per-node basis. to carefully manage the numbers on a per-node basis.
Recall from the answer to a previous Quick Quiz that the consequences Recall from the answer to a previous Quick Quiz that the consequences
...@@ -1091,8 +1099,8 @@ CPU has not yet passed through a quiescent state, ...@@ -1091,8 +1099,8 @@ CPU has not yet passed through a quiescent state,
while the <tt>-&gt;core_needs_qs</tt> flag indicates that the while the <tt>-&gt;core_needs_qs</tt> flag indicates that the
RCU core needs a quiescent state from the corresponding CPU. RCU core needs a quiescent state from the corresponding CPU.
The <tt>-&gt;gpwrap</tt> field indicates that the corresponding The <tt>-&gt;gpwrap</tt> field indicates that the corresponding
CPU has remained idle for so long that the <tt>completed</tt> CPU has remained idle for so long that the
and <tt>gpnum</tt> counters are in danger of overflow, which <tt>gp_seq</tt> counter is in danger of overflow, which
will cause the CPU to disregard the values of its counters on will cause the CPU to disregard the values of its counters on
its next exit from idle. its next exit from idle.
Finally, the <tt>rcu_qs_ctr_snap</tt> field is used to detect Finally, the <tt>rcu_qs_ctr_snap</tt> field is used to detect
...@@ -1130,10 +1138,10 @@ The CPU advances the callbacks in its <tt>rcu_data</tt> structure ...@@ -1130,10 +1138,10 @@ The CPU advances the callbacks in its <tt>rcu_data</tt> structure
whenever it notices that another RCU grace period has completed. whenever it notices that another RCU grace period has completed.
The CPU detects the completion of an RCU grace period by noticing The CPU detects the completion of an RCU grace period by noticing
that the value of its <tt>rcu_data</tt> structure's that the value of its <tt>rcu_data</tt> structure's
<tt>-&gt;completed</tt> field differs from that of its leaf <tt>-&gt;gp_seq</tt> field differs from that of its leaf
<tt>rcu_node</tt> structure. <tt>rcu_node</tt> structure.
Recall that each <tt>rcu_node</tt> structure's Recall that each <tt>rcu_node</tt> structure's
<tt>-&gt;completed</tt> field is updated at the end of each <tt>-&gt;gp_seq</tt> field is updated at the beginnings and ends of each
grace period. grace period.
<p> <p>
......
...@@ -357,7 +357,7 @@ parts, starting in this section with the various phases of ...@@ -357,7 +357,7 @@ parts, starting in this section with the various phases of
grace-period initialization. grace-period initialization.
<p>The first ordering-related grace-period initialization action is to <p>The first ordering-related grace-period initialization action is to
increment the <tt>rcu_state</tt> structure's <tt>-&gt;gpnum</tt> advance the <tt>rcu_state</tt> structure's <tt>-&gt;gp_seq</tt>
grace-period-number counter, as shown below: grace-period-number counter, as shown below:
</p><p><img src="TreeRCU-gp-init-1.svg" alt="TreeRCU-gp-init-1.svg" width="75%"> </p><p><img src="TreeRCU-gp-init-1.svg" alt="TreeRCU-gp-init-1.svg" width="75%">
...@@ -388,7 +388,7 @@ its last CPU and if the next <tt>rcu_node</tt> structure has no online CPUs). ...@@ -388,7 +388,7 @@ its last CPU and if the next <tt>rcu_node</tt> structure has no online CPUs).
<p>The final <tt>rcu_gp_init()</tt> pass through the <tt>rcu_node</tt> <p>The final <tt>rcu_gp_init()</tt> pass through the <tt>rcu_node</tt>
tree traverses breadth-first, setting each <tt>rcu_node</tt> structure's tree traverses breadth-first, setting each <tt>rcu_node</tt> structure's
<tt>-&gt;gpnum</tt> field to the newly incremented value from the <tt>-&gt;gp_seq</tt> field to the newly advanced value from the
<tt>rcu_state</tt> structure, as shown in the following diagram. <tt>rcu_state</tt> structure, as shown in the following diagram.
</p><p><img src="TreeRCU-gp-init-3.svg" alt="TreeRCU-gp-init-1.svg" width="75%"> </p><p><img src="TreeRCU-gp-init-3.svg" alt="TreeRCU-gp-init-1.svg" width="75%">
...@@ -398,9 +398,9 @@ tree traverses breadth-first, setting each <tt>rcu_node</tt> structure's ...@@ -398,9 +398,9 @@ tree traverses breadth-first, setting each <tt>rcu_node</tt> structure's
to notice that a new grace period has started, as described in the next to notice that a new grace period has started, as described in the next
section. section.
But because the grace-period kthread started the grace period at the But because the grace-period kthread started the grace period at the
root (with the increment of the <tt>rcu_state</tt> structure's root (with the advancing of the <tt>rcu_state</tt> structure's
<tt>-&gt;gpnum</tt> field) before setting each leaf <tt>rcu_node</tt> <tt>-&gt;gp_seq</tt> field) before setting each leaf <tt>rcu_node</tt>
structure's <tt>-&gt;gpnum</tt> field, each CPU's observation of structure's <tt>-&gt;gp_seq</tt> field, each CPU's observation of
the start of the grace period will happen after the actual start the start of the grace period will happen after the actual start
of the grace period. of the grace period.
...@@ -466,7 +466,7 @@ section that the grace period must wait on. ...@@ -466,7 +466,7 @@ section that the grace period must wait on.
<tr><td> <tr><td>
But a RCU read-side critical section might have started But a RCU read-side critical section might have started
after the beginning of the grace period after the beginning of the grace period
(the <tt>-&gt;gpnum++</tt> from earlier), so why should (the advancing of <tt>-&gt;gp_seq</tt> from earlier), so why should
the grace period wait on such a critical section? the grace period wait on such a critical section?
</td></tr> </td></tr>
<tr><th align="left">Answer:</th></tr> <tr><th align="left">Answer:</th></tr>
...@@ -609,10 +609,8 @@ states outstanding from other CPUs. ...@@ -609,10 +609,8 @@ states outstanding from other CPUs.
<h4><a name="Grace-Period Cleanup">Grace-Period Cleanup</a></h4> <h4><a name="Grace-Period Cleanup">Grace-Period Cleanup</a></h4>
<p>Grace-period cleanup first scans the <tt>rcu_node</tt> tree <p>Grace-period cleanup first scans the <tt>rcu_node</tt> tree
breadth-first setting all the <tt>-&gt;completed</tt> fields equal breadth-first advancing all the <tt>-&gt;gp_seq</tt> fields, then it
to the number of the newly completed grace period, then it sets advances the <tt>rcu_state</tt> structure's <tt>-&gt;gp_seq</tt> field.
the <tt>rcu_state</tt> structure's <tt>-&gt;completed</tt> field,
again to the number of the newly completed grace period.
The ordering effects are shown below: The ordering effects are shown below:
</p><p><img src="TreeRCU-gp-cleanup.svg" alt="TreeRCU-gp-cleanup.svg" width="75%"> </p><p><img src="TreeRCU-gp-cleanup.svg" alt="TreeRCU-gp-cleanup.svg" width="75%">
...@@ -634,7 +632,7 @@ grace-period cleanup is complete, the next grace period can begin. ...@@ -634,7 +632,7 @@ grace-period cleanup is complete, the next grace period can begin.
CPU has reported its quiescent state, but it may be some CPU has reported its quiescent state, but it may be some
milliseconds before RCU becomes aware of this. milliseconds before RCU becomes aware of this.
The latest reasonable candidate is once the <tt>rcu_state</tt> The latest reasonable candidate is once the <tt>rcu_state</tt>
structure's <tt>-&gt;completed</tt> field has been updated, structure's <tt>-&gt;gp_seq</tt> field has been updated,
but it is quite possible that some CPUs have already completed but it is quite possible that some CPUs have already completed
phase two of their updates by that time. phase two of their updates by that time.
In short, if you are going to work with RCU, you need to In short, if you are going to work with RCU, you need to
...@@ -647,7 +645,7 @@ grace-period cleanup is complete, the next grace period can begin. ...@@ -647,7 +645,7 @@ grace-period cleanup is complete, the next grace period can begin.
<h4><a name="Callback Invocation">Callback Invocation</a></h4> <h4><a name="Callback Invocation">Callback Invocation</a></h4>
<p>Once a given CPU's leaf <tt>rcu_node</tt> structure's <p>Once a given CPU's leaf <tt>rcu_node</tt> structure's
<tt>-&gt;completed</tt> field has been updated, that CPU can begin <tt>-&gt;gp_seq</tt> field has been updated, that CPU can begin
invoking its RCU callbacks that were waiting for this grace period invoking its RCU callbacks that were waiting for this grace period
to end. to end.
These callbacks are identified by <tt>rcu_advance_cbs()</tt>, These callbacks are identified by <tt>rcu_advance_cbs()</tt>,
......
...@@ -384,11 +384,11 @@ ...@@ -384,11 +384,11 @@
inkscape:window-height="1144" inkscape:window-height="1144"
id="namedview208" id="namedview208"
showgrid="true" showgrid="true"
inkscape:zoom="0.70710678" inkscape:zoom="0.78716603"
inkscape:cx="617.89017" inkscape:cx="513.06403"
inkscape:cy="542.52419" inkscape:cy="623.1214"
inkscape:window-x="86" inkscape:window-x="102"
inkscape:window-y="28" inkscape:window-y="38"
inkscape:window-maximized="0" inkscape:window-maximized="0"
inkscape:current-layer="g3188-3" inkscape:current-layer="g3188-3"
fit-margin-top="5" fit-margin-top="5"
...@@ -417,13 +417,15 @@ ...@@ -417,13 +417,15 @@
id="g3188"> id="g3188">
<text <text
xml:space="preserve" xml:space="preserve"
x="3199.1516" x="3145.9592"
y="13255.592" y="13255.592"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202" id="text202"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;font-family:Courier">-&gt;completed = -&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;font-family:Courier"><tspan
style="font-size:172.87567139px"
id="tspan3143">rcu_seq_end(&amp;rnp-&gt;gp_seq)</tspan></text>
<g <g
id="g3107" id="g3107"
transform="translate(947.90548,11584.029)"> transform="translate(947.90548,11584.029)">
...@@ -502,13 +504,15 @@ ...@@ -502,13 +504,15 @@
</g> </g>
<text <text
xml:space="preserve" xml:space="preserve"
x="5324.5371" x="5264.4731"
y="15414.598" y="15428.84"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-753" id="text202-36-7"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;completed = -&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier"><tspan
style="font-size:172.87567139px"
id="tspan3166-5">rcu_seq_end(&amp;rnp-&gt;gp_seq)</tspan></text>
</g> </g>
<g <g
style="fill:none;stroke-width:0.025in" style="fill:none;stroke-width:0.025in"
...@@ -547,15 +551,6 @@ ...@@ -547,15 +551,6 @@
sodipodi:linespacing="125%"><tspan sodipodi:linespacing="125%"><tspan
style="font-size:159.57754517px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Liberation Sans;-inkscape-font-specification:Liberation Sans" style="font-size:159.57754517px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Liberation Sans;-inkscape-font-specification:Liberation Sans"
id="tspan3104-6-5-6-0">Leaf</tspan></text> id="tspan3104-6-5-6-0">Leaf</tspan></text>
<text
xml:space="preserve"
x="7479.5796"
y="17699.943"
font-style="normal"
font-weight="bold"
font-size="192"
id="text202-9"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;completed = -&gt;gpnum</text>
<path <path
sodipodi:nodetypes="cc" sodipodi:nodetypes="cc"
inkscape:connector-curvature="0" inkscape:connector-curvature="0"
...@@ -566,15 +561,6 @@ ...@@ -566,15 +561,6 @@
style="fill:none;stroke-width:0.025in" style="fill:none;stroke-width:0.025in"
transform="translate(-737.93887,7732.6672)" transform="translate(-737.93887,7732.6672)"
id="g3188-3"> id="g3188-3">
<text
xml:space="preserve"
x="3225.7478"
y="13175.802"
font-style="normal"
font-weight="bold"
font-size="192"
id="text202-60"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;font-family:Courier">rsp-&gt;completed =</text>
<g <g
id="g3107-62" id="g3107-62"
transform="translate(947.90548,11584.029)"> transform="translate(947.90548,11584.029)">
...@@ -607,15 +593,6 @@ ...@@ -607,15 +593,6 @@
sodipodi:linespacing="125%"><tspan sodipodi:linespacing="125%"><tspan
style="font-size:159.57754517px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Liberation Sans;-inkscape-font-specification:Liberation Sans" style="font-size:159.57754517px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Liberation Sans;-inkscape-font-specification:Liberation Sans"
id="tspan3104-6-5-7">Root</tspan></text> id="tspan3104-6-5-7">Root</tspan></text>
<text
xml:space="preserve"
x="3225.7478"
y="13390.038"
font-style="normal"
font-weight="bold"
font-size="192"
id="text202-60-3"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier"> rnp-&gt;completed</text>
<flowRoot <flowRoot
xml:space="preserve" xml:space="preserve"
id="flowRoot3356" id="flowRoot3356"
...@@ -627,7 +604,18 @@ ...@@ -627,7 +604,18 @@
height="63.63961" height="63.63961"
x="332.34018" x="332.34018"
y="681.87292" /></flowRegion><flowPara y="681.87292" /></flowRegion><flowPara
id="flowPara3362" /></flowRoot> </g> id="flowPara3362" /></flowRoot> <text
xml:space="preserve"
x="3156.6121"
y="13317.754"
font-style="normal"
font-weight="bold"
font-size="192"
id="text202-36-6"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier"><tspan
style="font-size:172.87567139px"
id="tspan3166-0">rcu_seq_end(&amp;rsp-&gt;gp_seq)</tspan></text>
</g>
<g <g
style="fill:none;stroke-width:0.025in" style="fill:none;stroke-width:0.025in"
transform="translate(-858.40227,7769.0342)" transform="translate(-858.40227,7769.0342)"
...@@ -859,6 +847,17 @@ ...@@ -859,6 +847,17 @@
id="path3414-8-3-6-6" id="path3414-8-3-6-6"
inkscape:connector-curvature="0" inkscape:connector-curvature="0"
sodipodi:nodetypes="cc" /> sodipodi:nodetypes="cc" />
<text
xml:space="preserve"
x="7418.769"
y="17646.104"
font-style="normal"
font-weight="bold"
font-size="192"
id="text202-36-70"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier"><tspan
style="font-size:172.87567139px"
id="tspan3166-93">rcu_seq_end(&amp;rnp-&gt;gp_seq)</tspan></text>
</g> </g>
<g <g
transform="translate(-1642.5377,-11611.245)" transform="translate(-1642.5377,-11611.245)"
...@@ -887,13 +886,15 @@ ...@@ -887,13 +886,15 @@
</g> </g>
<text <text
xml:space="preserve" xml:space="preserve"
x="5327.3057" x="5274.1133"
y="15428.84" y="15428.84"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-36" id="text202-36"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;completed = -&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier"><tspan
style="font-size:172.87567139px"
id="tspan3166">rcu_seq_end(&amp;rnp-&gt;gp_seq)</tspan></text>
</g> </g>
<g <g
transform="translate(-151.71746,-11647.612)" transform="translate(-151.71746,-11647.612)"
...@@ -972,13 +973,15 @@ ...@@ -972,13 +973,15 @@
id="tspan3104-6-5-6-0-92">Leaf</tspan></text> id="tspan3104-6-5-6-0-92">Leaf</tspan></text>
<text <text
xml:space="preserve" xml:space="preserve"
x="7486.4907" x="7408.5918"
y="17670.119" y="17619.504"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-6" id="text202-36-2"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;completed = -&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier"><tspan
style="font-size:172.87567139px"
id="tspan3166-9">rcu_seq_end(&amp;rnp-&gt;gp_seq)</tspan></text>
</g> </g>
<g <g
transform="translate(-6817.1997,-11647.612)" transform="translate(-6817.1997,-11647.612)"
...@@ -1019,13 +1022,15 @@ ...@@ -1019,13 +1022,15 @@
id="tspan3104-6-5-6-0-1">Leaf</tspan></text> id="tspan3104-6-5-6-0-1">Leaf</tspan></text>
<text <text
xml:space="preserve" xml:space="preserve"
x="7474.1382" x="7416.8003"
y="17688.926" y="17619.504"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-5" id="text202-36-3"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;completed = -&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier"><tspan
style="font-size:172.87567139px"
id="tspan3166-56">rcu_seq_end(&amp;rnp-&gt;gp_seq)</tspan></text>
</g> </g>
<path <path
style="fill:none;stroke:#000000;stroke-width:13.29812908px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Lend)" style="fill:none;stroke:#000000;stroke-width:13.29812908px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Lend)"
...@@ -1059,15 +1064,6 @@ ...@@ -1059,15 +1064,6 @@
id="path3414-8-3-6" id="path3414-8-3-6"
inkscape:connector-curvature="0" inkscape:connector-curvature="0"
sodipodi:nodetypes="cc" /> sodipodi:nodetypes="cc" />
<text
xml:space="preserve"
x="7318.9653"
y="6031.6353"
font-style="normal"
font-weight="bold"
font-size="192"
id="text202-2"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;completed = -&gt;gpnum</text>
<g <g
style="fill:none;stroke-width:0.025in" style="fill:none;stroke-width:0.025in"
id="g4504-3-9" id="g4504-3-9"
...@@ -1123,4 +1119,15 @@ ...@@ -1123,4 +1119,15 @@
id="path3134-9-0-3-5" id="path3134-9-0-3-5"
d="m 6875.6003,15833.906 1595.7755,0" d="m 6875.6003,15833.906 1595.7755,0"
style="fill:none;stroke:#969696;stroke-width:53.19251633;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow1Send-36)" /> style="fill:none;stroke:#969696;stroke-width:53.19251633;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow1Send-36)" />
<text
xml:space="preserve"
x="7275.2612"
y="5971.8916"
font-style="normal"
font-weight="bold"
font-size="192"
id="text202-36-1"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier"><tspan
style="font-size:172.87567139px"
id="tspan3166-2">rcu_seq_end(&amp;rnp-&gt;gp_seq)</tspan></text>
</svg> </svg>
...@@ -272,13 +272,13 @@ ...@@ -272,13 +272,13 @@
inkscape:window-height="1144" inkscape:window-height="1144"
id="namedview208" id="namedview208"
showgrid="true" showgrid="true"
inkscape:zoom="0.70710678" inkscape:zoom="2.6330492"
inkscape:cx="617.89019" inkscape:cx="524.82797"
inkscape:cy="636.57143" inkscape:cy="519.31194"
inkscape:window-x="697" inkscape:window-x="79"
inkscape:window-y="28" inkscape:window-y="28"
inkscape:window-maximized="0" inkscape:window-maximized="0"
inkscape:current-layer="svg2" inkscape:current-layer="g3188"
fit-margin-top="5" fit-margin-top="5"
fit-margin-right="5" fit-margin-right="5"
fit-margin-left="5" fit-margin-left="5"
...@@ -305,13 +305,15 @@ ...@@ -305,13 +305,15 @@
id="g3188"> id="g3188">
<text <text
xml:space="preserve" xml:space="preserve"
x="3305.5364" x="3119.363"
y="13255.592" y="13255.592"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202" id="text202"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;font-family:Courier">rsp-&gt;gpnum++</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;font-family:Courier"><tspan
style="font-size:172.87567139px"
id="tspan3071">rcu_seq_start(rsp-&gt;gp_seq)</tspan></text>
<g <g
id="g3107" id="g3107"
transform="translate(947.90548,11584.029)"> transform="translate(947.90548,11584.029)">
......
...@@ -19,7 +19,7 @@ ...@@ -19,7 +19,7 @@
id="svg2" id="svg2"
version="1.1" version="1.1"
inkscape:version="0.48.4 r9939" inkscape:version="0.48.4 r9939"
sodipodi:docname="TreeRCU-gp-init-2.svg"> sodipodi:docname="TreeRCU-gp-init-3.svg">
<metadata <metadata
id="metadata212"> id="metadata212">
<rdf:RDF> <rdf:RDF>
...@@ -257,18 +257,22 @@ ...@@ -257,18 +257,22 @@
inkscape:window-width="1087" inkscape:window-width="1087"
inkscape:window-height="1144" inkscape:window-height="1144"
id="namedview208" id="namedview208"
showgrid="false" showgrid="true"
inkscape:zoom="0.70710678" inkscape:zoom="0.68224756"
inkscape:cx="617.89019" inkscape:cx="617.89019"
inkscape:cy="625.84293" inkscape:cy="625.84293"
inkscape:window-x="697" inkscape:window-x="54"
inkscape:window-y="28" inkscape:window-y="28"
inkscape:window-maximized="0" inkscape:window-maximized="0"
inkscape:current-layer="svg2" inkscape:current-layer="g3153"
fit-margin-top="5" fit-margin-top="5"
fit-margin-right="5" fit-margin-right="5"
fit-margin-left="5" fit-margin-left="5"
fit-margin-bottom="5" /> fit-margin-bottom="5">
<inkscape:grid
type="xygrid"
id="grid3090" />
</sodipodi:namedview>
<path <path
sodipodi:nodetypes="cccccccccccccccccccccccc" sodipodi:nodetypes="cccccccccccccccccccccccc"
inkscape:connector-curvature="0" inkscape:connector-curvature="0"
...@@ -281,13 +285,13 @@ ...@@ -281,13 +285,13 @@
id="g3188"> id="g3188">
<text <text
xml:space="preserve" xml:space="preserve"
x="3305.5364" x="3145.9592"
y="13255.592" y="13255.592"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202" id="text202"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;font-family:Courier">-&gt;gpnum = rsp-&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;font-family:Courier">-&gt;gp_seq = rsp-&gt;gp_seq</text>
<g <g
id="g3107" id="g3107"
transform="translate(947.90548,11584.029)"> transform="translate(947.90548,11584.029)">
...@@ -366,13 +370,13 @@ ...@@ -366,13 +370,13 @@
</g> </g>
<text <text
xml:space="preserve" xml:space="preserve"
x="5392.3345" x="5253.6904"
y="15407.104" y="15407.032"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-6" id="text202-6"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gpnum = rsp-&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gp_seq = rsp-&gt;gp_seq</text>
</g> </g>
<g <g
style="fill:none;stroke-width:0.025in" style="fill:none;stroke-width:0.025in"
...@@ -413,13 +417,13 @@ ...@@ -413,13 +417,13 @@
id="tspan3104-6-5-6-0">Leaf</tspan></text> id="tspan3104-6-5-6-0">Leaf</tspan></text>
<text <text
xml:space="preserve" xml:space="preserve"
x="7536.4883" x="7415.4365"
y="17640.934" y="17670.572"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-9" id="text202-9"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gpnum = rsp-&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gp_seq = rsp-&gt;gp_seq</text>
</g> </g>
<g <g
transform="translate(-1642.5375,-11610.962)" transform="translate(-1642.5375,-11610.962)"
...@@ -448,13 +452,13 @@ ...@@ -448,13 +452,13 @@
</g> </g>
<text <text
xml:space="preserve" xml:space="preserve"
x="5378.4146" x="5258.0688"
y="15436.927" y="15412.313"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-3" id="text202-3"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gpnum = rsp-&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gp_seq = rsp-&gt;gp_seq</text>
</g> </g>
<g <g
transform="translate(-151.71726,-11647.329)" transform="translate(-151.71726,-11647.329)"
...@@ -533,13 +537,13 @@ ...@@ -533,13 +537,13 @@
id="tspan3104-6-5-6-0-92">Leaf</tspan></text> id="tspan3104-6-5-6-0-92">Leaf</tspan></text>
<text <text
xml:space="preserve" xml:space="preserve"
x="7520.1294" x="7405.2607"
y="17673.639" y="17670.572"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-35" id="text202-35"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gpnum = rsp-&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gp_seq = rsp-&gt;gp_seq</text>
</g> </g>
<g <g
transform="translate(-6817.1998,-11647.329)" transform="translate(-6817.1998,-11647.329)"
...@@ -580,13 +584,13 @@ ...@@ -580,13 +584,13 @@
id="tspan3104-6-5-6-0-1">Leaf</tspan></text> id="tspan3104-6-5-6-0-1">Leaf</tspan></text>
<text <text
xml:space="preserve" xml:space="preserve"
x="7521.4663" x="7413.4688"
y="17666.062" y="17670.566"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-75" id="text202-75"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gpnum = rsp-&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gp_seq = rsp-&gt;gp_seq</text>
</g> </g>
<path <path
style="fill:none;stroke:#000000;stroke-width:13.29812908px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Lend)" style="fill:none;stroke:#000000;stroke-width:13.29812908px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Lend)"
...@@ -622,11 +626,11 @@ ...@@ -622,11 +626,11 @@
sodipodi:nodetypes="cc" /> sodipodi:nodetypes="cc" />
<text <text
xml:space="preserve" xml:space="preserve"
x="7370.856" x="7271.9297"
y="5997.5972" y="6023.2412"
font-style="normal" font-style="normal"
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-62" id="text202-62"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gpnum = rsp-&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">-&gt;gp_seq = rsp-&gt;gp_seq</text>
</svg> </svg>
...@@ -300,13 +300,13 @@ ...@@ -300,13 +300,13 @@
inkscape:window-height="1144" inkscape:window-height="1144"
id="namedview208" id="namedview208"
showgrid="true" showgrid="true"
inkscape:zoom="0.70710678" inkscape:zoom="0.96484375"
inkscape:cx="616.47598" inkscape:cx="507.0191"
inkscape:cy="595.41964" inkscape:cy="885.62207"
inkscape:window-x="813" inkscape:window-x="47"
inkscape:window-y="28" inkscape:window-y="28"
inkscape:window-maximized="0" inkscape:window-maximized="0"
inkscape:current-layer="g4405" inkscape:current-layer="g3115"
fit-margin-top="5" fit-margin-top="5"
fit-margin-right="5" fit-margin-right="5"
fit-margin-left="5" fit-margin-left="5"
...@@ -710,7 +710,7 @@ ...@@ -710,7 +710,7 @@
font-weight="bold" font-weight="bold"
font-size="192" font-size="192"
id="text202-6-6" id="text202-6-6"
style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">rdp-&gt;gpnum</text> style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">rdp-&gt;gp_seq</text>
<text <text
xml:space="preserve" xml:space="preserve"
x="5035.4155" x="5035.4155"
......
...@@ -172,7 +172,7 @@ it will print a message similar to the following: ...@@ -172,7 +172,7 @@ it will print a message similar to the following:
INFO: rcu_sched detected stalls on CPUs/tasks: INFO: rcu_sched detected stalls on CPUs/tasks:
2-...: (3 GPs behind) idle=06c/0/0 softirq=1453/1455 fqs=0 2-...: (3 GPs behind) idle=06c/0/0 softirq=1453/1455 fqs=0
16-...: (0 ticks this GP) idle=81c/0/0 softirq=764/764 fqs=0 16-...: (0 ticks this GP) idle=81c/0/0 softirq=764/764 fqs=0
(detected by 32, t=2603 jiffies, g=7073, c=7072, q=625) (detected by 32, t=2603 jiffies, g=7075, q=625)
This message indicates that CPU 32 detected that CPUs 2 and 16 were both This message indicates that CPU 32 detected that CPUs 2 and 16 were both
causing stalls, and that the stall was affecting RCU-sched. This message causing stalls, and that the stall was affecting RCU-sched. This message
...@@ -215,11 +215,10 @@ CPU since the last time that this CPU noted the beginning of a grace ...@@ -215,11 +215,10 @@ CPU since the last time that this CPU noted the beginning of a grace
period. period.
The "detected by" line indicates which CPU detected the stall (in this The "detected by" line indicates which CPU detected the stall (in this
case, CPU 32), how many jiffies have elapsed since the start of the case, CPU 32), how many jiffies have elapsed since the start of the grace
grace period (in this case 2603), the number of the last grace period period (in this case 2603), the grace-period sequence number (7075), and
to start and to complete (7073 and 7072, respectively), and an estimate an estimate of the total number of RCU callbacks queued across all CPUs
of the total number of RCU callbacks queued across all CPUs (625 in (625 in this case).
this case).
In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed
for each CPU: for each CPU:
...@@ -266,15 +265,16 @@ If the relevant grace-period kthread has been unable to run prior to ...@@ -266,15 +265,16 @@ If the relevant grace-period kthread has been unable to run prior to
the stall warning, as was the case in the "All QSes seen" line above, the stall warning, as was the case in the "All QSes seen" line above,
the following additional line is printed: the following additional line is printed:
kthread starved for 23807 jiffies! g7073 c7072 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 kthread starved for 23807 jiffies! g7075 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 ->cpu=5
Starving the grace-period kthreads of CPU time can of course result Starving the grace-period kthreads of CPU time can of course result
in RCU CPU stall warnings even when all CPUs and tasks have passed in RCU CPU stall warnings even when all CPUs and tasks have passed
through the required quiescent states. The "g" and "c" numbers flag the through the required quiescent states. The "g" number shows the current
number of the last grace period started and completed, respectively, grace-period sequence number, the "f" precedes the ->gp_flags command
the "f" precedes the ->gp_flags command to the grace-period kthread, to the grace-period kthread, the "RCU_GP_WAIT_FQS" indicates that the
the "RCU_GP_WAIT_FQS" indicates that the kthread is waiting for a short kthread is waiting for a short timeout, the "state" precedes value of the
timeout, and the "state" precedes value of the task_struct ->state field. task_struct ->state field, and the "cpu" indicates that the grace-period
kthread last ran on CPU 5.
Multiple Warnings From One Stall Multiple Warnings From One Stall
......
...@@ -588,6 +588,7 @@ It is extremely simple: ...@@ -588,6 +588,7 @@ It is extremely simple:
void synchronize_rcu(void) void synchronize_rcu(void)
{ {
write_lock(&rcu_gp_mutex); write_lock(&rcu_gp_mutex);
smp_mb__after_spinlock();
write_unlock(&rcu_gp_mutex); write_unlock(&rcu_gp_mutex);
} }
...@@ -609,12 +610,15 @@ don't forget about them when submitting patches making use of RCU!] ...@@ -609,12 +610,15 @@ don't forget about them when submitting patches making use of RCU!]
The rcu_read_lock() and rcu_read_unlock() primitive read-acquire The rcu_read_lock() and rcu_read_unlock() primitive read-acquire
and release a global reader-writer lock. The synchronize_rcu() and release a global reader-writer lock. The synchronize_rcu()
primitive write-acquires this same lock, then immediately releases primitive write-acquires this same lock, then releases it. This means
it. This means that once synchronize_rcu() exits, all RCU read-side that once synchronize_rcu() exits, all RCU read-side critical sections
critical sections that were in progress before synchronize_rcu() was that were in progress before synchronize_rcu() was called are guaranteed
called are guaranteed to have completed -- there is no way that to have completed -- there is no way that synchronize_rcu() would have
synchronize_rcu() would have been able to write-acquire the lock been able to write-acquire the lock otherwise. The smp_mb__after_spinlock()
otherwise. promotes synchronize_rcu() to a full memory barrier in compliance with
the "Memory-Barrier Guarantees" listed in:
Documentation/RCU/Design/Requirements/Requirements.html.
It is possible to nest rcu_read_lock(), since reader-writer locks may It is possible to nest rcu_read_lock(), since reader-writer locks may
be recursively acquired. Note also that rcu_read_lock() is immune be recursively acquired. Note also that rcu_read_lock() is immune
...@@ -816,11 +820,13 @@ RCU list traversal: ...@@ -816,11 +820,13 @@ RCU list traversal:
list_next_rcu list_next_rcu
list_for_each_entry_rcu list_for_each_entry_rcu
list_for_each_entry_continue_rcu list_for_each_entry_continue_rcu
list_for_each_entry_from_rcu
hlist_first_rcu hlist_first_rcu
hlist_next_rcu hlist_next_rcu
hlist_pprev_rcu hlist_pprev_rcu
hlist_for_each_entry_rcu hlist_for_each_entry_rcu
hlist_for_each_entry_rcu_bh hlist_for_each_entry_rcu_bh
hlist_for_each_entry_from_rcu
hlist_for_each_entry_continue_rcu hlist_for_each_entry_continue_rcu
hlist_for_each_entry_continue_rcu_bh hlist_for_each_entry_continue_rcu_bh
hlist_nulls_first_rcu hlist_nulls_first_rcu
......
...@@ -3632,8 +3632,8 @@ ...@@ -3632,8 +3632,8 @@
Set time (s) after boot for CPU-hotplug testing. Set time (s) after boot for CPU-hotplug testing.
rcutorture.onoff_interval= [KNL] rcutorture.onoff_interval= [KNL]
Set time (s) between CPU-hotplug operations, or Set time (jiffies) between CPU-hotplug operations,
zero to disable CPU-hotplug testing. or zero to disable CPU-hotplug testing.
rcutorture.shuffle_interval= [KNL] rcutorture.shuffle_interval= [KNL]
Set task-shuffle interval (s). Shuffling tasks Set task-shuffle interval (s). Shuffling tasks
......
...@@ -12039,9 +12039,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git ...@@ -12039,9 +12039,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
F: Documentation/RCU/ F: Documentation/RCU/
X: Documentation/RCU/torture.txt X: Documentation/RCU/torture.txt
F: include/linux/rcu* F: include/linux/rcu*
X: include/linux/srcu.h X: include/linux/srcu*.h
F: kernel/rcu/ F: kernel/rcu/
X: kernel/torture.c X: kernel/rcu/srcu*.c
REAL TIME CLOCK (RTC) SUBSYSTEM REAL TIME CLOCK (RTC) SUBSYSTEM
M: Alessandro Zummo <a.zummo@towertech.it> M: Alessandro Zummo <a.zummo@towertech.it>
...@@ -13078,8 +13078,8 @@ L: linux-kernel@vger.kernel.org ...@@ -13078,8 +13078,8 @@ L: linux-kernel@vger.kernel.org
W: http://www.rdrop.com/users/paulmck/RCU/ W: http://www.rdrop.com/users/paulmck/RCU/
S: Supported S: Supported
T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
F: include/linux/srcu.h F: include/linux/srcu*.h
F: kernel/rcu/srcu.c F: kernel/rcu/srcu*.c
SERIAL LOW-POWER INTER-CHIP MEDIA BUS (SLIMbus) SERIAL LOW-POWER INTER-CHIP MEDIA BUS (SLIMbus)
M: Srinivas Kandagatla <srinivas.kandagatla@linaro.org> M: Srinivas Kandagatla <srinivas.kandagatla@linaro.org>
...@@ -14438,6 +14438,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git ...@@ -14438,6 +14438,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
F: Documentation/RCU/torture.txt F: Documentation/RCU/torture.txt
F: kernel/torture.c F: kernel/torture.c
F: kernel/rcu/rcutorture.c F: kernel/rcu/rcutorture.c
F: kernel/rcu/rcuperf.c
F: kernel/locking/locktorture.c F: kernel/locking/locktorture.c
TOSHIBA ACPI EXTRAS DRIVER TOSHIBA ACPI EXTRAS DRIVER
......
...@@ -396,7 +396,16 @@ static inline void list_splice_tail_init_rcu(struct list_head *list, ...@@ -396,7 +396,16 @@ static inline void list_splice_tail_init_rcu(struct list_head *list,
* @member: the name of the list_head within the struct. * @member: the name of the list_head within the struct.
* *
* Continue to iterate over list of given type, continuing after * Continue to iterate over list of given type, continuing after
* the current position. * the current position which must have been in the list when the RCU read
* lock was taken.
* This would typically require either that you obtained the node from a
* previous walk of the list in the same RCU read-side critical section, or
* that you held some sort of non-RCU reference (such as a reference count)
* to keep the node alive *and* in the list.
*
* This iterator is similar to list_for_each_entry_from_rcu() except
* this starts after the given position and that one starts at the given
* position.
*/ */
#define list_for_each_entry_continue_rcu(pos, head, member) \ #define list_for_each_entry_continue_rcu(pos, head, member) \
for (pos = list_entry_rcu(pos->member.next, typeof(*pos), member); \ for (pos = list_entry_rcu(pos->member.next, typeof(*pos), member); \
...@@ -411,6 +420,14 @@ static inline void list_splice_tail_init_rcu(struct list_head *list, ...@@ -411,6 +420,14 @@ static inline void list_splice_tail_init_rcu(struct list_head *list,
* *
* Iterate over the tail of a list starting from a given position, * Iterate over the tail of a list starting from a given position,
* which must have been in the list when the RCU read lock was taken. * which must have been in the list when the RCU read lock was taken.
* This would typically require either that you obtained the node from a
* previous walk of the list in the same RCU read-side critical section, or
* that you held some sort of non-RCU reference (such as a reference count)
* to keep the node alive *and* in the list.
*
* This iterator is similar to list_for_each_entry_continue_rcu() except
* this starts from the given position and that one starts from the position
* after the given position.
*/ */
#define list_for_each_entry_from_rcu(pos, head, member) \ #define list_for_each_entry_from_rcu(pos, head, member) \
for (; &(pos)->member != (head); \ for (; &(pos)->member != (head); \
......
...@@ -64,7 +64,6 @@ void rcu_barrier_tasks(void); ...@@ -64,7 +64,6 @@ void rcu_barrier_tasks(void);
void __rcu_read_lock(void); void __rcu_read_lock(void);
void __rcu_read_unlock(void); void __rcu_read_unlock(void);
void rcu_read_unlock_special(struct task_struct *t);
void synchronize_rcu(void); void synchronize_rcu(void);
/* /*
...@@ -159,11 +158,11 @@ static inline void rcu_init_nohz(void) { } ...@@ -159,11 +158,11 @@ static inline void rcu_init_nohz(void) { }
} while (0) } while (0)
/* /*
* Note a voluntary context switch for RCU-tasks benefit. This is a * Note a quasi-voluntary context switch for RCU-tasks's benefit.
* macro rather than an inline function to avoid #include hell. * This is a macro rather than an inline function to avoid #include hell.
*/ */
#ifdef CONFIG_TASKS_RCU #ifdef CONFIG_TASKS_RCU
#define rcu_note_voluntary_context_switch_lite(t) \ #define rcu_tasks_qs(t) \
do { \ do { \
if (READ_ONCE((t)->rcu_tasks_holdout)) \ if (READ_ONCE((t)->rcu_tasks_holdout)) \
WRITE_ONCE((t)->rcu_tasks_holdout, false); \ WRITE_ONCE((t)->rcu_tasks_holdout, false); \
...@@ -171,14 +170,14 @@ static inline void rcu_init_nohz(void) { } ...@@ -171,14 +170,14 @@ static inline void rcu_init_nohz(void) { }
#define rcu_note_voluntary_context_switch(t) \ #define rcu_note_voluntary_context_switch(t) \
do { \ do { \
rcu_all_qs(); \ rcu_all_qs(); \
rcu_note_voluntary_context_switch_lite(t); \ rcu_tasks_qs(t); \
} while (0) } while (0)
void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func); void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func);
void synchronize_rcu_tasks(void); void synchronize_rcu_tasks(void);
void exit_tasks_rcu_start(void); void exit_tasks_rcu_start(void);
void exit_tasks_rcu_finish(void); void exit_tasks_rcu_finish(void);
#else /* #ifdef CONFIG_TASKS_RCU */ #else /* #ifdef CONFIG_TASKS_RCU */
#define rcu_note_voluntary_context_switch_lite(t) do { } while (0) #define rcu_tasks_qs(t) do { } while (0)
#define rcu_note_voluntary_context_switch(t) rcu_all_qs() #define rcu_note_voluntary_context_switch(t) rcu_all_qs()
#define call_rcu_tasks call_rcu_sched #define call_rcu_tasks call_rcu_sched
#define synchronize_rcu_tasks synchronize_sched #define synchronize_rcu_tasks synchronize_sched
...@@ -195,8 +194,8 @@ static inline void exit_tasks_rcu_finish(void) { } ...@@ -195,8 +194,8 @@ static inline void exit_tasks_rcu_finish(void) { }
*/ */
#define cond_resched_tasks_rcu_qs() \ #define cond_resched_tasks_rcu_qs() \
do { \ do { \
if (!cond_resched()) \ rcu_tasks_qs(current); \
rcu_note_voluntary_context_switch_lite(current); \ cond_resched(); \
} while (0) } while (0)
/* /*
...@@ -567,8 +566,8 @@ static inline void rcu_preempt_sleep_check(void) { } ...@@ -567,8 +566,8 @@ static inline void rcu_preempt_sleep_check(void) { }
* This is simply an identity function, but it documents where a pointer * This is simply an identity function, but it documents where a pointer
* is handed off from RCU to some other synchronization mechanism, for * is handed off from RCU to some other synchronization mechanism, for
* example, reference counting or locking. In C11, it would map to * example, reference counting or locking. In C11, it would map to
* kill_dependency(). It could be used as follows: * kill_dependency(). It could be used as follows::
* `` *
* rcu_read_lock(); * rcu_read_lock();
* p = rcu_dereference(gp); * p = rcu_dereference(gp);
* long_lived = is_long_lived(p); * long_lived = is_long_lived(p);
...@@ -579,7 +578,6 @@ static inline void rcu_preempt_sleep_check(void) { } ...@@ -579,7 +578,6 @@ static inline void rcu_preempt_sleep_check(void) { }
* p = rcu_pointer_handoff(p); * p = rcu_pointer_handoff(p);
* } * }
* rcu_read_unlock(); * rcu_read_unlock();
*``
*/ */
#define rcu_pointer_handoff(p) (p) #define rcu_pointer_handoff(p) (p)
......
...@@ -93,7 +93,7 @@ static inline void kfree_call_rcu(struct rcu_head *head, ...@@ -93,7 +93,7 @@ static inline void kfree_call_rcu(struct rcu_head *head,
#define rcu_note_context_switch(preempt) \ #define rcu_note_context_switch(preempt) \
do { \ do { \
rcu_sched_qs(); \ rcu_sched_qs(); \
rcu_note_voluntary_context_switch_lite(current); \ rcu_tasks_qs(current); \
} while (0) } while (0)
static inline int rcu_needs_cpu(u64 basemono, u64 *nextevt) static inline int rcu_needs_cpu(u64 basemono, u64 *nextevt)
......
...@@ -195,6 +195,16 @@ static inline int srcu_read_lock(struct srcu_struct *sp) __acquires(sp) ...@@ -195,6 +195,16 @@ static inline int srcu_read_lock(struct srcu_struct *sp) __acquires(sp)
return retval; return retval;
} }
/* Used by tracing, cannot be traced and cannot invoke lockdep. */
static inline notrace int
srcu_read_lock_notrace(struct srcu_struct *sp) __acquires(sp)
{
int retval;
retval = __srcu_read_lock(sp);
return retval;
}
/** /**
* srcu_read_unlock - unregister a old reader from an SRCU-protected structure. * srcu_read_unlock - unregister a old reader from an SRCU-protected structure.
* @sp: srcu_struct in which to unregister the old reader. * @sp: srcu_struct in which to unregister the old reader.
...@@ -209,6 +219,13 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx) ...@@ -209,6 +219,13 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx)
__srcu_read_unlock(sp, idx); __srcu_read_unlock(sp, idx);
} }
/* Used by tracing, cannot be traced and cannot call lockdep. */
static inline notrace void
srcu_read_unlock_notrace(struct srcu_struct *sp, int idx) __releases(sp)
{
__srcu_read_unlock(sp, idx);
}
/** /**
* smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock * smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock
* *
......
...@@ -64,6 +64,8 @@ struct torture_random_state { ...@@ -64,6 +64,8 @@ struct torture_random_state {
long trs_count; long trs_count;
}; };
#define DEFINE_TORTURE_RANDOM(name) struct torture_random_state name = { 0, 0 } #define DEFINE_TORTURE_RANDOM(name) struct torture_random_state name = { 0, 0 }
#define DEFINE_TORTURE_RANDOM_PERCPU(name) \
DEFINE_PER_CPU(struct torture_random_state, name)
unsigned long torture_random(struct torture_random_state *trsp); unsigned long torture_random(struct torture_random_state *trsp);
/* Task shuffler, which causes CPUs to occasionally go idle. */ /* Task shuffler, which causes CPUs to occasionally go idle. */
...@@ -79,7 +81,7 @@ void stutter_wait(const char *title); ...@@ -79,7 +81,7 @@ void stutter_wait(const char *title);
int torture_stutter_init(int s); int torture_stutter_init(int s);
/* Initialization and cleanup. */ /* Initialization and cleanup. */
bool torture_init_begin(char *ttype, bool v); bool torture_init_begin(char *ttype, int v);
void torture_init_end(void); void torture_init_end(void);
bool torture_cleanup_begin(void); bool torture_cleanup_begin(void);
void torture_cleanup_end(void); void torture_cleanup_end(void);
......
This diff is collapsed.
...@@ -21,6 +21,9 @@ ...@@ -21,6 +21,9 @@
* Davidlohr Bueso <dave@stgolabs.net> * Davidlohr Bueso <dave@stgolabs.net>
* Based on kernel/rcu/torture.c. * Based on kernel/rcu/torture.c.
*/ */
#define pr_fmt(fmt) fmt
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/module.h> #include <linux/module.h>
#include <linux/kthread.h> #include <linux/kthread.h>
...@@ -57,7 +60,7 @@ torture_param(int, shutdown_secs, 0, "Shutdown time (j), <= zero to disable."); ...@@ -57,7 +60,7 @@ torture_param(int, shutdown_secs, 0, "Shutdown time (j), <= zero to disable.");
torture_param(int, stat_interval, 60, torture_param(int, stat_interval, 60,
"Number of seconds between stats printk()s"); "Number of seconds between stats printk()s");
torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable"); torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
torture_param(bool, verbose, true, torture_param(int, verbose, 1,
"Enable verbose debugging printk()s"); "Enable verbose debugging printk()s");
static char *torture_type = "spin_lock"; static char *torture_type = "spin_lock";
......
...@@ -91,7 +91,17 @@ static inline void rcu_seq_end(unsigned long *sp) ...@@ -91,7 +91,17 @@ static inline void rcu_seq_end(unsigned long *sp)
WRITE_ONCE(*sp, rcu_seq_endval(sp)); WRITE_ONCE(*sp, rcu_seq_endval(sp));
} }
/* Take a snapshot of the update side's sequence number. */ /*
* rcu_seq_snap - Take a snapshot of the update side's sequence number.
*
* This function returns the earliest value of the grace-period sequence number
* that will indicate that a full grace period has elapsed since the current
* time. Once the grace-period sequence number has reached this value, it will
* be safe to invoke all callbacks that have been registered prior to the
* current time. This value is the current grace-period number plus two to the
* power of the number of low-order bits reserved for state, then rounded up to
* the next value in which the state bits are all zero.
*/
static inline unsigned long rcu_seq_snap(unsigned long *sp) static inline unsigned long rcu_seq_snap(unsigned long *sp)
{ {
unsigned long s; unsigned long s;
...@@ -107,6 +117,15 @@ static inline unsigned long rcu_seq_current(unsigned long *sp) ...@@ -107,6 +117,15 @@ static inline unsigned long rcu_seq_current(unsigned long *sp)
return READ_ONCE(*sp); return READ_ONCE(*sp);
} }
/*
* Given a snapshot from rcu_seq_snap(), determine whether or not the
* corresponding update-side operation has started.
*/
static inline bool rcu_seq_started(unsigned long *sp, unsigned long s)
{
return ULONG_CMP_LT((s - 1) & ~RCU_SEQ_STATE_MASK, READ_ONCE(*sp));
}
/* /*
* Given a snapshot from rcu_seq_snap(), determine whether or not a * Given a snapshot from rcu_seq_snap(), determine whether or not a
* full update-side operation has occurred. * full update-side operation has occurred.
...@@ -116,6 +135,45 @@ static inline bool rcu_seq_done(unsigned long *sp, unsigned long s) ...@@ -116,6 +135,45 @@ static inline bool rcu_seq_done(unsigned long *sp, unsigned long s)
return ULONG_CMP_GE(READ_ONCE(*sp), s); return ULONG_CMP_GE(READ_ONCE(*sp), s);
} }
/*
* Has a grace period completed since the time the old gp_seq was collected?
*/
static inline bool rcu_seq_completed_gp(unsigned long old, unsigned long new)
{
return ULONG_CMP_LT(old, new & ~RCU_SEQ_STATE_MASK);
}
/*
* Has a grace period started since the time the old gp_seq was collected?
*/
static inline bool rcu_seq_new_gp(unsigned long old, unsigned long new)
{
return ULONG_CMP_LT((old + RCU_SEQ_STATE_MASK) & ~RCU_SEQ_STATE_MASK,
new);
}
/*
* Roughly how many full grace periods have elapsed between the collection
* of the two specified grace periods?
*/
static inline unsigned long rcu_seq_diff(unsigned long new, unsigned long old)
{
unsigned long rnd_diff;
if (old == new)
return 0;
/*
* Compute the number of grace periods (still shifted up), plus
* one if either of new and old is not an exact grace period.
*/
rnd_diff = (new & ~RCU_SEQ_STATE_MASK) -
((old + RCU_SEQ_STATE_MASK) & ~RCU_SEQ_STATE_MASK) +
((new & RCU_SEQ_STATE_MASK) || (old & RCU_SEQ_STATE_MASK));
if (ULONG_CMP_GE(RCU_SEQ_STATE_MASK, rnd_diff))
return 1; /* Definitely no grace period has elapsed. */
return ((rnd_diff - RCU_SEQ_STATE_MASK - 1) >> RCU_SEQ_CTR_SHIFT) + 2;
}
/* /*
* debug_rcu_head_queue()/debug_rcu_head_unqueue() are used internally * debug_rcu_head_queue()/debug_rcu_head_unqueue() are used internally
* by call_rcu() and rcu callback execution, and are therefore not part of the * by call_rcu() and rcu callback execution, and are therefore not part of the
...@@ -276,6 +334,9 @@ static inline void rcu_init_levelspread(int *levelspread, const int *levelcnt) ...@@ -276,6 +334,9 @@ static inline void rcu_init_levelspread(int *levelspread, const int *levelcnt)
/* Is this rcu_node a leaf? */ /* Is this rcu_node a leaf? */
#define rcu_is_leaf_node(rnp) ((rnp)->level == rcu_num_lvls - 1) #define rcu_is_leaf_node(rnp) ((rnp)->level == rcu_num_lvls - 1)
/* Is this rcu_node the last leaf? */
#define rcu_is_last_leaf_node(rsp, rnp) ((rnp) == &(rsp)->node[rcu_num_nodes - 1])
/* /*
* Do a full breadth-first scan of the rcu_node structures for the * Do a full breadth-first scan of the rcu_node structures for the
* specified rcu_state structure. * specified rcu_state structure.
...@@ -405,8 +466,7 @@ enum rcutorture_type { ...@@ -405,8 +466,7 @@ enum rcutorture_type {
#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU) #if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU)
void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags,
unsigned long *gpnum, unsigned long *completed); unsigned long *gp_seq);
void rcutorture_record_test_transition(void);
void rcutorture_record_progress(unsigned long vernum); void rcutorture_record_progress(unsigned long vernum);
void do_trace_rcu_torture_read(const char *rcutorturename, void do_trace_rcu_torture_read(const char *rcutorturename,
struct rcu_head *rhp, struct rcu_head *rhp,
...@@ -415,15 +475,11 @@ void do_trace_rcu_torture_read(const char *rcutorturename, ...@@ -415,15 +475,11 @@ void do_trace_rcu_torture_read(const char *rcutorturename,
unsigned long c); unsigned long c);
#else #else
static inline void rcutorture_get_gp_data(enum rcutorture_type test_type, static inline void rcutorture_get_gp_data(enum rcutorture_type test_type,
int *flags, int *flags, unsigned long *gp_seq)
unsigned long *gpnum,
unsigned long *completed)
{ {
*flags = 0; *flags = 0;
*gpnum = 0; *gp_seq = 0;
*completed = 0;
} }
static inline void rcutorture_record_test_transition(void) { }
static inline void rcutorture_record_progress(unsigned long vernum) { } static inline void rcutorture_record_progress(unsigned long vernum) { }
#ifdef CONFIG_RCU_TRACE #ifdef CONFIG_RCU_TRACE
void do_trace_rcu_torture_read(const char *rcutorturename, void do_trace_rcu_torture_read(const char *rcutorturename,
...@@ -441,31 +497,26 @@ void do_trace_rcu_torture_read(const char *rcutorturename, ...@@ -441,31 +497,26 @@ void do_trace_rcu_torture_read(const char *rcutorturename,
static inline void srcutorture_get_gp_data(enum rcutorture_type test_type, static inline void srcutorture_get_gp_data(enum rcutorture_type test_type,
struct srcu_struct *sp, int *flags, struct srcu_struct *sp, int *flags,
unsigned long *gpnum, unsigned long *gp_seq)
unsigned long *completed)
{ {
if (test_type != SRCU_FLAVOR) if (test_type != SRCU_FLAVOR)
return; return;
*flags = 0; *flags = 0;
*completed = sp->srcu_idx; *gp_seq = sp->srcu_idx;
*gpnum = *completed;
} }
#elif defined(CONFIG_TREE_SRCU) #elif defined(CONFIG_TREE_SRCU)
void srcutorture_get_gp_data(enum rcutorture_type test_type, void srcutorture_get_gp_data(enum rcutorture_type test_type,
struct srcu_struct *sp, int *flags, struct srcu_struct *sp, int *flags,
unsigned long *gpnum, unsigned long *completed); unsigned long *gp_seq);
#endif #endif
#ifdef CONFIG_TINY_RCU #ifdef CONFIG_TINY_RCU
static inline unsigned long rcu_batches_started(void) { return 0; } static inline unsigned long rcu_get_gp_seq(void) { return 0; }
static inline unsigned long rcu_batches_started_bh(void) { return 0; } static inline unsigned long rcu_bh_get_gp_seq(void) { return 0; }
static inline unsigned long rcu_batches_started_sched(void) { return 0; } static inline unsigned long rcu_sched_get_gp_seq(void) { return 0; }
static inline unsigned long rcu_batches_completed(void) { return 0; }
static inline unsigned long rcu_batches_completed_bh(void) { return 0; }
static inline unsigned long rcu_batches_completed_sched(void) { return 0; }
static inline unsigned long rcu_exp_batches_completed(void) { return 0; } static inline unsigned long rcu_exp_batches_completed(void) { return 0; }
static inline unsigned long rcu_exp_batches_completed_sched(void) { return 0; } static inline unsigned long rcu_exp_batches_completed_sched(void) { return 0; }
static inline unsigned long static inline unsigned long
...@@ -474,19 +525,16 @@ static inline void rcu_force_quiescent_state(void) { } ...@@ -474,19 +525,16 @@ static inline void rcu_force_quiescent_state(void) { }
static inline void rcu_bh_force_quiescent_state(void) { } static inline void rcu_bh_force_quiescent_state(void) { }
static inline void rcu_sched_force_quiescent_state(void) { } static inline void rcu_sched_force_quiescent_state(void) { }
static inline void show_rcu_gp_kthreads(void) { } static inline void show_rcu_gp_kthreads(void) { }
static inline int rcu_get_gp_kthreads_prio(void) { return 0; }
#else /* #ifdef CONFIG_TINY_RCU */ #else /* #ifdef CONFIG_TINY_RCU */
extern unsigned long rcutorture_testseq; unsigned long rcu_get_gp_seq(void);
extern unsigned long rcutorture_vernum; unsigned long rcu_bh_get_gp_seq(void);
unsigned long rcu_batches_started(void); unsigned long rcu_sched_get_gp_seq(void);
unsigned long rcu_batches_started_bh(void);
unsigned long rcu_batches_started_sched(void);
unsigned long rcu_batches_completed(void);
unsigned long rcu_batches_completed_bh(void);
unsigned long rcu_batches_completed_sched(void);
unsigned long rcu_exp_batches_completed(void); unsigned long rcu_exp_batches_completed(void);
unsigned long rcu_exp_batches_completed_sched(void); unsigned long rcu_exp_batches_completed_sched(void);
unsigned long srcu_batches_completed(struct srcu_struct *sp); unsigned long srcu_batches_completed(struct srcu_struct *sp);
void show_rcu_gp_kthreads(void); void show_rcu_gp_kthreads(void);
int rcu_get_gp_kthreads_prio(void);
void rcu_force_quiescent_state(void); void rcu_force_quiescent_state(void);
void rcu_bh_force_quiescent_state(void); void rcu_bh_force_quiescent_state(void);
void rcu_sched_force_quiescent_state(void); void rcu_sched_force_quiescent_state(void);
......
...@@ -19,6 +19,9 @@ ...@@ -19,6 +19,9 @@
* *
* Authors: Paul E. McKenney <paulmck@us.ibm.com> * Authors: Paul E. McKenney <paulmck@us.ibm.com>
*/ */
#define pr_fmt(fmt) fmt
#include <linux/types.h> #include <linux/types.h>
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/init.h> #include <linux/init.h>
...@@ -88,7 +91,7 @@ torture_param(int, nreaders, -1, "Number of RCU reader threads"); ...@@ -88,7 +91,7 @@ torture_param(int, nreaders, -1, "Number of RCU reader threads");
torture_param(int, nwriters, -1, "Number of RCU updater threads"); torture_param(int, nwriters, -1, "Number of RCU updater threads");
torture_param(bool, shutdown, !IS_ENABLED(MODULE), torture_param(bool, shutdown, !IS_ENABLED(MODULE),
"Shutdown at end of performance tests."); "Shutdown at end of performance tests.");
torture_param(bool, verbose, true, "Enable verbose debugging printk()s"); torture_param(int, verbose, 1, "Enable verbose debugging printk()s");
torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable"); torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable");
static char *perf_type = "rcu"; static char *perf_type = "rcu";
...@@ -135,8 +138,8 @@ struct rcu_perf_ops { ...@@ -135,8 +138,8 @@ struct rcu_perf_ops {
void (*cleanup)(void); void (*cleanup)(void);
int (*readlock)(void); int (*readlock)(void);
void (*readunlock)(int idx); void (*readunlock)(int idx);
unsigned long (*started)(void); unsigned long (*get_gp_seq)(void);
unsigned long (*completed)(void); unsigned long (*gp_diff)(unsigned long new, unsigned long old);
unsigned long (*exp_completed)(void); unsigned long (*exp_completed)(void);
void (*async)(struct rcu_head *head, rcu_callback_t func); void (*async)(struct rcu_head *head, rcu_callback_t func);
void (*gp_barrier)(void); void (*gp_barrier)(void);
...@@ -176,8 +179,8 @@ static struct rcu_perf_ops rcu_ops = { ...@@ -176,8 +179,8 @@ static struct rcu_perf_ops rcu_ops = {
.init = rcu_sync_perf_init, .init = rcu_sync_perf_init,
.readlock = rcu_perf_read_lock, .readlock = rcu_perf_read_lock,
.readunlock = rcu_perf_read_unlock, .readunlock = rcu_perf_read_unlock,
.started = rcu_batches_started, .get_gp_seq = rcu_get_gp_seq,
.completed = rcu_batches_completed, .gp_diff = rcu_seq_diff,
.exp_completed = rcu_exp_batches_completed, .exp_completed = rcu_exp_batches_completed,
.async = call_rcu, .async = call_rcu,
.gp_barrier = rcu_barrier, .gp_barrier = rcu_barrier,
...@@ -206,8 +209,8 @@ static struct rcu_perf_ops rcu_bh_ops = { ...@@ -206,8 +209,8 @@ static struct rcu_perf_ops rcu_bh_ops = {
.init = rcu_sync_perf_init, .init = rcu_sync_perf_init,
.readlock = rcu_bh_perf_read_lock, .readlock = rcu_bh_perf_read_lock,
.readunlock = rcu_bh_perf_read_unlock, .readunlock = rcu_bh_perf_read_unlock,
.started = rcu_batches_started_bh, .get_gp_seq = rcu_bh_get_gp_seq,
.completed = rcu_batches_completed_bh, .gp_diff = rcu_seq_diff,
.exp_completed = rcu_exp_batches_completed_sched, .exp_completed = rcu_exp_batches_completed_sched,
.async = call_rcu_bh, .async = call_rcu_bh,
.gp_barrier = rcu_barrier_bh, .gp_barrier = rcu_barrier_bh,
...@@ -263,8 +266,8 @@ static struct rcu_perf_ops srcu_ops = { ...@@ -263,8 +266,8 @@ static struct rcu_perf_ops srcu_ops = {
.init = rcu_sync_perf_init, .init = rcu_sync_perf_init,
.readlock = srcu_perf_read_lock, .readlock = srcu_perf_read_lock,
.readunlock = srcu_perf_read_unlock, .readunlock = srcu_perf_read_unlock,
.started = NULL, .get_gp_seq = srcu_perf_completed,
.completed = srcu_perf_completed, .gp_diff = rcu_seq_diff,
.exp_completed = srcu_perf_completed, .exp_completed = srcu_perf_completed,
.async = srcu_call_rcu, .async = srcu_call_rcu,
.gp_barrier = srcu_rcu_barrier, .gp_barrier = srcu_rcu_barrier,
...@@ -292,8 +295,8 @@ static struct rcu_perf_ops srcud_ops = { ...@@ -292,8 +295,8 @@ static struct rcu_perf_ops srcud_ops = {
.cleanup = srcu_sync_perf_cleanup, .cleanup = srcu_sync_perf_cleanup,
.readlock = srcu_perf_read_lock, .readlock = srcu_perf_read_lock,
.readunlock = srcu_perf_read_unlock, .readunlock = srcu_perf_read_unlock,
.started = NULL, .get_gp_seq = srcu_perf_completed,
.completed = srcu_perf_completed, .gp_diff = rcu_seq_diff,
.exp_completed = srcu_perf_completed, .exp_completed = srcu_perf_completed,
.async = srcu_call_rcu, .async = srcu_call_rcu,
.gp_barrier = srcu_rcu_barrier, .gp_barrier = srcu_rcu_barrier,
...@@ -322,8 +325,8 @@ static struct rcu_perf_ops sched_ops = { ...@@ -322,8 +325,8 @@ static struct rcu_perf_ops sched_ops = {
.init = rcu_sync_perf_init, .init = rcu_sync_perf_init,
.readlock = sched_perf_read_lock, .readlock = sched_perf_read_lock,
.readunlock = sched_perf_read_unlock, .readunlock = sched_perf_read_unlock,
.started = rcu_batches_started_sched, .get_gp_seq = rcu_sched_get_gp_seq,
.completed = rcu_batches_completed_sched, .gp_diff = rcu_seq_diff,
.exp_completed = rcu_exp_batches_completed_sched, .exp_completed = rcu_exp_batches_completed_sched,
.async = call_rcu_sched, .async = call_rcu_sched,
.gp_barrier = rcu_barrier_sched, .gp_barrier = rcu_barrier_sched,
...@@ -350,8 +353,8 @@ static struct rcu_perf_ops tasks_ops = { ...@@ -350,8 +353,8 @@ static struct rcu_perf_ops tasks_ops = {
.init = rcu_sync_perf_init, .init = rcu_sync_perf_init,
.readlock = tasks_perf_read_lock, .readlock = tasks_perf_read_lock,
.readunlock = tasks_perf_read_unlock, .readunlock = tasks_perf_read_unlock,
.started = rcu_no_completed, .get_gp_seq = rcu_no_completed,
.completed = rcu_no_completed, .gp_diff = rcu_seq_diff,
.async = call_rcu_tasks, .async = call_rcu_tasks,
.gp_barrier = rcu_barrier_tasks, .gp_barrier = rcu_barrier_tasks,
.sync = synchronize_rcu_tasks, .sync = synchronize_rcu_tasks,
...@@ -359,9 +362,11 @@ static struct rcu_perf_ops tasks_ops = { ...@@ -359,9 +362,11 @@ static struct rcu_perf_ops tasks_ops = {
.name = "tasks" .name = "tasks"
}; };
static bool __maybe_unused torturing_tasks(void) static unsigned long rcuperf_seq_diff(unsigned long new, unsigned long old)
{ {
return cur_ops == &tasks_ops; if (!cur_ops->gp_diff)
return new - old;
return cur_ops->gp_diff(new, old);
} }
/* /*
...@@ -444,8 +449,7 @@ rcu_perf_writer(void *arg) ...@@ -444,8 +449,7 @@ rcu_perf_writer(void *arg)
b_rcu_perf_writer_started = b_rcu_perf_writer_started =
cur_ops->exp_completed() / 2; cur_ops->exp_completed() / 2;
} else { } else {
b_rcu_perf_writer_started = b_rcu_perf_writer_started = cur_ops->get_gp_seq();
cur_ops->completed();
} }
} }
...@@ -502,7 +506,7 @@ rcu_perf_writer(void *arg) ...@@ -502,7 +506,7 @@ rcu_perf_writer(void *arg)
cur_ops->exp_completed() / 2; cur_ops->exp_completed() / 2;
} else { } else {
b_rcu_perf_writer_finished = b_rcu_perf_writer_finished =
cur_ops->completed(); cur_ops->get_gp_seq();
} }
if (shutdown) { if (shutdown) {
smp_mb(); /* Assign before wake. */ smp_mb(); /* Assign before wake. */
...@@ -527,7 +531,7 @@ rcu_perf_writer(void *arg) ...@@ -527,7 +531,7 @@ rcu_perf_writer(void *arg)
return 0; return 0;
} }
static inline void static void
rcu_perf_print_module_parms(struct rcu_perf_ops *cur_ops, const char *tag) rcu_perf_print_module_parms(struct rcu_perf_ops *cur_ops, const char *tag)
{ {
pr_alert("%s" PERF_FLAG pr_alert("%s" PERF_FLAG
...@@ -582,8 +586,8 @@ rcu_perf_cleanup(void) ...@@ -582,8 +586,8 @@ rcu_perf_cleanup(void)
t_rcu_perf_writer_finished - t_rcu_perf_writer_finished -
t_rcu_perf_writer_started, t_rcu_perf_writer_started,
ngps, ngps,
b_rcu_perf_writer_finished - rcuperf_seq_diff(b_rcu_perf_writer_finished,
b_rcu_perf_writer_started); b_rcu_perf_writer_started));
for (i = 0; i < nrealwriters; i++) { for (i = 0; i < nrealwriters; i++) {
if (!writer_durations) if (!writer_durations)
break; break;
...@@ -671,12 +675,11 @@ rcu_perf_init(void) ...@@ -671,12 +675,11 @@ rcu_perf_init(void)
break; break;
} }
if (i == ARRAY_SIZE(perf_ops)) { if (i == ARRAY_SIZE(perf_ops)) {
pr_alert("rcu-perf: invalid perf type: \"%s\"\n", pr_alert("rcu-perf: invalid perf type: \"%s\"\n", perf_type);
perf_type);
pr_alert("rcu-perf types:"); pr_alert("rcu-perf types:");
for (i = 0; i < ARRAY_SIZE(perf_ops); i++) for (i = 0; i < ARRAY_SIZE(perf_ops); i++)
pr_alert(" %s", perf_ops[i]->name); pr_cont(" %s", perf_ops[i]->name);
pr_alert("\n"); pr_cont("\n");
firsterr = -EINVAL; firsterr = -EINVAL;
goto unwind; goto unwind;
} }
......
This diff is collapsed.
...@@ -26,6 +26,8 @@ ...@@ -26,6 +26,8 @@
* *
*/ */
#define pr_fmt(fmt) "rcu: " fmt
#include <linux/export.h> #include <linux/export.h>
#include <linux/mutex.h> #include <linux/mutex.h>
#include <linux/percpu.h> #include <linux/percpu.h>
...@@ -390,7 +392,8 @@ void _cleanup_srcu_struct(struct srcu_struct *sp, bool quiesced) ...@@ -390,7 +392,8 @@ void _cleanup_srcu_struct(struct srcu_struct *sp, bool quiesced)
} }
if (WARN_ON(rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)) != SRCU_STATE_IDLE) || if (WARN_ON(rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)) != SRCU_STATE_IDLE) ||
WARN_ON(srcu_readers_active(sp))) { WARN_ON(srcu_readers_active(sp))) {
pr_info("%s: Active srcu_struct %p state: %d\n", __func__, sp, rcu_seq_state(READ_ONCE(sp->srcu_gp_seq))); pr_info("%s: Active srcu_struct %p state: %d\n",
__func__, sp, rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)));
return; /* Caller forgot to stop doing call_srcu()? */ return; /* Caller forgot to stop doing call_srcu()? */
} }
free_percpu(sp->sda); free_percpu(sp->sda);
...@@ -641,6 +644,9 @@ static void srcu_funnel_exp_start(struct srcu_struct *sp, struct srcu_node *snp, ...@@ -641,6 +644,9 @@ static void srcu_funnel_exp_start(struct srcu_struct *sp, struct srcu_node *snp,
* period s. Losers must either ensure that their desired grace-period * period s. Losers must either ensure that their desired grace-period
* number is recorded on at least their leaf srcu_node structure, or they * number is recorded on at least their leaf srcu_node structure, or they
* must take steps to invoke their own callbacks. * must take steps to invoke their own callbacks.
*
* Note that this function also does the work of srcu_funnel_exp_start(),
* in some cases by directly invoking it.
*/ */
static void srcu_funnel_gp_start(struct srcu_struct *sp, struct srcu_data *sdp, static void srcu_funnel_gp_start(struct srcu_struct *sp, struct srcu_data *sdp,
unsigned long s, bool do_norm) unsigned long s, bool do_norm)
...@@ -823,17 +829,17 @@ static void srcu_leak_callback(struct rcu_head *rhp) ...@@ -823,17 +829,17 @@ static void srcu_leak_callback(struct rcu_head *rhp)
* more than one CPU, this means that when "func()" is invoked, each CPU * more than one CPU, this means that when "func()" is invoked, each CPU
* is guaranteed to have executed a full memory barrier since the end of * is guaranteed to have executed a full memory barrier since the end of
* its last corresponding SRCU read-side critical section whose beginning * its last corresponding SRCU read-side critical section whose beginning
* preceded the call to call_rcu(). It also means that each CPU executing * preceded the call to call_srcu(). It also means that each CPU executing
* an SRCU read-side critical section that continues beyond the start of * an SRCU read-side critical section that continues beyond the start of
* "func()" must have executed a memory barrier after the call_rcu() * "func()" must have executed a memory barrier after the call_srcu()
* but before the beginning of that SRCU read-side critical section. * but before the beginning of that SRCU read-side critical section.
* Note that these guarantees include CPUs that are offline, idle, or * Note that these guarantees include CPUs that are offline, idle, or
* executing in user mode, as well as CPUs that are executing in the kernel. * executing in user mode, as well as CPUs that are executing in the kernel.
* *
* Furthermore, if CPU A invoked call_rcu() and CPU B invoked the * Furthermore, if CPU A invoked call_srcu() and CPU B invoked the
* resulting SRCU callback function "func()", then both CPU A and CPU * resulting SRCU callback function "func()", then both CPU A and CPU
* B are guaranteed to execute a full memory barrier during the time * B are guaranteed to execute a full memory barrier during the time
* interval between the call to call_rcu() and the invocation of "func()". * interval between the call to call_srcu() and the invocation of "func()".
* This guarantee applies even if CPU A and CPU B are the same CPU (but * This guarantee applies even if CPU A and CPU B are the same CPU (but
* again only if the system has more than one CPU). * again only if the system has more than one CPU).
* *
...@@ -1246,13 +1252,12 @@ static void process_srcu(struct work_struct *work) ...@@ -1246,13 +1252,12 @@ static void process_srcu(struct work_struct *work)
void srcutorture_get_gp_data(enum rcutorture_type test_type, void srcutorture_get_gp_data(enum rcutorture_type test_type,
struct srcu_struct *sp, int *flags, struct srcu_struct *sp, int *flags,
unsigned long *gpnum, unsigned long *completed) unsigned long *gp_seq)
{ {
if (test_type != SRCU_FLAVOR) if (test_type != SRCU_FLAVOR)
return; return;
*flags = 0; *flags = 0;
*completed = rcu_seq_ctr(sp->srcu_gp_seq); *gp_seq = rcu_seq_current(&sp->srcu_gp_seq);
*gpnum = rcu_seq_ctr(sp->srcu_gp_seq_needed);
} }
EXPORT_SYMBOL_GPL(srcutorture_get_gp_data); EXPORT_SYMBOL_GPL(srcutorture_get_gp_data);
...@@ -1263,16 +1268,17 @@ void srcu_torture_stats_print(struct srcu_struct *sp, char *tt, char *tf) ...@@ -1263,16 +1268,17 @@ void srcu_torture_stats_print(struct srcu_struct *sp, char *tt, char *tf)
unsigned long s0 = 0, s1 = 0; unsigned long s0 = 0, s1 = 0;
idx = sp->srcu_idx & 0x1; idx = sp->srcu_idx & 0x1;
pr_alert("%s%s Tree SRCU per-CPU(idx=%d):", tt, tf, idx); pr_alert("%s%s Tree SRCU g%ld per-CPU(idx=%d):",
tt, tf, rcu_seq_current(&sp->srcu_gp_seq), idx);
for_each_possible_cpu(cpu) { for_each_possible_cpu(cpu) {
unsigned long l0, l1; unsigned long l0, l1;
unsigned long u0, u1; unsigned long u0, u1;
long c0, c1; long c0, c1;
struct srcu_data *counts; struct srcu_data *sdp;
counts = per_cpu_ptr(sp->sda, cpu); sdp = per_cpu_ptr(sp->sda, cpu);
u0 = counts->srcu_unlock_count[!idx]; u0 = sdp->srcu_unlock_count[!idx];
u1 = counts->srcu_unlock_count[idx]; u1 = sdp->srcu_unlock_count[idx];
/* /*
* Make sure that a lock is always counted if the corresponding * Make sure that a lock is always counted if the corresponding
...@@ -1280,12 +1286,13 @@ void srcu_torture_stats_print(struct srcu_struct *sp, char *tt, char *tf) ...@@ -1280,12 +1286,13 @@ void srcu_torture_stats_print(struct srcu_struct *sp, char *tt, char *tf)
*/ */
smp_rmb(); smp_rmb();
l0 = counts->srcu_lock_count[!idx]; l0 = sdp->srcu_lock_count[!idx];
l1 = counts->srcu_lock_count[idx]; l1 = sdp->srcu_lock_count[idx];
c0 = l0 - u0; c0 = l0 - u0;
c1 = l1 - u1; c1 = l1 - u1;
pr_cont(" %d(%ld,%ld)", cpu, c0, c1); pr_cont(" %d(%ld,%ld %1p)",
cpu, c0, c1, rcu_segcblist_head(&sdp->srcu_cblist));
s0 += c0; s0 += c0;
s1 += c1; s1 += c1;
} }
......
...@@ -122,10 +122,8 @@ void rcu_check_callbacks(int user) ...@@ -122,10 +122,8 @@ void rcu_check_callbacks(int user)
{ {
if (user) if (user)
rcu_sched_qs(); rcu_sched_qs();
else if (!in_softirq()) if (user || !in_softirq())
rcu_bh_qs(); rcu_bh_qs();
if (user)
rcu_note_voluntary_context_switch(current);
} }
/* /*
......
This diff is collapsed.
...@@ -81,18 +81,16 @@ struct rcu_node { ...@@ -81,18 +81,16 @@ struct rcu_node {
raw_spinlock_t __private lock; /* Root rcu_node's lock protects */ raw_spinlock_t __private lock; /* Root rcu_node's lock protects */
/* some rcu_state fields as well as */ /* some rcu_state fields as well as */
/* following. */ /* following. */
unsigned long gpnum; /* Current grace period for this node. */ unsigned long gp_seq; /* Track rsp->rcu_gp_seq. */
/* This will either be equal to or one */ unsigned long gp_seq_needed; /* Track rsp->rcu_gp_seq_needed. */
/* behind the root rcu_node's gpnum. */ unsigned long completedqs; /* All QSes done for this node. */
unsigned long completed; /* Last GP completed for this node. */
/* This will either be equal to or one */
/* behind the root rcu_node's gpnum. */
unsigned long qsmask; /* CPUs or groups that need to switch in */ unsigned long qsmask; /* CPUs or groups that need to switch in */
/* order for current grace period to proceed.*/ /* order for current grace period to proceed.*/
/* In leaf rcu_node, each bit corresponds to */ /* In leaf rcu_node, each bit corresponds to */
/* an rcu_data structure, otherwise, each */ /* an rcu_data structure, otherwise, each */
/* bit corresponds to a child rcu_node */ /* bit corresponds to a child rcu_node */
/* structure. */ /* structure. */
unsigned long rcu_gp_init_mask; /* Mask of offline CPUs at GP init. */
unsigned long qsmaskinit; unsigned long qsmaskinit;
/* Per-GP initial value for qsmask. */ /* Per-GP initial value for qsmask. */
/* Initialized from ->qsmaskinitnext at the */ /* Initialized from ->qsmaskinitnext at the */
...@@ -158,7 +156,6 @@ struct rcu_node { ...@@ -158,7 +156,6 @@ struct rcu_node {
struct swait_queue_head nocb_gp_wq[2]; struct swait_queue_head nocb_gp_wq[2];
/* Place for rcu_nocb_kthread() to wait GP. */ /* Place for rcu_nocb_kthread() to wait GP. */
#endif /* #ifdef CONFIG_RCU_NOCB_CPU */ #endif /* #ifdef CONFIG_RCU_NOCB_CPU */
u8 need_future_gp[4]; /* Counts of upcoming GP requests. */
raw_spinlock_t fqslock ____cacheline_internodealigned_in_smp; raw_spinlock_t fqslock ____cacheline_internodealigned_in_smp;
spinlock_t exp_lock ____cacheline_internodealigned_in_smp; spinlock_t exp_lock ____cacheline_internodealigned_in_smp;
...@@ -168,22 +165,6 @@ struct rcu_node { ...@@ -168,22 +165,6 @@ struct rcu_node {
bool exp_need_flush; /* Need to flush workitem? */ bool exp_need_flush; /* Need to flush workitem? */
} ____cacheline_internodealigned_in_smp; } ____cacheline_internodealigned_in_smp;
/* Accessors for ->need_future_gp[] array. */
#define need_future_gp_mask() \
(ARRAY_SIZE(((struct rcu_node *)NULL)->need_future_gp) - 1)
#define need_future_gp_element(rnp, c) \
((rnp)->need_future_gp[(c) & need_future_gp_mask()])
#define need_any_future_gp(rnp) \
({ \
int __i; \
bool __nonzero = false; \
\
for (__i = 0; __i < ARRAY_SIZE((rnp)->need_future_gp); __i++) \
__nonzero = __nonzero || \
READ_ONCE((rnp)->need_future_gp[__i]); \
__nonzero; \
})
/* /*
* Bitmasks in an rcu_node cover the interval [grplo, grphi] of CPU IDs, and * Bitmasks in an rcu_node cover the interval [grplo, grphi] of CPU IDs, and
* are indexed relative to this interval rather than the global CPU ID space. * are indexed relative to this interval rather than the global CPU ID space.
...@@ -206,16 +187,14 @@ union rcu_noqs { ...@@ -206,16 +187,14 @@ union rcu_noqs {
/* Per-CPU data for read-copy update. */ /* Per-CPU data for read-copy update. */
struct rcu_data { struct rcu_data {
/* 1) quiescent-state and grace-period handling : */ /* 1) quiescent-state and grace-period handling : */
unsigned long completed; /* Track rsp->completed gp number */ unsigned long gp_seq; /* Track rsp->rcu_gp_seq counter. */
/* in order to detect GP end. */ unsigned long gp_seq_needed; /* Track rsp->rcu_gp_seq_needed ctr. */
unsigned long gpnum; /* Highest gp number that this CPU */
/* is aware of having started. */
unsigned long rcu_qs_ctr_snap;/* Snapshot of rcu_qs_ctr to check */ unsigned long rcu_qs_ctr_snap;/* Snapshot of rcu_qs_ctr to check */
/* for rcu_all_qs() invocations. */ /* for rcu_all_qs() invocations. */
union rcu_noqs cpu_no_qs; /* No QSes yet for this CPU. */ union rcu_noqs cpu_no_qs; /* No QSes yet for this CPU. */
bool core_needs_qs; /* Core waits for quiesc state. */ bool core_needs_qs; /* Core waits for quiesc state. */
bool beenonline; /* CPU online at least once. */ bool beenonline; /* CPU online at least once. */
bool gpwrap; /* Possible gpnum/completed wrap. */ bool gpwrap; /* Possible ->gp_seq wrap. */
struct rcu_node *mynode; /* This CPU's leaf of hierarchy */ struct rcu_node *mynode; /* This CPU's leaf of hierarchy */
unsigned long grpmask; /* Mask to apply to leaf qsmask. */ unsigned long grpmask; /* Mask to apply to leaf qsmask. */
unsigned long ticks_this_gp; /* The number of scheduling-clock */ unsigned long ticks_this_gp; /* The number of scheduling-clock */
...@@ -239,7 +218,6 @@ struct rcu_data { ...@@ -239,7 +218,6 @@ struct rcu_data {
/* 4) reasons this CPU needed to be kicked by force_quiescent_state */ /* 4) reasons this CPU needed to be kicked by force_quiescent_state */
unsigned long dynticks_fqs; /* Kicked due to dynticks idle. */ unsigned long dynticks_fqs; /* Kicked due to dynticks idle. */
unsigned long offline_fqs; /* Kicked due to being offline. */
unsigned long cond_resched_completed; unsigned long cond_resched_completed;
/* Grace period that needs help */ /* Grace period that needs help */
/* from cond_resched(). */ /* from cond_resched(). */
...@@ -278,12 +256,16 @@ struct rcu_data { ...@@ -278,12 +256,16 @@ struct rcu_data {
/* Leader CPU takes GP-end wakeups. */ /* Leader CPU takes GP-end wakeups. */
#endif /* #ifdef CONFIG_RCU_NOCB_CPU */ #endif /* #ifdef CONFIG_RCU_NOCB_CPU */
/* 7) RCU CPU stall data. */ /* 7) Diagnostic data, including RCU CPU stall warnings. */
unsigned int softirq_snap; /* Snapshot of softirq activity. */ unsigned int softirq_snap; /* Snapshot of softirq activity. */
/* ->rcu_iw* fields protected by leaf rcu_node ->lock. */ /* ->rcu_iw* fields protected by leaf rcu_node ->lock. */
struct irq_work rcu_iw; /* Check for non-irq activity. */ struct irq_work rcu_iw; /* Check for non-irq activity. */
bool rcu_iw_pending; /* Is ->rcu_iw pending? */ bool rcu_iw_pending; /* Is ->rcu_iw pending? */
unsigned long rcu_iw_gpnum; /* ->gpnum associated with ->rcu_iw. */ unsigned long rcu_iw_gp_seq; /* ->gp_seq associated with ->rcu_iw. */
unsigned long rcu_ofl_gp_seq; /* ->gp_seq at last offline. */
short rcu_ofl_gp_flags; /* ->gp_flags at last offline. */
unsigned long rcu_onl_gp_seq; /* ->gp_seq at last online. */
short rcu_onl_gp_flags; /* ->gp_flags at last online. */
int cpu; int cpu;
struct rcu_state *rsp; struct rcu_state *rsp;
...@@ -340,8 +322,7 @@ struct rcu_state { ...@@ -340,8 +322,7 @@ struct rcu_state {
u8 boost ____cacheline_internodealigned_in_smp; u8 boost ____cacheline_internodealigned_in_smp;
/* Subject to priority boost. */ /* Subject to priority boost. */
unsigned long gpnum; /* Current gp number. */ unsigned long gp_seq; /* Grace-period sequence #. */
unsigned long completed; /* # of last completed gp. */
struct task_struct *gp_kthread; /* Task for grace periods. */ struct task_struct *gp_kthread; /* Task for grace periods. */
struct swait_queue_head gp_wq; /* Where GP task waits. */ struct swait_queue_head gp_wq; /* Where GP task waits. */
short gp_flags; /* Commands for GP task. */ short gp_flags; /* Commands for GP task. */
...@@ -373,6 +354,8 @@ struct rcu_state { ...@@ -373,6 +354,8 @@ struct rcu_state {
/* but in jiffies. */ /* but in jiffies. */
unsigned long gp_activity; /* Time of last GP kthread */ unsigned long gp_activity; /* Time of last GP kthread */
/* activity in jiffies. */ /* activity in jiffies. */
unsigned long gp_req_activity; /* Time of last GP request */
/* in jiffies. */
unsigned long jiffies_stall; /* Time at which to check */ unsigned long jiffies_stall; /* Time at which to check */
/* for CPU stalls. */ /* for CPU stalls. */
unsigned long jiffies_resched; /* Time at which to resched */ unsigned long jiffies_resched; /* Time at which to resched */
...@@ -384,6 +367,10 @@ struct rcu_state { ...@@ -384,6 +367,10 @@ struct rcu_state {
const char *name; /* Name of structure. */ const char *name; /* Name of structure. */
char abbr; /* Abbreviated name. */ char abbr; /* Abbreviated name. */
struct list_head flavors; /* List of RCU flavors. */ struct list_head flavors; /* List of RCU flavors. */
spinlock_t ofl_lock ____cacheline_internodealigned_in_smp;
/* Synchronize offline with */
/* GP pre-initialization. */
}; };
/* Values for rcu_state structure's gp_flags field. */ /* Values for rcu_state structure's gp_flags field. */
...@@ -394,16 +381,20 @@ struct rcu_state { ...@@ -394,16 +381,20 @@ struct rcu_state {
#define RCU_GP_IDLE 0 /* Initial state and no GP in progress. */ #define RCU_GP_IDLE 0 /* Initial state and no GP in progress. */
#define RCU_GP_WAIT_GPS 1 /* Wait for grace-period start. */ #define RCU_GP_WAIT_GPS 1 /* Wait for grace-period start. */
#define RCU_GP_DONE_GPS 2 /* Wait done for grace-period start. */ #define RCU_GP_DONE_GPS 2 /* Wait done for grace-period start. */
#define RCU_GP_WAIT_FQS 3 /* Wait for force-quiescent-state time. */ #define RCU_GP_ONOFF 3 /* Grace-period initialization hotplug. */
#define RCU_GP_DOING_FQS 4 /* Wait done for force-quiescent-state time. */ #define RCU_GP_INIT 4 /* Grace-period initialization. */
#define RCU_GP_CLEANUP 5 /* Grace-period cleanup started. */ #define RCU_GP_WAIT_FQS 5 /* Wait for force-quiescent-state time. */
#define RCU_GP_CLEANED 6 /* Grace-period cleanup complete. */ #define RCU_GP_DOING_FQS 6 /* Wait done for force-quiescent-state time. */
#define RCU_GP_CLEANUP 7 /* Grace-period cleanup started. */
#define RCU_GP_CLEANED 8 /* Grace-period cleanup complete. */
#ifndef RCU_TREE_NONCORE #ifndef RCU_TREE_NONCORE
static const char * const gp_state_names[] = { static const char * const gp_state_names[] = {
"RCU_GP_IDLE", "RCU_GP_IDLE",
"RCU_GP_WAIT_GPS", "RCU_GP_WAIT_GPS",
"RCU_GP_DONE_GPS", "RCU_GP_DONE_GPS",
"RCU_GP_ONOFF",
"RCU_GP_INIT",
"RCU_GP_WAIT_FQS", "RCU_GP_WAIT_FQS",
"RCU_GP_DOING_FQS", "RCU_GP_DOING_FQS",
"RCU_GP_CLEANUP", "RCU_GP_CLEANUP",
...@@ -449,10 +440,13 @@ static bool rcu_preempt_has_tasks(struct rcu_node *rnp); ...@@ -449,10 +440,13 @@ static bool rcu_preempt_has_tasks(struct rcu_node *rnp);
static void rcu_print_detail_task_stall(struct rcu_state *rsp); static void rcu_print_detail_task_stall(struct rcu_state *rsp);
static int rcu_print_task_stall(struct rcu_node *rnp); static int rcu_print_task_stall(struct rcu_node *rnp);
static int rcu_print_task_exp_stall(struct rcu_node *rnp); static int rcu_print_task_exp_stall(struct rcu_node *rnp);
static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp); static void rcu_preempt_check_blocked_tasks(struct rcu_state *rsp,
struct rcu_node *rnp);
static void rcu_preempt_check_callbacks(void); static void rcu_preempt_check_callbacks(void);
void call_rcu(struct rcu_head *head, rcu_callback_t func); void call_rcu(struct rcu_head *head, rcu_callback_t func);
static void __init __rcu_init_preempt(void); static void __init __rcu_init_preempt(void);
static void dump_blkd_tasks(struct rcu_state *rsp, struct rcu_node *rnp,
int ncheck);
static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags); static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags);
static void rcu_preempt_boost_start_gp(struct rcu_node *rnp); static void rcu_preempt_boost_start_gp(struct rcu_node *rnp);
static void invoke_rcu_callbacks_kthread(void); static void invoke_rcu_callbacks_kthread(void);
...@@ -489,7 +483,6 @@ static void __init rcu_spawn_nocb_kthreads(void); ...@@ -489,7 +483,6 @@ static void __init rcu_spawn_nocb_kthreads(void);
#ifdef CONFIG_RCU_NOCB_CPU #ifdef CONFIG_RCU_NOCB_CPU
static void __init rcu_organize_nocb_kthreads(struct rcu_state *rsp); static void __init rcu_organize_nocb_kthreads(struct rcu_state *rsp);
#endif /* #ifdef CONFIG_RCU_NOCB_CPU */ #endif /* #ifdef CONFIG_RCU_NOCB_CPU */
static void __maybe_unused rcu_kick_nohz_cpu(int cpu);
static bool init_nocb_callback_list(struct rcu_data *rdp); static bool init_nocb_callback_list(struct rcu_data *rdp);
static void rcu_bind_gp_kthread(void); static void rcu_bind_gp_kthread(void);
static bool rcu_nohz_full_cpu(struct rcu_state *rsp); static bool rcu_nohz_full_cpu(struct rcu_state *rsp);
......
...@@ -472,6 +472,7 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) ...@@ -472,6 +472,7 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp)
static void sync_rcu_exp_select_cpus(struct rcu_state *rsp, static void sync_rcu_exp_select_cpus(struct rcu_state *rsp,
smp_call_func_t func) smp_call_func_t func)
{ {
int cpu;
struct rcu_node *rnp; struct rcu_node *rnp;
trace_rcu_exp_grace_period(rsp->name, rcu_exp_gp_seq_endval(rsp), TPS("reset")); trace_rcu_exp_grace_period(rsp->name, rcu_exp_gp_seq_endval(rsp), TPS("reset"));
...@@ -486,13 +487,20 @@ static void sync_rcu_exp_select_cpus(struct rcu_state *rsp, ...@@ -486,13 +487,20 @@ static void sync_rcu_exp_select_cpus(struct rcu_state *rsp,
rnp->rew.rew_func = func; rnp->rew.rew_func = func;
rnp->rew.rew_rsp = rsp; rnp->rew.rew_rsp = rsp;
if (!READ_ONCE(rcu_par_gp_wq) || if (!READ_ONCE(rcu_par_gp_wq) ||
rcu_scheduler_active != RCU_SCHEDULER_RUNNING) { rcu_scheduler_active != RCU_SCHEDULER_RUNNING ||
/* No workqueues yet. */ rcu_is_last_leaf_node(rsp, rnp)) {
/* No workqueues yet or last leaf, do direct call. */
sync_rcu_exp_select_node_cpus(&rnp->rew.rew_work); sync_rcu_exp_select_node_cpus(&rnp->rew.rew_work);
continue; continue;
} }
INIT_WORK(&rnp->rew.rew_work, sync_rcu_exp_select_node_cpus); INIT_WORK(&rnp->rew.rew_work, sync_rcu_exp_select_node_cpus);
queue_work_on(rnp->grplo, rcu_par_gp_wq, &rnp->rew.rew_work); preempt_disable();
cpu = cpumask_next(rnp->grplo - 1, cpu_online_mask);
/* If all offline, queue the work on an unbound CPU. */
if (unlikely(cpu > rnp->grphi))
cpu = WORK_CPU_UNBOUND;
queue_work_on(cpu, rcu_par_gp_wq, &rnp->rew.rew_work);
preempt_enable();
rnp->exp_need_flush = true; rnp->exp_need_flush = true;
} }
......
This diff is collapsed.
This diff is collapsed.
...@@ -20,6 +20,9 @@ ...@@ -20,6 +20,9 @@
* Author: Paul E. McKenney <paulmck@us.ibm.com> * Author: Paul E. McKenney <paulmck@us.ibm.com>
* Based on kernel/rcu/torture.c. * Based on kernel/rcu/torture.c.
*/ */
#define pr_fmt(fmt) fmt
#include <linux/types.h> #include <linux/types.h>
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/init.h> #include <linux/init.h>
...@@ -53,7 +56,7 @@ MODULE_LICENSE("GPL"); ...@@ -53,7 +56,7 @@ MODULE_LICENSE("GPL");
MODULE_AUTHOR("Paul E. McKenney <paulmck@us.ibm.com>"); MODULE_AUTHOR("Paul E. McKenney <paulmck@us.ibm.com>");
static char *torture_type; static char *torture_type;
static bool verbose; static int verbose;
/* Mediate rmmod and system shutdown. Concurrent rmmod & shutdown illegal! */ /* Mediate rmmod and system shutdown. Concurrent rmmod & shutdown illegal! */
#define FULLSTOP_DONTSTOP 0 /* Normal operation. */ #define FULLSTOP_DONTSTOP 0 /* Normal operation. */
...@@ -98,7 +101,7 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes, ...@@ -98,7 +101,7 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes,
if (!cpu_online(cpu) || !cpu_is_hotpluggable(cpu)) if (!cpu_online(cpu) || !cpu_is_hotpluggable(cpu))
return false; return false;
if (verbose) if (verbose > 1)
pr_alert("%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"torture_onoff task: offlining %d\n", "torture_onoff task: offlining %d\n",
torture_type, cpu); torture_type, cpu);
...@@ -111,7 +114,7 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes, ...@@ -111,7 +114,7 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes,
"torture_onoff task: offline %d failed: errno %d\n", "torture_onoff task: offline %d failed: errno %d\n",
torture_type, cpu, ret); torture_type, cpu, ret);
} else { } else {
if (verbose) if (verbose > 1)
pr_alert("%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"torture_onoff task: offlined %d\n", "torture_onoff task: offlined %d\n",
torture_type, cpu); torture_type, cpu);
...@@ -147,7 +150,7 @@ bool torture_online(int cpu, long *n_onl_attempts, long *n_onl_successes, ...@@ -147,7 +150,7 @@ bool torture_online(int cpu, long *n_onl_attempts, long *n_onl_successes,
if (cpu_online(cpu) || !cpu_is_hotpluggable(cpu)) if (cpu_online(cpu) || !cpu_is_hotpluggable(cpu))
return false; return false;
if (verbose) if (verbose > 1)
pr_alert("%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"torture_onoff task: onlining %d\n", "torture_onoff task: onlining %d\n",
torture_type, cpu); torture_type, cpu);
...@@ -160,7 +163,7 @@ bool torture_online(int cpu, long *n_onl_attempts, long *n_onl_successes, ...@@ -160,7 +163,7 @@ bool torture_online(int cpu, long *n_onl_attempts, long *n_onl_successes,
"torture_onoff task: online %d failed: errno %d\n", "torture_onoff task: online %d failed: errno %d\n",
torture_type, cpu, ret); torture_type, cpu, ret);
} else { } else {
if (verbose) if (verbose > 1)
pr_alert("%s" TORTURE_FLAG pr_alert("%s" TORTURE_FLAG
"torture_onoff task: onlined %d\n", "torture_onoff task: onlined %d\n",
torture_type, cpu); torture_type, cpu);
...@@ -647,7 +650,7 @@ static void torture_stutter_cleanup(void) ...@@ -647,7 +650,7 @@ static void torture_stutter_cleanup(void)
* The runnable parameter points to a flag that controls whether or not * The runnable parameter points to a flag that controls whether or not
* the test is currently runnable. If there is no such flag, pass in NULL. * the test is currently runnable. If there is no such flag, pass in NULL.
*/ */
bool torture_init_begin(char *ttype, bool v) bool torture_init_begin(char *ttype, int v)
{ {
mutex_lock(&fullstop_mutex); mutex_lock(&fullstop_mutex);
if (torture_type != NULL) { if (torture_type != NULL) {
......
...@@ -70,4 +70,5 @@ else ...@@ -70,4 +70,5 @@ else
else else
print_warning $nclosecalls "Reader Batch close calls in" $(($dur/60)) minute run: $i print_warning $nclosecalls "Reader Batch close calls in" $(($dur/60)) minute run: $i
fi fi
echo $nclosecalls "Reader Batch close calls in" $(($dur/60)) minute run: $i > $i/console.log.rcu.diags
fi fi
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment