1. 11 May, 2022 2 commits
    • Kalesh Singh's avatar
      rcu: Move expedited grace period (GP) work to RT kthread_worker · 9621fbee
      Kalesh Singh authored
      Enabling CONFIG_RCU_BOOST did not reduce RCU expedited grace-period
      latency because its workqueues run at SCHED_OTHER, and thus can be
      delayed by normal processes.  This commit avoids these delays by moving
      the expedited GP work items to a real-time-priority kthread_worker.
      
      This option is controlled by CONFIG_RCU_EXP_KTHREAD and disabled by
      default on PREEMPT_RT=y kernels which disable expedited grace periods
      after boot by unconditionally setting rcupdate.rcu_normal_after_boot=1.
      
      The results were evaluated on arm64 Android devices (6GB ram) running
      5.10 kernel, and capturing trace data in critical user-level code.
      
      The table below shows the resulting order-of-magnitude improvements
      in synchronize_rcu_expedited() latency:
      
      ------------------------------------------------------------------------
      |                          |   workqueues  |  kthread_worker |  Diff   |
      ------------------------------------------------------------------------
      | Count                    |          725  |            688  |         |
      ------------------------------------------------------------------------
      | Min Duration       (ns)  |          326  |            447  |  37.12% |
      ------------------------------------------------------------------------
      | Q1                 (ns)  |       39,428  |         38,971  |  -1.16% |
      ------------------------------------------------------------------------
      | Q2 - Median        (ns)  |       98,225  |         69,743  | -29.00% |
      ------------------------------------------------------------------------
      | Q3                 (ns)  |      342,122  |        126,638  | -62.98% |
      ------------------------------------------------------------------------
      | Max Duration       (ns)  |  372,766,967  |      2,329,671  | -99.38% |
      ------------------------------------------------------------------------
      | Avg Duration       (ns)  |    2,746,353  |        151,242  | -94.49% |
      ------------------------------------------------------------------------
      | Standard Deviation (ns)  |   19,327,765  |        294,408  |         |
      ------------------------------------------------------------------------
      
      The below table show the range of maximums/minimums for
      synchronize_rcu_expedited() latency from all experiments:
      
      ------------------------------------------------------------------------
      |                          |   workqueues  |  kthread_worker |  Diff   |
      ------------------------------------------------------------------------
      | Total No. of Experiments |           25  |             23  |         |
      ------------------------------------------------------------------------
      | Largest  Maximum   (ns)  |  372,766,967  |      2,329,671  | -99.38% |
      ------------------------------------------------------------------------
      | Smallest Maximum   (ns)  |       38,819  |         86,954  | 124.00% |
      ------------------------------------------------------------------------
      | Range of Maximums  (ns)  |  372,728,148  |      2,242,717  |         |
      ------------------------------------------------------------------------
      | Largest  Minimum   (ns)  |       88,623  |         27,588  | -68.87% |
      ------------------------------------------------------------------------
      | Smallest Minimum   (ns)  |          326  |            447  |  37.12% |
      ------------------------------------------------------------------------
      | Range of Minimums  (ns)  |       88,297  |         27,141  |         |
      ------------------------------------------------------------------------
      
      Cc: "Paul E. McKenney" <paulmck@kernel.org>
      Cc: Tejun Heo <tj@kernel.org>
      Reported-by: default avatarTim Murray <timmurray@google.com>
      Reported-by: default avatarWei Wang <wvw@google.com>
      Tested-by: default avatarKyle Lin <kylelin@google.com>
      Tested-by: default avatarChunwei Lu <chunweilu@google.com>
      Tested-by: default avatarLulu Wang <luluw@google.com>
      Signed-off-by: default avatarKalesh Singh <kaleshsingh@google.com>
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      9621fbee
    • Uladzislau Rezki's avatar
      rcu: Introduce CONFIG_RCU_EXP_CPU_STALL_TIMEOUT · 28b3ae42
      Uladzislau Rezki authored
      Currently both expedited and regular grace period stall warnings use
      a single timeout value that with units of seconds.  However, recent
      Android use cases problem require a sub-100-millisecond expedited RCU CPU
      stall warning.  Given that expedited RCU grace periods normally complete
      in far less than a single millisecond, especially for small systems,
      this is not unreasonable.
      
      Therefore introduce the CONFIG_RCU_EXP_CPU_STALL_TIMEOUT kernel
      configuration that defaults to 20 msec on Android and remains the same
      as that of the non-expedited stall warnings otherwise.  It also can be
      changed in run-time via: /sys/.../parameters/rcu_exp_cpu_stall_timeout.
      
      [ paulmck: Default of zero to use CONFIG_RCU_STALL_TIMEOUT. ]
      Signed-off-by: default avatarUladzislau Rezki <uladzislau.rezki@sony.com>
      Signed-off-by: default avatarUladzislau Rezki (Sony) <urezki@gmail.com>
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      28b3ae42
  2. 03 Apr, 2022 8 commits
  3. 02 Apr, 2022 30 commits