Commit 198e2f18 authored by akpm@osdl.org's avatar akpm@osdl.org Committed by Linus Torvalds

[PATCH] scheduler cache-hot-autodetect

)

From: Ingo Molnar <mingo@elte.hu>

This is the latest version of the scheduler cache-hot-auto-tune patch.

The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:

- I've added a 'domain distance' function, which is used to cache
  measurement results. Each distance is only measured once. This means
  that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
  distances 0 and 1, and on SMP distance 0 is measured. The code walks
  the domain tree to determine the distance, so it automatically follows
  whatever hierarchy an architecture sets up. This cuts down on the boot
  time significantly and removes the O(N^2) limit. The only assumption
  is that migration costs can be expressed as a function of domain
  distance - this covers the overwhelming majority of existing systems,
  and is a good guess even for more assymetric systems.

  [ People hacking systems that have assymetries that break this
    assumption (e.g. different CPU speeds) should experiment a bit with
    the cpu_distance() function. Adding a ->migration_distance factor to
    the domain structure would be one possible solution - but lets first
    see the problem systems, if they exist at all. Lets not overdesign. ]

Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:

- Instead of relying on a single cache-size provided by the platform and
  sticking to it, the code now auto-detects the 'effective migration
  cost' between two measured CPUs, via iterating through a wide range of
  cachesizes. The code searches for the maximum migration cost, which
  occurs when the working set of the test-workload falls just below the
  'effective cache size'. I.e. real-life optimized search is done for
  the maximum migration cost, between two real CPUs.

  This, amongst other things, has the positive effect hat if e.g. two
  CPUs share a L2/L3 cache, a different (and accurate) migration cost
  will be found than between two CPUs on the same system that dont share
  any caches.

(The reliable measurement of migration costs is tricky - see the source
for details.)

Furthermore i've added various boot-time options to override/tune
migration behavior.

Firstly, there's a blanket override for autodetection:

	migration_cost=1000,2000,3000

will override the depth 0/1/2 values with 1msec/2msec/3msec values.

Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:

	migration_factor=120

will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.

I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:

Dual Celeron (128K L2 cache):

 ---------------------
 migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
 ---------------------
           [00]    [01]
 [00]:     -     1.7(1)
 [01]:   1.7(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (0) 1.7 (1784008)
 ---------------------

Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.

Dual HT P4 (512K L2 cache):

 ---------------------
 migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
 ---------------------
           [00]    [01]    [02]    [03]
 [00]:     -     0.4(1)  0.0(0)  0.4(1)
 [01]:   0.4(1)    -     0.4(1)  0.0(0)
 [02]:   0.0(0)  0.4(1)    -     0.4(1)
 [03]:   0.4(1)  0.0(0)  0.4(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (33900) 0.4 (448514)
 ---------------------

Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.

8-way P3/Xeon [2MB L2 cache]:

 ---------------------
 migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
 ---------------------
           [00]    [01]    [02]    [03]    [04]    [05]    [06]    [07]
 [00]:     -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [01]:  19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [02]:  19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [03]:  19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [04]:  19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1)
 [05]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1)
 [06]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1)
 [07]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (0) 19.2 (19281756)
 ---------------------

This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
Signed-off-by: default avatarAshok Raj <ashok.raj@intel.com>
Signed-off-by: default avatarKen Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: default avatarJohn Hawkes <hawkes@sgi.com>
Signed-off-by: default avatarAndrew Morton <akpm@osdl.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@osdl.org>
parent 4dc7a0bb
...@@ -856,6 +856,49 @@ running once the system is up. ...@@ -856,6 +856,49 @@ running once the system is up.
mga= [HW,DRM] mga= [HW,DRM]
migration_cost=
[KNL,SMP] debug: override scheduler migration costs
Format: <level-1-usecs>,<level-2-usecs>,...
This debugging option can be used to override the
default scheduler migration cost matrix. The numbers
are indexed by 'CPU domain distance'.
E.g. migration_cost=1000,2000,3000 on an SMT NUMA
box will set up an intra-core migration cost of
1 msec, an inter-core migration cost of 2 msecs,
and an inter-node migration cost of 3 msecs.
WARNING: using the wrong values here can break
scheduler performance, so it's only for scheduler
development purposes, not production environments.
migration_debug=
[KNL,SMP] migration cost auto-detect verbosity
Format=<0|1|2>
If a system's migration matrix reported at bootup
seems erroneous then this option can be used to
increase verbosity of the detection process.
We default to 0 (no extra messages), 1 will print
some more information, and 2 will be really
verbose (probably only useful if you also have a
serial console attached to the system).
migration_factor=
[KNL,SMP] multiply/divide migration costs by a factor
Format=<percent>
This debug option can be used to proportionally
increase or decrease the auto-detected migration
costs for all entries of the migration matrix.
E.g. migration_factor=150 will increase migration
costs by 50%. (and thus the scheduler will be less
eager migrating cache-hot tasks)
migration_factor=80 will decrease migration costs
by 20%. (thus the scheduler will be more eager to
migrate tasks)
WARNING: using the wrong values here can break
scheduler performance, so it's only for scheduler
development purposes, not production environments.
mousedev.tap_time= mousedev.tap_time=
[MOUSE] Maximum time between finger touching and [MOUSE] Maximum time between finger touching and
leaving touchpad surface for touch to be considered leaving touchpad surface for touch to be considered
......
...@@ -1096,6 +1096,7 @@ static void smp_tune_scheduling (void) ...@@ -1096,6 +1096,7 @@ static void smp_tune_scheduling (void)
cachesize = 16; /* Pentiums, 2x8kB cache */ cachesize = 16; /* Pentiums, 2x8kB cache */
bandwidth = 100; bandwidth = 100;
} }
max_cache_size = cachesize * 1024;
} }
} }
......
...@@ -696,6 +696,7 @@ static void ...@@ -696,6 +696,7 @@ static void
get_max_cacheline_size (void) get_max_cacheline_size (void)
{ {
unsigned long line_size, max = 1; unsigned long line_size, max = 1;
unsigned int cache_size = 0;
u64 l, levels, unique_caches; u64 l, levels, unique_caches;
pal_cache_config_info_t cci; pal_cache_config_info_t cci;
s64 status; s64 status;
...@@ -725,6 +726,8 @@ get_max_cacheline_size (void) ...@@ -725,6 +726,8 @@ get_max_cacheline_size (void)
line_size = 1 << cci.pcci_line_size; line_size = 1 << cci.pcci_line_size;
if (line_size > max) if (line_size > max)
max = line_size; max = line_size;
if (cache_size < cci.pcci_cache_size)
cache_size = cci.pcci_cache_size;
if (!cci.pcci_unified) { if (!cci.pcci_unified) {
status = ia64_pal_cache_config_info(l, status = ia64_pal_cache_config_info(l,
/* cache_type (instruction)= */ 1, /* cache_type (instruction)= */ 1,
...@@ -741,6 +744,9 @@ get_max_cacheline_size (void) ...@@ -741,6 +744,9 @@ get_max_cacheline_size (void)
ia64_i_cache_stride_shift = cci.pcci_stride; ia64_i_cache_stride_shift = cci.pcci_stride;
} }
out: out:
#ifdef CONFIG_SMP
max_cache_size = max(max_cache_size, cache_size);
#endif
if (max > ia64_max_cacheline_size) if (max > ia64_max_cacheline_size)
ia64_max_cacheline_size = max; ia64_max_cacheline_size = max;
} }
......
...@@ -99,7 +99,7 @@ static void remove_bpts(void); ...@@ -99,7 +99,7 @@ static void remove_bpts(void);
static void insert_bpts(void); static void insert_bpts(void);
static struct bpt *at_breakpoint(unsigned pc); static struct bpt *at_breakpoint(unsigned pc);
static void bpt_cmds(void); static void bpt_cmds(void);
static void cacheflush(void); void cacheflush(void);
#ifdef CONFIG_SMP #ifdef CONFIG_SMP
static void cpu_cmd(void); static void cpu_cmd(void);
#endif /* CONFIG_SMP */ #endif /* CONFIG_SMP */
......
...@@ -72,7 +72,6 @@ static inline int node_to_first_cpu(int node) ...@@ -72,7 +72,6 @@ static inline int node_to_first_cpu(int node)
.max_interval = 32, \ .max_interval = 32, \
.busy_factor = 32, \ .busy_factor = 32, \
.imbalance_pct = 125, \ .imbalance_pct = 125, \
.cache_hot_time = (10*1000000), \
.cache_nice_tries = 1, \ .cache_nice_tries = 1, \
.busy_idx = 3, \ .busy_idx = 3, \
.idle_idx = 1, \ .idle_idx = 1, \
......
...@@ -55,7 +55,6 @@ void build_cpu_to_node_map(void); ...@@ -55,7 +55,6 @@ void build_cpu_to_node_map(void);
.max_interval = 4, \ .max_interval = 4, \
.busy_factor = 64, \ .busy_factor = 64, \
.imbalance_pct = 125, \ .imbalance_pct = 125, \
.cache_hot_time = (10*1000000), \
.per_cpu_gain = 100, \ .per_cpu_gain = 100, \
.cache_nice_tries = 2, \ .cache_nice_tries = 2, \
.busy_idx = 2, \ .busy_idx = 2, \
...@@ -81,7 +80,6 @@ void build_cpu_to_node_map(void); ...@@ -81,7 +80,6 @@ void build_cpu_to_node_map(void);
.max_interval = 8*(min(num_online_cpus(), 32)), \ .max_interval = 8*(min(num_online_cpus(), 32)), \
.busy_factor = 64, \ .busy_factor = 64, \
.imbalance_pct = 125, \ .imbalance_pct = 125, \
.cache_hot_time = (10*1000000), \
.cache_nice_tries = 2, \ .cache_nice_tries = 2, \
.busy_idx = 3, \ .busy_idx = 3, \
.idle_idx = 2, \ .idle_idx = 2, \
......
...@@ -27,7 +27,6 @@ extern unsigned char __node_distances[MAX_COMPACT_NODES][MAX_COMPACT_NODES]; ...@@ -27,7 +27,6 @@ extern unsigned char __node_distances[MAX_COMPACT_NODES][MAX_COMPACT_NODES];
.max_interval = 32, \ .max_interval = 32, \
.busy_factor = 32, \ .busy_factor = 32, \
.imbalance_pct = 125, \ .imbalance_pct = 125, \
.cache_hot_time = (10*1000), \
.cache_nice_tries = 1, \ .cache_nice_tries = 1, \
.per_cpu_gain = 100, \ .per_cpu_gain = 100, \
.flags = SD_LOAD_BALANCE \ .flags = SD_LOAD_BALANCE \
......
...@@ -39,7 +39,6 @@ static inline int node_to_first_cpu(int node) ...@@ -39,7 +39,6 @@ static inline int node_to_first_cpu(int node)
.max_interval = 32, \ .max_interval = 32, \
.busy_factor = 32, \ .busy_factor = 32, \
.imbalance_pct = 125, \ .imbalance_pct = 125, \
.cache_hot_time = (10*1000000), \
.cache_nice_tries = 1, \ .cache_nice_tries = 1, \
.per_cpu_gain = 100, \ .per_cpu_gain = 100, \
.busy_idx = 3, \ .busy_idx = 3, \
......
...@@ -39,7 +39,6 @@ extern int __node_distance(int, int); ...@@ -39,7 +39,6 @@ extern int __node_distance(int, int);
.max_interval = 32, \ .max_interval = 32, \
.busy_factor = 32, \ .busy_factor = 32, \
.imbalance_pct = 125, \ .imbalance_pct = 125, \
.cache_hot_time = (10*1000000), \
.cache_nice_tries = 2, \ .cache_nice_tries = 2, \
.busy_idx = 3, \ .busy_idx = 3, \
.idle_idx = 2, \ .idle_idx = 2, \
......
...@@ -631,6 +631,13 @@ struct sched_domain { ...@@ -631,6 +631,13 @@ struct sched_domain {
extern void partition_sched_domains(cpumask_t *partition1, extern void partition_sched_domains(cpumask_t *partition1,
cpumask_t *partition2); cpumask_t *partition2);
/*
* Maximum cache size the migration-costs auto-tuning code will
* search from:
*/
extern unsigned int max_cache_size;
#endif /* CONFIG_SMP */ #endif /* CONFIG_SMP */
......
...@@ -86,7 +86,6 @@ ...@@ -86,7 +86,6 @@
.max_interval = 2, \ .max_interval = 2, \
.busy_factor = 8, \ .busy_factor = 8, \
.imbalance_pct = 110, \ .imbalance_pct = 110, \
.cache_hot_time = 0, \
.cache_nice_tries = 0, \ .cache_nice_tries = 0, \
.per_cpu_gain = 25, \ .per_cpu_gain = 25, \
.busy_idx = 0, \ .busy_idx = 0, \
...@@ -117,7 +116,6 @@ ...@@ -117,7 +116,6 @@
.max_interval = 4, \ .max_interval = 4, \
.busy_factor = 64, \ .busy_factor = 64, \
.imbalance_pct = 125, \ .imbalance_pct = 125, \
.cache_hot_time = (5*1000000/2), \
.cache_nice_tries = 1, \ .cache_nice_tries = 1, \
.per_cpu_gain = 100, \ .per_cpu_gain = 100, \
.busy_idx = 2, \ .busy_idx = 2, \
......
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment