Commit bccab6a0 authored by Mike Snitzer's avatar Mike Snitzer

dm cache: switch the "default" cache replacement policy from mq to smq

The Stochastic multiqueue (SMQ) policy (vs MQ) offers the promise of
less memory utilization, improved performance and increased adaptability
in the face of changing workloads.  SMQ also does not have any
cumbersome tuning knobs.

Users may switch from "mq" to "smq" simply by appropriately reloading a
DM table that is using the cache target.  Doing so will cause all of the
mq policy's hints to be dropped.  Also, performance of the cache may
degrade slightly until smq recalculates the origin device's hotspots
that should be cached.

In the future the "mq" policy will just silently make use of "smq" and
the mq code will be removed.
Signed-off-by: default avatarMike Snitzer <snitzer@redhat.com>
Acked-by: default avatarJoe Thornber <ejt@redhat.com>
parent 6096d91a
...@@ -25,10 +25,10 @@ trying to see when the io scheduler has let the ios run. ...@@ -25,10 +25,10 @@ trying to see when the io scheduler has let the ios run.
Overview of supplied cache replacement policies Overview of supplied cache replacement policies
=============================================== ===============================================
multiqueue multiqueue (mq)
---------- ---------------
This policy is the default. This policy has been deprecated in favor of the smq policy (see below).
The multiqueue policy has three sets of 16 queues: one set for entries The multiqueue policy has three sets of 16 queues: one set for entries
waiting for the cache and another two for those in the cache (a set for waiting for the cache and another two for those in the cache (a set for
...@@ -73,6 +73,67 @@ If you're trying to quickly warm a new cache device you may wish to ...@@ -73,6 +73,67 @@ If you're trying to quickly warm a new cache device you may wish to
reduce these to encourage promotion. Remember to switch them back to reduce these to encourage promotion. Remember to switch them back to
their defaults after the cache fills though. their defaults after the cache fills though.
Stochastic multiqueue (smq)
---------------------------
This policy is the default.
The stochastic multi-queue (smq) policy addresses some of the problems
with the multiqueue (mq) policy.
The smq policy (vs mq) offers the promise of less memory utilization,
improved performance and increased adaptability in the face of changing
workloads. SMQ also does not have any cumbersome tuning knobs.
Users may switch from "mq" to "smq" simply by appropriately reloading a
DM table that is using the cache target. Doing so will cause all of the
mq policy's hints to be dropped. Also, performance of the cache may
degrade slightly until smq recalculates the origin device's hotspots
that should be cached.
Memory usage:
The mq policy uses a lot of memory; 88 bytes per cache block on a 64
bit machine.
SMQ uses 28bit indexes to implement it's data structures rather than
pointers. It avoids storing an explicit hit count for each block. It
has a 'hotspot' queue rather than a pre cache which uses a quarter of
the entries (each hotspot block covers a larger area than a single
cache block).
All these mean smq uses ~25bytes per cache block. Still a lot of
memory, but a substantial improvement nontheless.
Level balancing:
MQ places entries in different levels of the multiqueue structures
based on their hit count (~ln(hit count)). This means the bottom
levels generally have the most entries, and the top ones have very
few. Having unbalanced levels like this reduces the efficacy of the
multiqueue.
SMQ does not maintain a hit count, instead it swaps hit entries with
the least recently used entry from the level above. The over all
ordering being a side effect of this stochastic process. With this
scheme we can decide how many entries occupy each multiqueue level,
resulting in better promotion/demotion decisions.
Adaptability:
The MQ policy maintains a hit count for each cache block. For a
different block to get promoted to the cache it's hit count has to
exceed the lowest currently in the cache. This means it can take a
long time for the cache to adapt between varying IO patterns.
Periodically degrading the hit counts could help with this, but I
haven't found a nice general solution.
SMQ doesn't maintain hit counts, so a lot of this problem just goes
away. In addition it tracks performance of the hotspot queue, which
is used to decide which blocks to promote. If the hotspot queue is
performing badly then it starts moving entries more quickly between
levels. This lets it adapt to new IO patterns very quickly.
Performance:
Testing SMQ shows substantially better performance than MQ.
cleaner cleaner
------- -------
......
...@@ -1431,21 +1431,12 @@ static struct dm_cache_policy *mq_create(dm_cblock_t cache_size, ...@@ -1431,21 +1431,12 @@ static struct dm_cache_policy *mq_create(dm_cblock_t cache_size,
static struct dm_cache_policy_type mq_policy_type = { static struct dm_cache_policy_type mq_policy_type = {
.name = "mq", .name = "mq",
.version = {1, 3, 0}, .version = {1, 4, 0},
.hint_size = 4, .hint_size = 4,
.owner = THIS_MODULE, .owner = THIS_MODULE,
.create = mq_create .create = mq_create
}; };
static struct dm_cache_policy_type default_policy_type = {
.name = "default",
.version = {1, 3, 0},
.hint_size = 4,
.owner = THIS_MODULE,
.create = mq_create,
.real = &mq_policy_type
};
static int __init mq_init(void) static int __init mq_init(void)
{ {
int r; int r;
...@@ -1455,36 +1446,21 @@ static int __init mq_init(void) ...@@ -1455,36 +1446,21 @@ static int __init mq_init(void)
__alignof__(struct entry), __alignof__(struct entry),
0, NULL); 0, NULL);
if (!mq_entry_cache) if (!mq_entry_cache)
goto bad; return -ENOMEM;
r = dm_cache_policy_register(&mq_policy_type); r = dm_cache_policy_register(&mq_policy_type);
if (r) { if (r) {
DMERR("register failed %d", r); DMERR("register failed %d", r);
goto bad_register_mq; kmem_cache_destroy(mq_entry_cache);
return -ENOMEM;
} }
r = dm_cache_policy_register(&default_policy_type);
if (!r) {
DMINFO("version %u.%u.%u loaded",
mq_policy_type.version[0],
mq_policy_type.version[1],
mq_policy_type.version[2]);
return 0; return 0;
}
DMERR("register failed (as default) %d", r);
dm_cache_policy_unregister(&mq_policy_type);
bad_register_mq:
kmem_cache_destroy(mq_entry_cache);
bad:
return -ENOMEM;
} }
static void __exit mq_exit(void) static void __exit mq_exit(void)
{ {
dm_cache_policy_unregister(&mq_policy_type); dm_cache_policy_unregister(&mq_policy_type);
dm_cache_policy_unregister(&default_policy_type);
kmem_cache_destroy(mq_entry_cache); kmem_cache_destroy(mq_entry_cache);
} }
......
...@@ -1748,6 +1748,15 @@ static struct dm_cache_policy_type smq_policy_type = { ...@@ -1748,6 +1748,15 @@ static struct dm_cache_policy_type smq_policy_type = {
.create = smq_create .create = smq_create
}; };
static struct dm_cache_policy_type default_policy_type = {
.name = "default",
.version = {1, 0, 0},
.hint_size = 4,
.owner = THIS_MODULE,
.create = smq_create,
.real = &smq_policy_type
};
static int __init smq_init(void) static int __init smq_init(void)
{ {
int r; int r;
...@@ -1758,12 +1767,20 @@ static int __init smq_init(void) ...@@ -1758,12 +1767,20 @@ static int __init smq_init(void)
return -ENOMEM; return -ENOMEM;
} }
r = dm_cache_policy_register(&default_policy_type);
if (r) {
DMERR("register failed (as default) %d", r);
dm_cache_policy_unregister(&smq_policy_type);
return -ENOMEM;
}
return 0; return 0;
} }
static void __exit smq_exit(void) static void __exit smq_exit(void)
{ {
dm_cache_policy_unregister(&smq_policy_type); dm_cache_policy_unregister(&smq_policy_type);
dm_cache_policy_unregister(&default_policy_type);
} }
module_init(smq_init); module_init(smq_init);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment