• Jens Axboe's avatar
    blk-mq: implement new and more efficient tagging scheme · 4bb659b1
    Jens Axboe authored
    blk-mq currently uses percpu_ida for tag allocation. But that only
    works well if the ratio between tag space and number of CPUs is
    sufficiently high. For most devices and systems, that is not the
    case. The end result if that we either only utilize the tag space
    partially, or we end up attempting to fully exhaust it and run
    into lots of lock contention with stealing between CPUs. This is
    not optimal.
    
    This new tagging scheme is a hybrid bitmap allocator. It uses
    two tricks to both be SMP friendly and allow full exhaustion
    of the space:
    
    1) We cache the last allocated (or freed) tag on a per blk-mq
       software context basis. This allows us to limit the space
       we have to search. The key element here is not caching it
       in the shared tag structure, otherwise we end up dirtying
       more shared cache lines on each allocate/free operation.
    
    2) The tag space is split into cache line sized groups, and
       each context will start off randomly in that space. Even up
       to full utilization of the space, this divides the tag users
       efficiently into cache line groups, avoiding dirtying the same
       one both between allocators and between allocator and freeer.
    
    This scheme shows drastically better behaviour, both on small
    tag spaces but on large ones as well. It has been tested extensively
    to show better performance for all the cases blk-mq cares about.
    Signed-off-by: default avatarJens Axboe <axboe@fb.com>
    4bb659b1
blk-mq-tag.c 9.67 KB