• Chengming Zhou's avatar
    slub: Delay freezing of partial slabs · 8cd3fa42
    Chengming Zhou authored
    Now we will freeze slabs when moving them out of node partial list to
    cpu partial list, this method needs two cmpxchg_double operations:
    
    1. freeze slab (acquire_slab()) under the node list_lock
    2. get_freelist() when pick used in ___slab_alloc()
    
    Actually we don't need to freeze when moving slabs out of node partial
    list, we can delay freezing to when use slab freelist in ___slab_alloc(),
    so we can save one cmpxchg_double().
    
    And there are other good points:
     - The moving of slabs between node partial list and cpu partial list
       becomes simpler, since we don't need to freeze or unfreeze at all.
    
     - The node list_lock contention would be less, since we don't need to
       freeze any slab under the node list_lock.
    
    We can achieve this because there is no concurrent path would manipulate
    the partial slab list except the __slab_free() path, which is now
    serialized by slab_test_node_partial() under the list_lock.
    
    Since the slab returned by get_partial() interfaces is not frozen anymore
    and no freelist is returned in the partial_context, so we need to use the
    introduced freeze_slab() to freeze it and get its freelist.
    
    Similarly, the slabs on the CPU partial list are not frozen anymore,
    we need to freeze_slab() on it before use.
    
    We can now delete acquire_slab() as it became unused.
    Signed-off-by: default avatarChengming Zhou <zhouchengming@bytedance.com>
    Reviewed-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Tested-by: default avatarHyeonggon Yoo <42.hyeyoo@gmail.com>
    Reviewed-by: default avatarHyeonggon Yoo <42.hyeyoo@gmail.com>
    Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
    8cd3fa42
slub.c 161 KB