• Nishanth Aravamudan's avatar
    page allocator: smarter retry of costly-order allocations · a41f24ea
    Nishanth Aravamudan authored
    Because of page order checks in __alloc_pages(), hugepage (and similarly
    large order) allocations will not retry unless explicitly marked
    __GFP_REPEAT. However, the current retry logic is nearly an infinite
    loop (or until reclaim does no progress whatsoever). For these costly
    allocations, that seems like overkill and could potentially never
    terminate. Mel observed that allowing current __GFP_REPEAT semantics for
    hugepage allocations essentially killed the system. I believe this is
    because we may continue to reclaim small orders of pages all over, but
    never have enough to satisfy the hugepage allocation request. This is
    clearly only a problem for large order allocations, of which hugepages
    are the most obvious (to me).
    
    Modify try_to_free_pages() to indicate how many pages were reclaimed.
    Use that information in __alloc_pages() to eventually fail a large
    __GFP_REPEAT allocation when we've reclaimed an order of pages equal to
    or greater than the allocation's order. This relies on lumpy reclaim
    functioning as advertised. Due to fragmentation, lumpy reclaim may not
    be able to free up the order needed in one invocation, so multiple
    iterations may be requred. In other words, the more fragmented memory
    is, the more retry attempts __GFP_REPEAT will make (particularly for
    higher order allocations).
    
    This changes the semantics of __GFP_REPEAT subtly, but *only* for
    allocations > PAGE_ALLOC_COSTLY_ORDER. With this patch, for those size
    allocations, we will try up to some point (at least 1<<order reclaimed
    pages), rather than forever (which is the case for allocations <=
    PAGE_ALLOC_COSTLY_ORDER).
    
    This change improves the /proc/sys/vm/nr_hugepages interface with a
    follow-on patch that makes pool allocations use __GFP_REPEAT. Rather
    than administrators repeatedly echo'ing a particular value into the
    sysctl, and forcing reclaim into action manually, this change allows for
    the sysctl to attempt a reasonable effort itself. Similarly, dynamic
    pool growth should be more successful under load, as lumpy reclaim can
    try to free up pages, rather than failing right away.
    
    Choosing to reclaim only up to the order of the requested allocation
    strikes a balance between not failing hugepage allocations and returning
    to the caller when it's unlikely to every succeed. Because of lumpy
    reclaim, if we have freed the order requested, hopefully it has been in
    big chunks and those chunks will allow our allocation to succeed. If
    that isn't the case after freeing up the current order, I don't think it
    is likely to succeed in the future, although it is possible given a
    particular fragmentation pattern.
    Signed-off-by: default avatarNishanth Aravamudan <nacc@us.ibm.com>
    Cc: Andy Whitcroft <apw@shadowen.org>
    Tested-by: default avatarMel Gorman <mel@csn.ul.ie>
    Cc: Dave Hansen <haveblue@us.ibm.com>
    Cc: Christoph Lameter <clameter@sgi.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    a41f24ea
vmscan.c 57 KB