• Kirill A. Shutemov's avatar
    thp: implement refcounting for huge zero page · 97ae1749
    Kirill A. Shutemov authored
    H.  Peter Anvin doesn't like huge zero page which sticks in memory forever
    after the first allocation.  Here's implementation of lockless refcounting
    for huge zero page.
    
    We have two basic primitives: {get,put}_huge_zero_page(). They
    manipulate reference counter.
    
    If counter is 0, get_huge_zero_page() allocates a new huge page and takes
    two references: one for caller and one for shrinker.  We free the page
    only in shrinker callback if counter is 1 (only shrinker has the
    reference).
    
    put_huge_zero_page() only decrements counter.  Counter is never zero in
    put_huge_zero_page() since shrinker holds on reference.
    
    Freeing huge zero page in shrinker callback helps to avoid frequent
    allocate-free.
    
    Refcounting has cost.  On 4 socket machine I observe ~1% slowdown on
    parallel (40 processes) read page faulting comparing to lazy huge page
    allocation.  I think it's pretty reasonable for synthetic benchmark.
    
    [lliubbo@gmail.com: fix mismerge]
    Signed-off-by: default avatarKirill A. Shutemov <kirill.shutemov@linux.intel.com>
    Cc: Andrea Arcangeli <aarcange@redhat.com>
    Cc: Andi Kleen <ak@linux.intel.com>
    Cc: "H. Peter Anvin" <hpa@linux.intel.com>
    Cc: Mel Gorman <mel@csn.ul.ie>
    Cc: David Rientjes <rientjes@google.com>
    Signed-off-by: default avatarBob Liu <lliubbo@gmail.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    97ae1749
huge_memory.c 69.9 KB