• Mel Gorman's avatar
    cpuset: mm: reduce large amounts of memory barrier related damage v3 · cc9a6c87
    Mel Gorman authored
    Commit c0ff7453 ("cpuset,mm: fix no node to alloc memory when
    changing cpuset's mems") wins a super prize for the largest number of
    memory barriers entered into fast paths for one commit.
    
    [get|put]_mems_allowed is incredibly heavy with pairs of full memory
    barriers inserted into a number of hot paths.  This was detected while
    investigating at large page allocator slowdown introduced some time
    after 2.6.32.  The largest portion of this overhead was shown by
    oprofile to be at an mfence introduced by this commit into the page
    allocator hot path.
    
    For extra style points, the commit introduced the use of yield() in an
    implementation of what looks like a spinning mutex.
    
    This patch replaces the full memory barriers on both read and write
    sides with a sequence counter with just read barriers on the fast path
    side.  This is much cheaper on some architectures, including x86.  The
    main bulk of the patch is the retry logic if the nodemask changes in a
    manner that can cause a false failure.
    
    While updating the nodemask, a check is made to see if a false failure
    is a risk.  If it is, the sequence number gets bumped and parallel
    allocators will briefly stall while the nodemask update takes place.
    
    In a page fault test microbenchmark, oprofile samples from
    __alloc_pages_nodemask went from 4.53% of all samples to 1.15%.  The
    actual results were
    
                                 3.3.0-rc3          3.3.0-rc3
                                 rc3-vanilla        nobarrier-v2r1
        Clients   1 UserTime       0.07 (  0.00%)   0.08 (-14.19%)
        Clients   2 UserTime       0.07 (  0.00%)   0.07 (  2.72%)
        Clients   4 UserTime       0.08 (  0.00%)   0.07 (  3.29%)
        Clients   1 SysTime        0.70 (  0.00%)   0.65 (  6.65%)
        Clients   2 SysTime        0.85 (  0.00%)   0.82 (  3.65%)
        Clients   4 SysTime        1.41 (  0.00%)   1.41 (  0.32%)
        Clients   1 WallTime       0.77 (  0.00%)   0.74 (  4.19%)
        Clients   2 WallTime       0.47 (  0.00%)   0.45 (  3.73%)
        Clients   4 WallTime       0.38 (  0.00%)   0.37 (  1.58%)
        Clients   1 Flt/sec/cpu  497620.28 (  0.00%) 520294.53 (  4.56%)
        Clients   2 Flt/sec/cpu  414639.05 (  0.00%) 429882.01 (  3.68%)
        Clients   4 Flt/sec/cpu  257959.16 (  0.00%) 258761.48 (  0.31%)
        Clients   1 Flt/sec      495161.39 (  0.00%) 517292.87 (  4.47%)
        Clients   2 Flt/sec      820325.95 (  0.00%) 850289.77 (  3.65%)
        Clients   4 Flt/sec      1020068.93 (  0.00%) 1022674.06 (  0.26%)
        MMTests Statistics: duration
        Sys Time Running Test (seconds)             135.68    132.17
        User+Sys Time Running Test (seconds)         164.2    160.13
        Total Elapsed Time (seconds)                123.46    120.87
    
    The overall improvement is small but the System CPU time is much
    improved and roughly in correlation to what oprofile reported (these
    performance figures are without profiling so skew is expected).  The
    actual number of page faults is noticeably improved.
    
    For benchmarks like kernel builds, the overall benefit is marginal but
    the system CPU time is slightly reduced.
    
    To test the actual bug the commit fixed I opened two terminals.  The
    first ran within a cpuset and continually ran a small program that
    faulted 100M of anonymous data.  In a second window, the nodemask of the
    cpuset was continually randomised in a loop.
    
    Without the commit, the program would fail every so often (usually
    within 10 seconds) and obviously with the commit everything worked fine.
    With this patch applied, it also worked fine so the fix should be
    functionally equivalent.
    Signed-off-by: default avatarMel Gorman <mgorman@suse.de>
    Cc: Miao Xie <miaox@cn.fujitsu.com>
    Cc: David Rientjes <rientjes@google.com>
    Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
    Cc: Christoph Lameter <cl@linux.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    cc9a6c87
fork.c 43.3 KB