• Baolin Wang's avatar
    mm: huge_memory: batch tlb flush when splitting a pte-mapped THP · 3027c6f8
    Baolin Wang authored
    I can observe an obvious tlb flush hotspot when splitting a pte-mapped THP
    on my ARM64 server, and the distribution of this hotspot is as follows:
    
       - 16.85% split_huge_page_to_list
          + 7.80% down_write
          - 7.49% try_to_migrate
             - 7.48% rmap_walk_anon
                  7.23% ptep_clear_flush
          + 1.52% __split_huge_page
    
    The reason is that the split_huge_page_to_list() will build migration
    entries for each subpage of a pte-mapped Anon THP by try_to_migrate(), or
    unmap for file THP, and it will clear and tlb flush for each subpage's
    pte.  Moreover, the split_huge_page_to_list() will set TTU_SPLIT_HUGE_PMD
    flag to ensure the THP is already a pte-mapped THP before splitting it to
    some normal pages.
    
    Actually, there is no need to flush tlb for each subpage immediately,
    instead we can batch tlb flush for the pte-mapped THP to improve the
    performance.
    
    After this patch, we can see the batch tlb flush can improve the latency
    obviously when running thpscale.
    
                                 k6.5-base                   patched
    Amean     fault-both-1      1071.17 (   0.00%)      901.83 *  15.81%*
    Amean     fault-both-3      2386.08 (   0.00%)     1865.32 *  21.82%*
    Amean     fault-both-5      2851.10 (   0.00%)     2273.84 *  20.25%*
    Amean     fault-both-7      3679.91 (   0.00%)     2881.66 *  21.69%*
    Amean     fault-both-12     5916.66 (   0.00%)     4369.55 *  26.15%*
    Amean     fault-both-18     7981.36 (   0.00%)     6303.57 *  21.02%*
    Amean     fault-both-24    10950.79 (   0.00%)     8752.56 *  20.07%*
    Amean     fault-both-30    14077.35 (   0.00%)    10170.01 *  27.76%*
    Amean     fault-both-32    13061.57 (   0.00%)    11630.08 *  10.96%*
    
    Link: https://lkml.kernel.org/r/431d9fb6823036369dcb1d3b2f63732f01df21a7.1698488264.git.baolin.wang@linux.alibaba.comSigned-off-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
    Reviewed-by: default avatar"Huang, Ying" <ying.huang@intel.com>
    Reviewed-by: default avatarYang Shi <shy828301@gmail.com>
    Reviewed-by: default avatarAlistair Popple <apopple@nvidia.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    3027c6f8
huge_memory.c 88.2 KB