• Mike Kravetz's avatar
    hugetlb: skip to end of PT page mapping when pte not present · e95a9851
    Mike Kravetz authored
    Patch series "hugetlb: speed up linear address scanning", v2.
    
    At unmap, fork and remap time hugetlb address ranges are linearly scanned.
    We can optimize these scans if the ranges are sparsely populated.
    
    Also, enable page table "Lazy copy" for hugetlb at fork.
    
    NOTE: Architectures not defining CONFIG_ARCH_WANT_GENERAL_HUGETLB need to
    add an arch specific version hugetlb_mask_last_page() to take advantage of
    sparse address scanning improvements.  Baolin Wang added the routine for
    arm64.  Other architectures which could be optimized are: ia64, mips,
    parisc, powerpc, s390, sh and sparc.
    
    
    This patch (of 4):
    
    HugeTLB address ranges are linearly scanned during fork, unmap and remap
    operations.  If a non-present entry is encountered, the code currently
    continues to the next huge page aligned address.  However, a non-present
    entry implies that the page table page for that entry is not present. 
    Therefore, the linear scan can skip to the end of range mapped by the page
    table page.  This can speed operations on large sparsely populated hugetlb
    mappings.
    
    Create a new routine hugetlb_mask_last_page() that will return an address
    mask.  When the mask is ORed with an address, the result will be the
    address of the last huge page mapped by the associated page table page. 
    Use this mask to update addresses in routines which linearly scan hugetlb
    address ranges when a non-present pte is encountered.
    
    hugetlb_mask_last_page is related to the implementation of huge_pte_offset
    as hugetlb_mask_last_page is called when huge_pte_offset returns NULL. 
    This patch only provides a complete hugetlb_mask_last_page implementation
    when CONFIG_ARCH_WANT_GENERAL_HUGETLB is defined.  Architectures which
    provide their own versions of huge_pte_offset can also provide their own
    version of hugetlb_mask_last_page.
    
    Link: https://lkml.kernel.org/r/20220621235620.291305-1-mike.kravetz@oracle.com
    Link: https://lkml.kernel.org/r/20220621235620.291305-2-mike.kravetz@oracle.comSigned-off-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
    Tested-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
    Reviewed-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
    Acked-by: default avatarMuchun Song <songmuchun@bytedance.com>
    Reported-by: default avatarkernel test robot <lkp@intel.com>
    Cc: Michal Hocko <mhocko@suse.com>
    Cc: Peter Xu <peterx@redhat.com>
    Cc: Naoya Horiguchi <naoya.horiguchi@linux.dev>
    Cc: James Houghton <jthoughton@google.com>
    Cc: Mina Almasry <almasrymina@google.com>
    Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com>
    Cc: Anshuman Khandual <anshuman.khandual@arm.com>
    Cc: Paul Walmsley <paul.walmsley@sifive.com>
    Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
    Cc: Catalin Marinas <catalin.marinas@arm.com>
    Cc: Will Deacon <will@kernel.org>
    Cc: Rolf Eike Beer <eike-kernel@sf-tec.de>
    Cc: David Hildenbrand <david@redhat.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    e95a9851
hugetlb.c 200 KB