• Ryan Roberts's avatar
    arm64/mm: implement new wrprotect_ptes() batch API · 311a6cf2
    Ryan Roberts authored
    Optimize the contpte implementation to fix some of the fork performance
    regression introduced by the initial contpte commit.  Subsequent patches
    will solve it entirely.
    
    During fork(), any private memory in the parent must be write-protected. 
    Previously this was done 1 PTE at a time.  But the core-mm supports
    batched wrprotect via the new wrprotect_ptes() API.  So let's implement
    that API and for fully covered contpte mappings, we no longer need to
    unfold the contpte.  This has 2 benefits:
    
      - reduced unfolding, reduces the number of tlbis that must be issued.
      - The memory remains contpte-mapped ("folded") in the parent, so it
        continues to benefit from the more efficient use of the TLB after
        the fork.
    
    The optimization to wrprotect a whole contpte block without unfolding is
    possible thanks to the tightening of the Arm ARM in respect to the
    definition and behaviour when 'Misprogramming the Contiguous bit'.  See
    section D21194 at https://developer.arm.com/documentation/102105/ja-07/
    
    Link: https://lkml.kernel.org/r/20240215103205.2607016-14-ryan.roberts@arm.comSigned-off-by: default avatarRyan Roberts <ryan.roberts@arm.com>
    Tested-by: default avatarJohn Hubbard <jhubbard@nvidia.com>
    Acked-by: default avatarMark Rutland <mark.rutland@arm.com>
    Acked-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
    Cc: Alistair Popple <apopple@nvidia.com>
    Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
    Cc: Ard Biesheuvel <ardb@kernel.org>
    Cc: Barry Song <21cnbao@gmail.com>
    Cc: Borislav Petkov (AMD) <bp@alien8.de>
    Cc: Dave Hansen <dave.hansen@linux.intel.com>
    Cc: David Hildenbrand <david@redhat.com>
    Cc: "H. Peter Anvin" <hpa@zytor.com>
    Cc: Ingo Molnar <mingo@redhat.com>
    Cc: James Morse <james.morse@arm.com>
    Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
    Cc: Marc Zyngier <maz@kernel.org>
    Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Will Deacon <will@kernel.org>
    Cc: Yang Shi <shy828301@gmail.com>
    Cc: Zi Yan <ziy@nvidia.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    311a6cf2
pgtable.h 39.1 KB