• Andy Lutomirski's avatar
    x86/mm: Implement PCID based optimization: try to preserve old TLB entries using PCID · 10af6235
    Andy Lutomirski authored
    PCID is a "process context ID" -- it's what other architectures call
    an address space ID.  Every non-global TLB entry is tagged with a
    PCID, only TLB entries that match the currently selected PCID are
    used, and we can switch PGDs without flushing the TLB.  x86's
    PCID is 12 bits.
    
    This is an unorthodox approach to using PCID.  x86's PCID is far too
    short to uniquely identify a process, and we can't even really
    uniquely identify a running process because there are monster
    systems with over 4096 CPUs.  To make matters worse, past attempts
    to use all 12 PCID bits have resulted in slowdowns instead of
    speedups.
    
    This patch uses PCID differently.  We use a PCID to identify a
    recently-used mm on a per-cpu basis.  An mm has no fixed PCID
    binding at all; instead, we give it a fresh PCID each time it's
    loaded except in cases where we want to preserve the TLB, in which
    case we reuse a recent value.
    
    Here are some benchmark results, done on a Skylake laptop at 2.3 GHz
    (turbo off, intel_pstate requesting max performance) under KVM with
    the guest using idle=poll (to avoid artifacts when bouncing between
    CPUs).  I haven't done any real statistics here -- I just ran them
    in a loop and picked the fastest results that didn't look like
    outliers.  Unpatched means commit a4eb8b99, so all the
    bookkeeping overhead is gone.
    
    ping-pong between two mms on the same CPU using eventfd:
    
      patched:         1.22µs
      patched, nopcid: 1.33µs
      unpatched:       1.34µs
    
    Same ping-pong, but now touch 512 pages (all zero-page to minimize
    cache misses) each iteration.  dTLB misses are measured by
    dtlb_load_misses.miss_causes_a_walk:
    
      patched:         1.8µs  11M  dTLB misses
      patched, nopcid: 6.2µs, 207M dTLB misses
      unpatched:       6.1µs, 190M dTLB misses
    Signed-off-by: default avatarAndy Lutomirski <luto@kernel.org>
    Reviewed-by: default avatarNadav Amit <nadav.amit@gmail.com>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Arjan van de Ven <arjan@linux.intel.com>
    Cc: Borislav Petkov <bp@alien8.de>
    Cc: Dave Hansen <dave.hansen@intel.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Mel Gorman <mgorman@suse.de>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: linux-mm@kvack.org
    Link: http://lkml.kernel.org/r/9ee75f17a81770feed616358e6860d98a2a5b1e7.1500957502.git.luto@kernel.orgSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
    10af6235
mmu_context.h 8.41 KB