1. 09 Aug, 2019 11 commits
    • Steve Capper's avatar
      arm64: mm: Remove vabits_user · 2c624fe6
      Steve Capper authored
      Previous patches have enabled 52-bit kernel + user VAs and there is no
      longer any scenario where user VA != kernel VA size.
      
      This patch removes the, now redundant, vabits_user variable and replaces
      usage with vabits_actual where appropriate.
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      2c624fe6
    • Steve Capper's avatar
      arm64: mm: Introduce 52-bit Kernel VAs · b6d00d47
      Steve Capper authored
      Most of the machinery is now in place to enable 52-bit kernel VAs that
      are detectable at boot time.
      
      This patch adds a Kconfig option for 52-bit user and kernel addresses
      and plumbs in the requisite CONFIG_ macros as well as sets TCR.T1SZ,
      physvirt_offset and vmemmap at early boot.
      
      To simplify things this patch also removes the 52-bit user/48-bit kernel
      kconfig option.
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      b6d00d47
    • Steve Capper's avatar
      arm64: mm: Modify calculation of VMEMMAP_SIZE · ce3aaed8
      Steve Capper authored
      In a later patch we will need to have a slightly larger VMEMMAP region
      to accommodate boot time selection between 48/52-bit kernel VAs.
      
      This patch modifies the formula for computing VMEMMAP_SIZE to depend
      explicitly on the PAGE_OFFSET and start of kernel addressable memory.
      (This allows for a slightly larger direct linear map in future).
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      ce3aaed8
    • Steve Capper's avatar
      arm64: mm: Separate out vmemmap · c8b6d2cc
      Steve Capper authored
      vmemmap is a preprocessor definition that depends on a variable,
      memstart_addr. In a later patch we will need to expand the size of
      the VMEMMAP region and optionally modify vmemmap depending upon
      whether or not hardware support is available for 52-bit virtual
      addresses.
      
      This patch changes vmemmap to be a variable. As the old definition
      depended on a variable load, this should not affect performance
      noticeably.
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      c8b6d2cc
    • Steve Capper's avatar
      arm64: mm: Logic to make offset_ttbr1 conditional · c812026c
      Steve Capper authored
      When running with a 52-bit userspace VA and a 48-bit kernel VA we offset
      ttbr1_el1 to allow the kernel pagetables with a 52-bit PTRS_PER_PGD to
      be used for both userspace and kernel.
      
      Moving on to a 52-bit kernel VA we no longer require this offset to
      ttbr1_el1 should we be running on a system with HW support for 52-bit
      VAs.
      
      This patch introduces conditional logic to offset_ttbr1 to query
      SYS_ID_AA64MMFR2_EL1 whenever 52-bit VAs are selected. If there is HW
      support for 52-bit VAs then the ttbr1 offset is skipped.
      
      We choose to read a system register rather than vabits_actual because
      offset_ttbr1 can be called in places where the kernel data is not
      actually mapped.
      
      Calls to offset_ttbr1 appear to be made from rarely called code paths so
      this extra logic is not expected to adversely affect performance.
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      c812026c
    • Steve Capper's avatar
      arm64: mm: Introduce vabits_actual · 5383cc6e
      Steve Capper authored
      In order to support 52-bit kernel addresses detectable at boot time, one
      needs to know the actual VA_BITS detected. A new variable vabits_actual
      is introduced in this commit and employed for the KVM hypervisor layout,
      KASAN, fault handling and phys-to/from-virt translation where there
      would normally be compile time constants.
      
      In order to maintain performance in phys_to_virt, another variable
      physvirt_offset is introduced.
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      5383cc6e
    • Steve Capper's avatar
      arm64: mm: Introduce VA_BITS_MIN · 90ec95cd
      Steve Capper authored
      In order to support 52-bit kernel addresses detectable at boot time, the
      kernel needs to know the most conservative VA_BITS possible should it
      need to fall back to this quantity due to lack of hardware support.
      
      A new compile time constant VA_BITS_MIN is introduced in this patch and
      it is employed in the KASAN end address, KASLR, and EFI stub.
      
      For Arm, if 52-bit VA support is unavailable the fallback is to 48-bits.
      
      In other words: VA_BITS_MIN = min (48, VA_BITS)
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      90ec95cd
    • Steve Capper's avatar
      arm64: dump: De-constify VA_START and KASAN_SHADOW_START · 99426e5e
      Steve Capper authored
      The kernel page table dumper assumes that the placement of VA regions is
      constant and determined at compile time. As we are about to introduce
      variable VA logic, we need to be able to determine certain regions at
      boot time.
      
      Specifically the VA_START and KASAN_SHADOW_START will depend on whether
      or not the system is booted with 52-bit kernel VAs.
      
      This patch adds logic to the kernel page table dumper s.t. these regions
      can be computed at boot time.
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      99426e5e
    • Steve Capper's avatar
      arm64: kasan: Switch to using KASAN_SHADOW_OFFSET · 6bd1d0be
      Steve Capper authored
      KASAN_SHADOW_OFFSET is a constant that is supplied to gcc as a command
      line argument and affects the codegen of the inline address sanetiser.
      
      Essentially, for an example memory access:
          *ptr1 = val;
      The compiler will insert logic similar to the below:
          shadowValue = *(ptr1 >> KASAN_SHADOW_SCALE_SHIFT + KASAN_SHADOW_OFFSET)
          if (somethingWrong(shadowValue))
              flagAnError();
      
      This code sequence is inserted into many places, thus
      KASAN_SHADOW_OFFSET is essentially baked into many places in the kernel
      text.
      
      If we want to run a single kernel binary with multiple address spaces,
      then we need to do this with KASAN_SHADOW_OFFSET fixed.
      
      Thankfully, due to the way the KASAN_SHADOW_OFFSET is used to provide
      shadow addresses we know that the end of the shadow region is constant
      w.r.t. VA space size:
          KASAN_SHADOW_END = ~0 >> KASAN_SHADOW_SCALE_SHIFT + KASAN_SHADOW_OFFSET
      
      This means that if we increase the size of the VA space, the start of
      the KASAN region expands into lower addresses whilst the end of the
      KASAN region is fixed.
      
      Currently the arm64 code computes KASAN_SHADOW_OFFSET at build time via
      build scripts with the VA size used as a parameter. (There are build
      time checks in the C code too to ensure that expected values are being
      derived). It is sufficient, and indeed is a simplification, to remove
      the build scripts (and build time checks) entirely and instead provide
      KASAN_SHADOW_OFFSET values.
      
      This patch removes the logic to compute the KASAN_SHADOW_OFFSET in the
      arm64 Makefile, and instead we adopt the approach used by x86 to supply
      offset values in kConfig. To help debug/develop future VA space changes,
      the Makefile logic has been preserved in a script file in the arm64
      Documentation folder.
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      6bd1d0be
    • Steve Capper's avatar
      arm64: mm: Flip kernel VA space · 14c127c9
      Steve Capper authored
      In order to allow for a KASAN shadow that changes size at boot time, one
      must fix the KASAN_SHADOW_END for both 48 & 52-bit VAs and "grow" the
      start address. Also, it is highly desirable to maintain the same
      function addresses in the kernel .text between VA sizes. Both of these
      requirements necessitate us to flip the kernel address space halves s.t.
      the direct linear map occupies the lower addresses.
      
      This patch puts the direct linear map in the lower addresses of the
      kernel VA range and everything else in the higher ranges.
      
      We need to adjust:
       *) KASAN shadow region placement logic,
       *) KASAN_SHADOW_OFFSET computation logic,
       *) virt_to_phys, phys_to_virt checks,
       *) page table dumper.
      
      These are all small changes, that need to take place atomically, so they
      are bundled into this commit.
      
      As part of the re-arrangement, a guard region of 2MB (to preserve
      alignment for fixed map) is added after the vmemmap. Otherwise the
      vmemmap could intersect with IS_ERR pointers.
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      14c127c9
    • Steve Capper's avatar
      arm64: mm: Remove bit-masking optimisations for PAGE_OFFSET and VMEMMAP_START · 9cb1c5dd
      Steve Capper authored
      Currently there are assumptions about the alignment of VMEMMAP_START
      and PAGE_OFFSET that won't be valid after this series is applied.
      
      These assumptions are in the form of bitwise operators being used
      instead of addition and subtraction when calculating addresses.
      
      This patch replaces these bitwise operators with addition/subtraction.
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will@kernel.org>
      9cb1c5dd
  2. 05 Aug, 2019 1 commit
  3. 04 Aug, 2019 10 commits
  4. 03 Aug, 2019 18 commits