• Alexander Gordeev's avatar
    s390/mm: Uncouple physical vs virtual address spaces · c98d2eca
    Alexander Gordeev authored
    The uncoupling physical vs virtual address spaces brings
    the following benefits to s390:
    
    - virtual memory layout flexibility;
    - closes the address gap between kernel and modules, it
      caused s390-only problems in the past (e.g. 'perf' bugs);
    - allows getting rid of trampolines used for module calls
      into kernel;
    - allows simplifying BPF trampoline;
    - minor performance improvement in branch prediction;
    - kernel randomization entropy is magnitude bigger, as it is
      derived from the amount of available virtual, not physical
      memory;
    
    The whole change could be described in two pictures below:
    before and after the change.
    
    Some aspects of the virtual memory layout setup are not
    clarified (number of page levels, alignment, DMA memory),
    since these are not a part of this change or secondary
    with regard to how the uncoupling itself is implemented.
    
    The focus of the pictures is to explain why __va() and __pa()
    macros are implemented the way they are.
    
            Memory layout in V==R mode:
    
    |    Physical      |    Virtual       |
    +- 0 --------------+- 0 --------------+ identity mapping start
    |                  | S390_lowcore     | Low-address memory
    |                  +- 8 KB -----------+
    |                  |                  |
    |                  | identity         | phys == virt
    |                  | mapping          | virt == phys
    |                  |                  |
    +- AMODE31_START --+- AMODE31_START --+ .amode31 rand. phys/virt start
    |.amode31 text/data|.amode31 text/data|
    +- AMODE31_END ----+- AMODE31_END ----+ .amode31 rand. phys/virt start
    |                  |                  |
    |                  |                  |
    +- __kaslr_offset, __kaslr_offset_phys| kernel rand. phys/virt start
    |                  |                  |
    | kernel text/data | kernel text/data | phys == kvirt
    |                  |                  |
    +------------------+------------------+ kernel phys/virt end
    |                  |                  |
    |                  |                  |
    |                  |                  |
    |                  |                  |
    +- ident_map_size -+- ident_map_size -+ identity mapping end
                       |                  |
                       |  ... unused gap  |
                       |                  |
                       +---- vmemmap -----+ 'struct page' array start
                       |                  |
                       | virtually mapped |
                       | memory map       |
                       |                  |
                       +- __abs_lowcore --+
                       |                  |
                       | Absolute Lowcore |
                       |                  |
                       +- __memcpy_real_area
                       |                  |
                       |  Real Memory Copy|
                       |                  |
                       +- VMALLOC_START --+ vmalloc area start
                       |                  |
                       |  vmalloc area    |
                       |                  |
                       +- MODULES_VADDR --+ modules area start
                       |                  |
                       |  modules area    |
                       |                  |
                       +------------------+ UltraVisor Secure Storage limit
                       |                  |
                       |  ... unused gap  |
                       |                  |
                       +KASAN_SHADOW_START+ KASAN shadow memory start
                       |                  |
                       |   KASAN shadow   |
                       |                  |
                       +------------------+ ASCE limit
    
            Memory layout in V!=R mode:
    
    |    Physical      |    Virtual       |
    +- 0 --------------+- 0 --------------+
    |                  | S390_lowcore     | Low-address memory
    |                  +- 8 KB -----------+
    |                  |                  |
    |                  |                  |
    |                  | ... unused gap   |
    |                  |                  |
    +- AMODE31_START --+- AMODE31_START --+ .amode31 rand. phys/virt start
    |.amode31 text/data|.amode31 text/data|
    +- AMODE31_END ----+- AMODE31_END ----+ .amode31 rand. phys/virt end (<2GB)
    |                  |                  |
    |                  |                  |
    +- __kaslr_offset_phys		     | kernel rand. phys start
    |                  |                  |
    | kernel text/data |                  |
    |                  |                  |
    +------------------+		     | kernel phys end
    |                  |                  |
    |                  |                  |
    |                  |                  |
    |                  |                  |
    +- ident_map_size -+		     |
                       |                  |
                       |  ... unused gap  |
                       |                  |
                       +- __identity_base + identity mapping start (>= 2GB)
                       |                  |
                       | identity         | phys == virt - __identity_base
                       | mapping          | virt == phys + __identity_base
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       |                  |
                       +---- vmemmap -----+ 'struct page' array start
                       |                  |
                       | virtually mapped |
                       | memory map       |
                       |                  |
                       +- __abs_lowcore --+
                       |                  |
                       | Absolute Lowcore |
                       |                  |
                       +- __memcpy_real_area
                       |                  |
                       |  Real Memory Copy|
                       |                  |
                       +- VMALLOC_START --+ vmalloc area start
                       |                  |
                       |  vmalloc area    |
                       |                  |
                       +- MODULES_VADDR --+ modules area start
                       |                  |
                       |  modules area    |
                       |                  |
                       +- __kaslr_offset -+ kernel rand. virt start
                       |                  |
                       | kernel text/data | phys == (kvirt - __kaslr_offset) +
                       |                  |         __kaslr_offset_phys
                       +- kernel .bss end + kernel rand. virt end
                       |                  |
                       |  ... unused gap  |
                       |                  |
                       +------------------+ UltraVisor Secure Storage limit
                       |                  |
                       |  ... unused gap  |
                       |                  |
                       +KASAN_SHADOW_START+ KASAN shadow memory start
                       |                  |
                       |   KASAN shadow   |
                       |                  |
                       +------------------+ ASCE limit
    
    Unused gaps in the virtual memory layout could be present
    or not - depending on how partucular system is configured.
    No page tables are created for the unused gaps.
    
    The relative order of vmalloc, modules and kernel image in
    virtual memory is defined by following considerations:
    
    - start of the modules area and end of the kernel should reside
      within 4GB to accommodate relative 32-bit jumps. The best way
      to achieve that is to place kernel next to modules;
    
    - vmalloc and module areas should locate next to each other
      to prevent failures and extra reworks in user level tools
      (makedumpfile, crash, etc.) which treat vmalloc and module
      addresses similarily;
    
    - kernel needs to be the last area in the virtual memory
      layout to easily distinguish between kernel and non-kernel
      virtual addresses. That is needed to (again) simplify
      handling of addresses in user level tools and make __pa()
      macro faster (see below);
    
    Concluding the above, the relative order of the considered
    virtual areas in memory is: vmalloc - modules - kernel.
    Therefore, the only change to the current memory layout is
    moving kernel to the end of virtual address space.
    
    With that approach the implementation of __pa() macro is
    straightforward - all linear virtual addresses less than
    kernel base are considered identity mapping:
    
    	phys == virt - __identity_base
    
    All addresses greater than kernel base are kernel ones:
    
    	phys == (kvirt - __kaslr_offset) + __kaslr_offset_phys
    
    By contrast, __va() macro deals only with identity mapping
    addresses:
    
    	virt == phys + __identity_base
    
    .amode31 section is mapped separately and is not covered by
    __pa() macro. In fact, it could have been handled easily by
    checking whether a virtual address is within the section or
    not, but there is no need for that. Thus, let __pa() code
    do as little machine cycles as possible.
    
    The KASAN shadow memory is located at the very end of the
    virtual memory layout, at addresses higher than the kernel.
    However, that is not a linear mapping and no code other than
    KASAN instrumentation or API is expected to access it.
    
    When KASLR mode is enabled the kernel base address randomized
    within a memory window that spans whole unused virtual address
    space. The size of that window depends from the amount of
    physical memory available to the system, the limit imposed by
    UltraVisor (if present) and the vmalloc area size as provided
    by vmalloc= kernel command line parameter.
    
    In case the virtual memory is exhausted the minimum size of
    the randomization window is forcefully set to 2GB, which
    amounts to in 15 bits of entropy if KASAN is enabled or 17
    bits of entropy in default configuration.
    
    The default kernel offset 0x100000 is used as a magic value
    both in the decompressor code and vmlinux linker script, but
    it will be removed with a follow-up change.
    Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
    Signed-off-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
    c98d2eca
pgtable.h 55.8 KB