- 20 Jul, 2018 4 commits
-
-
Joerg Roedel authored
The SWITCH_TO_KERNEL_STACK macro only checks for CPL == 0 to go down the slow and paranoid entry path. The problem is that this check also returns true when coming from VM86 mode. This is not a problem by itself, as the paranoid path handles VM86 stack-frames just fine, but it is not necessary as the normal code path handles VM86 mode as well (and faster). Extend the check to include VM86 mode. This also makes an optimization of the paranoid path possible. Signed-off-by:
Joerg Roedel <jroedel@suse.de> Signed-off-by:
Thomas Gleixner <tglx@linutronix.de> Cc: "H . Peter Anvin" <hpa@zytor.com> Cc: linux-mm@kvack.org Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Andy Lutomirski <luto@kernel.org> Cc: Dave Hansen <dave.hansen@intel.com> Cc: Josh Poimboeuf <jpoimboe@redhat.com> Cc: Juergen Gross <jgross@suse.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Jiri Kosina <jkosina@suse.cz> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com> Cc: Brian Gerst <brgerst@gmail.com> Cc: David Laight <David.Laight@aculab.com> Cc: Denys Vlasenko <dvlasenk@redhat.com> Cc: Eduardo Valentin <eduval@amazon.com> Cc: Greg KH <gregkh@linuxfoundation.org> Cc: Will Deacon <will.deacon@arm.com> Cc: aliguori@amazon.com Cc: daniel.gruss@iaik.tugraz.at Cc: hughd@google.com Cc: keescook@google.com Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: Waiman Long <llong@redhat.com> Cc: Pavel Machek <pavel@ucw.cz> Cc: "David H . Gutteridge" <dhgutteridge@sympatico.ca> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: joro@8bytes.org Link: https://lkml.kernel.org/r/1532103744-31902-3-git-send-email-joro@8bytes.org
-
Joerg Roedel authored
The ring-buffer is accessed in the NMI handler, so it's better to avoid faulting on it. Sync the vmalloc range with all page-tables in system to make sure everyone has it mapped. This fixes a WARN_ON_ONCE() that can be triggered with PTI enabled on x86-32: WARNING: CPU: 4 PID: 0 at arch/x86/mm/fault.c:320 vmalloc_fault+0x220/0x230 This triggers because with PTI enabled on an PAE kernel the PMDs are no longer shared between the page-tables, so the vmalloc changes do not propagate automatically. Note: Andy said rightfully that we should try to fix the vmalloc code for that case, but that's not a hot fix for the issue at hand. Fixes: 7757d607 ("x86/pti: Allow CONFIG_PAGE_TABLE_ISOLATION for x86_32") Signed-off-by:
-
Jiang Biao authored
pti_user_pagetable_walk_pmd() can return NULL, so the return value should be checked to prevent a NULL pointer dereference. Add the check and a warning when the PMD allocation fails. Signed-off-by:
Jiang Biao <jiang.biao2@zte.com.cn> Signed-off-by:
-
Jiang Biao authored
pti_user_pagetable_walk_p4d() can return NULL, so the return value should be checked to prevent a NULL pointer dereference. Add the check and a warning when the P4D allocation fails. Signed-off-by:
-
- 19 Jul, 2018 36 commits
-
-
Joerg Roedel authored
Add code to check whether the kernel is entered and left with the correct CR3 and make it depend on CONFIG_DEBUG_ENTRY. This is needed because there is no NX protection of user-addresses in the kernel-CR3 on x86-32 and that type of bug would not be detected otherwise. Signed-off-by:
Pavel Machek <pavel@ucw.cz> Cc: "H . Peter Anvin" <hpa@zytor.com> Cc: linux-mm@kvack.org Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Andy Lutomirski <luto@kernel.org> Cc: Dave Hansen <dave.hansen@intel.com> Cc: Josh Poimboeuf <jpoimboe@redhat.com> Cc: Juergen Gross <jgross@suse.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Jiri Kosina <jkosina@suse.cz> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com> Cc: Brian Gerst <brgerst@gmail.com> Cc: David Laight <David.Laight@aculab.com> Cc: Denys Vlasenko <dvlasenk@redhat.com> Cc: Eduardo Valentin <eduval@amazon.com> Cc: Greg KH <gregkh@linuxfoundation.org> Cc: Will Deacon <will.deacon@arm.com> Cc: aliguori@amazon.com Cc: daniel.gruss@iaik.tugraz.at Cc: hughd@google.com Cc: keescook@google.com Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: Waiman Long <llong@redhat.com> Cc: "David H . Gutteridge" <dhgutteridge@sympatico.ca> Cc: joro@8bytes.org Link: https://lkml.kernel.org/r/1531906876-13451-40-git-send-email-joro@8bytes.org
-
Joerg Roedel authored
Warn the user in case the performance can be significantly improved by switching to a 64-bit kernel. Suggested-by:
Andy Lutomirski <luto@kernel.org> Signed-off-by:
-
Joerg Roedel authored
Allow PTI to be compiled on x86_32. Signed-off-by:
-
Joerg Roedel authored
This adds the needed special case for PAE to get the LDT mapped into the user page-table when PTI is enabled. The big difference to the other paging modes is that on PAE there is no full top-level PGD entry available for the LDT, but only a PMD entry. Signed-off-by:
-
Joerg Roedel authored
This splits out the mapping sanity check and the actual mapping of the LDT to user-space from the map_ldt_struct() function in a way so that it is re-usable for PAE paging. Signed-off-by:
-
Joerg Roedel authored
It marks the end of the address-space range reserved for the LDT. The LDT-code will use it when unmapping the LDT for user-space. Signed-off-by:
-
Joerg Roedel authored
Reserve 2MB/4MB of address-space for mapping the LDT to user-space on 32 bit PTI kernels. Signed-off-by:
-
Joerg Roedel authored
When PTI is enabled, separate kernel PMDs in the user page-table are required to map the per-process LDT for user-space. Signed-off-by:
-
Joerg Roedel authored
Define INIT_PGD to point to the correct initial page-table for 32 and 64 bit and use it where needed. This fixes the build on 32 bit with CONFIG_PAGE_TABLE_ISOLATION enabled. Signed-off-by:
-
Joerg Roedel authored
The mapping for entry-text might have changed in the kernel after it was cloned to the user page-table. Clone again to update the user page-table to bring the mapping in sync with the kernel again. Signed-off-by:
-
Joerg Roedel authored
Introduce a new function to finalize the kernel mappings for the userspace page-table after all ro/nx protections have been applied to the kernel mappings. Also move the call to pti_clone_kernel_text() to that function so that it will run on 32 bit kernels too. Signed-off-by:
-
Joerg Roedel authored
Mapping the kernel text area to user-space makes only sense if it has the same permissions as in the kernel page-table. If permissions are different this will cause a TLB reload when using the kernel page-table, which is as good as not mapping it at all. On 64-bit kernels this patch makes no difference, as the whole range cloned by pti_clone_kernel_text() is mapped RO anyway. On 32 bit there are writeable mappings in the range, so just keep the permissions as they are. Signed-off-by:
-
Joerg Roedel authored
The pti_clone_kernel_text() function references __end_rodata_hpage_align, which is only present on x86-64. This makes sense as the end of the rodata section is not huge-page aligned on 32 bit. Nevertheless a symbol is required for the function that points at the right address for both 32 and 64 bit. Introduce __end_rodata_aligned for that purpose and use it in pti_clone_kernel_text(). Signed-off-by:
-
Joerg Roedel authored
Cloning on the P4D level would clone the complete kernel address space into the user-space page-tables for PAE kernels. Cloning on PMD level is fine for PAE and legacy paging. Signed-off-by:
-
Joerg Roedel authored
Move it out of the X86_64 specific processor defines so that its visible for 32bit too. Signed-off-by:
-
Joerg Roedel authored
The addr counter will overflow if the last PMD of the address space is cloned, resulting in an endless loop. Check for that and bail out of the loop when it happens. Signed-off-by:
-
Joerg Roedel authored
Also populate the user-spage pgd's in the user page-table. Signed-off-by:
-
Joerg Roedel authored
When a PGD entry is populated, make sure to populate it in the user page-table too. Signed-off-by:
-
Joerg Roedel authored
Generic page-table code populates all non-leaf entries with _KERNPG_TABLE bits set. This is fine for all paging modes except PAE. In PAE mode only a subset of the bits is allowed to be set. Make sure to only set allowed bits by masking out the reserved bits. Signed-off-by:
-
Joerg Roedel authored
These two functions are required for PTI on 32 bit: * pgdp_maps_userspace() * pgd_large() Also re-implement pgdp_maps_userspace() so that it will work on 64 and 32 bit kernels. Signed-off-by:
-
Joerg Roedel authored
There it is also usable from 32 bit code. Signed-off-by:
-
Joerg Roedel authored
Make them available on 32 bit and clone_pgd_range() happy. Signed-off-by:
-
Joerg Roedel authored
Allocate a kernel and a user page-table root when PTI is enabled. Also allocate a full page per root for PAE because otherwise the bit to flip in CR3 to switch between them would be non-constant, which creates a lot of hassle. Keep that for a later optimization. Signed-off-by:
-
Joerg Roedel authored
With PTI the per-process LDT must be mapped into the kernel address-space for each process, which requires separate kernel PMDs per PGD. Signed-off-by:
-
Joerg Roedel authored
The way page-table folding is implemented on 32 bit, these functions are not only setting, but also PUDs and even PMDs. Give the function a more generic name to reflect that. Signed-off-by:
-
Joerg Roedel authored
The function does not update sp0 anymore but updates makes the task-stack visible for entry code. This is by either writing it to sp1 or by doing a hypercall. Rename the function to get rid of the misleading name. Signed-off-by:
-
Joerg Roedel authored
The NMI handler is special, as it needs to leave with the same CR3 as it was entered with. This is required because the NMI can happen within kernel context but with user CR3 already loaded, i.e. after switching to user CR3 but before returning to user space. Signed-off-by:
-
Joerg Roedel authored
Add unconditional cr3 switches between user and kernel cr3 to all non-NMI entry and exit points. Signed-off-by:
-
Joerg Roedel authored
The common exception entry code now handles the entry-from-sysenter stack situation and makes sure to leave with the same stack as it entered the kernel. So there is no need anymore for the special handling in the debug entry code. Signed-off-by:
-
Joerg Roedel authored
It is possible that the kernel is entered from kernel-mode and on the entry-stack. The most common way this happens is when an exception is triggered while loading the user-space segment registers on the kernel-to-userspace exit path. The segment loading needs to be done after the entry-stack switch, because the stack-switch needs kernel %fs for per_cpu access. When this happens, make sure to leave the kernel with the entry-stack again, so that the interrupted code-path runs on the right stack when switching to the user-cr3. Detect this condition on kernel-entry by checking CS.RPL and %esp, and if it happens, copy over the complete content of the entry stack to the task-stack. This needs to be done because once the exception handler is entereed, the task might be scheduled out or even migrated to a different CPU, so this cannot rely on the entry-stack contents. Leave a marker in the stack-frame to detect this condition on the exit path. On the exit path the copy is reversed, copy all of the remaining task-stack back to the entry-stack and switch to it. Signed-off-by:
-
Joerg Roedel authored
These macros will be used in the NMI handler code and replace plain SAVE_ALL and RESTORE_REGS there. The NMI-specific CR3-switch will be added to these macros later. Signed-off-by:
-
Joerg Roedel authored
Switch back to the trampoline stack before returning to userspace. Signed-off-by:
-
Joerg Roedel authored
Use the entry-stack as a trampoline to enter the kernel. The entry-stack is already in the cpu_entry_area and will be mapped to userspace when PTI is enabled. Signed-off-by:
-
Joerg Roedel authored
Use a separate return path when returning to the kernel. This allows to put the PTI cr3-switch and the switch to the entry-stack into the return-to-user path without further checking. Signed-off-by:
-
Joerg Roedel authored
NMI will no longer use most of the shared return path, because NMI needs special handling when the CR3 switches for PTI are added. Prepare for that change. Signed-off-by:
-
Joerg Roedel authored
This makes it easier to split up the shared iret code path. Signed-off-by:
-
