- 29 Mar, 2021 40 commits
-
-
Christophe Leroy authored
Until now, non volatile registers were restored everytime they were saved, ie using EXC_XFER_STD meant saving and restoring them while EXC_XFER_LITE meant neither saving not restoring them. Now that they are always saved, EXC_XFER_STD means to restore them and EXC_XFER_LITE means to not restore them. Most of the users of EXC_XFER_STD only need to retrieve the non volatile registers. For them there is no need to restore the non volatile registers as they have not been modified. Only very few exceptions require non volatile registers restore. Opencode the few places which require saving of non volatile registers. Signed-off-by:
Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by:
Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/d1cb12d8023cc6afc1f07150565571373c04945c.1615552867.git.christophe.leroy@csgroup.eu
-
Christophe Leroy authored
In order to increase flexibility, add a macro that will for now call transfer_to_handler. As transfer_to_handler doesn't do the actual transfer anymore, also name it prepare_transfer_to_handler. The following patches will progressively remove the use of transfer_to_handler label. Signed-off-by:
-
Christophe Leroy authored
Saving the trap number into the stack goes into the exception prolog, as EXC_XFER_xxx will soon disappear. Signed-off-by:
-
Christophe Leroy authored
Now that non volatile registers are saved at all time, no need to split bad_page_fault() out of do_page_fault(). Remove handle_page_fault() and use do_page_fault() directly. Signed-off-by:
-
Christophe Leroy authored
All exception handlers take regs as first parameter. Instead of setting r3 just before each call to a handler, set it in transfer_to_handler. Signed-off-by:
-
Christophe Leroy authored
Since commit 06d67d54 ("powerpc: make process.c suitable for both 32-bit and 64-bit"), thread.regs is set on task creation, no need to set it again and again at each interrupt entry as it never change. Suggested-by:
Nicholas Piggin <npiggin@gmail.com> Signed-off-by:
-
Christophe Leroy authored
This patch replaces the PPC32 ASM exception exit by C exception exit. Signed-off-by:
-
Christophe Leroy authored
In preparation of handling exception entry and exit in C, in order to simplify the handling, always save non volatile registers when entering an exception. Signed-off-by:
-
Christophe Leroy authored
Now that the MMU is re-enabled before calling the transfer function, we don't need anymore that hack with the address of the handler and the return function sitting just after the 'bl' to the transfer fonction, that function is retrieving via a read relative to 'lr'. Do a regular call to the transfer function, then to the handler, then branch to the return function. Signed-off-by:
-
Christophe Leroy authored
Refactor booke critical registers saving into a few macros and move it into the exception prolog directly. Keep the dedicated transfert_to_handler entry point for the moment allthough they are empty. They will be removed in a later patch to reduce churn. Signed-off-by:
-
Christophe Leroy authored
Now that the prolog continuation is separated in .text, give it a name and mark it _ASM_NOKPROBE_SYMBOL. Signed-off-by:
-
Christophe Leroy authored
The space in the head section is rather constrained by the fact that exception vectors are spread every 0x100 bytes and sometimes we need to have "out of line" code because it doesn't fit. Now that we are enabling MMU early in the prolog, take that opportunity to jump somewhere else in the .text section where we don't have any space constraint. Signed-off-by:
-
Christophe Leroy authored
Everywhere where it is possible, use START_EXCEPTION(). This will help for proper exception init in future patches. Signed-off-by:
-
Christophe Leroy authored
For consistency, add in the macro the label used by exception prolog to branch to stack overflow processing. While at it, enclose the macro in #ifdef CONFIG_VMAP_STACK on the 8xx as already done on book3s/32. Signed-off-by:
-
Christophe Leroy authored
The check of the emergency context initialisation in vmap_stack_overflow is buggy for the SMP case, as it compares r1 with 0 while in the SMP case r1 is offseted by the CPU id. Instead of fixing it, just perform static initialisation of the first emergency context. Signed-off-by:
-
Christophe Leroy authored
On 40x and 8xx, kernel text is pinned. On book3s/32, kernel text is mapped by BATs. Enable instruction translation at the same time as data translation, it makes things simpler. In syscall handler, MSR_RI can also be set at the same time because srr0/srr1 are already saved and r1 is set properly. On booke, translation is always on, so at the end all PPC32 have translation on early. Just update msr. Also update comment in power_save_ppc32_restore(). Signed-off-by:
-
Christophe Leroy authored
8xx requires to tag the DAR with a magic value in order to fixup DAR on faults generated by 'dcbX', as the 8xx forgets to update the DAR for those faults. Do the tagging as early as possible, that is before enabling MMU. Signed-off-by:
-
Christophe Leroy authored
If the code can use a stack in vm area, it can also use a stack in linear space. Simplify code by removing old non VMAP stack code on PPC32. That means the data translation is now re-enabled early in exception prolog in all cases, not only when using VMAP stacks. While we are touching EXCEPTION_PROLOG macros, remove the unused for_rtas parameter in EXCEPTION_PROLOG_1. Signed-off-by:
-
Christophe Leroy authored
ksp_limit is there to help detect stack overflows. That is specific to ppc32 as it was removed from ppc64 in commit cbc9565e ("powerpc: Remove ksp_limit on ppc64"). There are other means for detecting stack overflows. As ppc64 has proven to not need it, ppc32 should be able to do without it too. Lets remove it and simplify exception handling. Signed-off-by:
-
Christophe Leroy authored
8xx has registers SPRN_NRI, SPRN_EID and SPRN_EIE for changing MSR EE and RI. Use SPRN_EID in exception prolog to set RI. On an 8xx, it reduces the null_syscall test by 3 cycles. Signed-off-by:
-
Christophe Leroy authored
The handling of SPRN_DBCR0 and other registers can easily be done in C instead of ASM. Signed-off-by:
-
Christophe Leroy authored
There is no need for this to be in asm, use the new interrupt entry wrapper. Signed-off-by:
-
Christophe Leroy authored
There is no need for this to be in asm anymore, use the new interrupt entry wrapper. Signed-off-by:
-
Christophe Leroy authored
Ensure normal exception handler are able to manage stuff with MMU enabled. For that we use CONFIG_VMAP_STACK related code allthough there is no intention to really activate CONFIG_VMAP_STACK on powerpc 40x for the moment. 40x uses SPRN_DEAR instead of SPRN_DAR and SPRN_ESR instead of SPRN_DSISR. Take it into account in common macros. 40x MSR value doesn't fit on 15 bits, use LOAD_REG_IMMEDIATE() in common macros that will be used also with 40x. Signed-off-by:
-
Christophe Leroy authored
In order the enable MMU early in exception prolog, implement CONFIG_VMAP_STACK principles in critical exception prolog. There is no intention to use CONFIG_VMAP_STACK on 40x, but related code will be used to enable MMU early in exception in a later patch. Also address (critirq_ctx - PAGE_OFFSET) directly instead of using tophys() in order to win one instruction. Signed-off-by:
-
Christophe Leroy authored
In order to ease preparation for CONFIG_VMAP_STACK, reorder a few instruction, especially save r1 into stack frame earlier. Signed-off-by:
-
Christophe Leroy authored
In order to be able to switch MMU on in exception prolog, save SRR0 and SRR1 earlier. Also save r10 and r11 into stack earlier to better match with the normal exception prolog. Signed-off-by:
-
Christophe Leroy authored
Change CRITICAL_EXCEPTION_PROLOG macro to a gas macro to remove the ugly ; and \ on each line. Signed-off-by:
-
Christophe Leroy authored
SPRN_SPRG_SCRATCH5 is used to save SPRN_PID. SPRN_SPRG_SCRATCH6 is already available. SPRN_PID is only 8 bits. We have r12 that contains CR. We only need to preserve CR0, so we have space available in r12 to save PID. Keep PID in r12 and free up SPRN_SPRG_SCRATCH5. Then In TLB miss handlers, instead of using SPRN_SPRG_SCRATCH0 and SPRN_SPRG_SCRATCH1, use SPRN_SPRG_SCRATCH5 and SPRN_SPRG_SCRATCH6 to avoid future conflicts with normal exception prologs. Signed-off-by:
-
Christophe Leroy authored
unrecoverable_exception() is never expected to return, most callers have an infiniteloop in case it returns. Ensure it really never returns by terminating it with a BUG(), and declare it __no_return. It always GCC to really simplify functions calling it. In the exemple below, it avoids the stack frame in the likely fast path and avoids code duplication for the exit. With this patch: 00000348 <interrupt_exit_kernel_prepare>: 348: 81 43 00 84 lwz r10,132(r3) 34c: 71 48 00 02 andi. r8,r10,2 350: 41 82 00 2c beq 37c <interrupt_exit_kernel_prepare+0x34> 354: 71 4a 40 00 andi. r10,r10,16384 358: 40 82 00 20 bne 378 <interrupt_exit_kernel_prepare+0x30> 35c: 80 62 00 70 lwz r3,112(r2) 360: 74 63 00 01 andis. r3,r3,1 364: 40 82 00 28 bne 38c <interrupt_exit_kernel_prepare+0x44> 368: 7d 40 00 a6 mfmsr r10 36c: 7c 11 13 a6 mtspr 81,r0 370: 7c 12 13 a6 mtspr 82,r0 374: 4e 80 00 20 blr 378: 48 00 00 00 b 378 <interrupt_exit_kernel_prepare+0x30> 37c: 94 21 ff f0 stwu r1,-16(r1) 380: 7c 08 02 a6 mflr r0 384: 90 01 00 14 stw r0,20(r1) 388: 48 00 00 01 bl 388 <interrupt_exit_kernel_prepare+0x40> 388: R_PPC_REL24 unrecoverable_exception 38c: 38 e2 00 70 addi r7,r2,112 390: 3d 00 00 01 lis r8,1 394: 7c c0 38 28 lwarx r6,0,r7 398: 7c c6 40 78 andc r6,r6,r8 39c: 7c c0 39 2d stwcx. r6,0,r7 3a0: 40 a2 ff f4 bne 394 <interrupt_exit_kernel_prepare+0x4c> 3a4: 38 60 00 01 li r3,1 3a8: 4b ff ff c0 b 368 <interrupt_exit_kernel_prepare+0x20> Without this patch: 00000348 <interrupt_exit_kernel_prepare>: 348: 94 21 ff f0 stwu r1,-16(r1) 34c: 93 e1 00 0c stw r31,12(r1) 350: 7c 7f 1b 78 mr r31,r3 354: 81 23 00 84 lwz r9,132(r3) 358: 71 2a 00 02 andi. r10,r9,2 35c: 41 82 00 34 beq 390 <interrupt_exit_kernel_prepare+0x48> 360: 71 29 40 00 andi. r9,r9,16384 364: 40 82 00 28 bne 38c <interrupt_exit_kernel_prepare+0x44> 368: 80 62 00 70 lwz r3,112(r2) 36c: 74 63 00 01 andis. r3,r3,1 370: 40 82 00 3c bne 3ac <interrupt_exit_kernel_prepare+0x64> 374: 7d 20 00 a6 mfmsr r9 378: 7c 11 13 a6 mtspr 81,r0 37c: 7c 12 13 a6 mtspr 82,r0 380: 83 e1 00 0c lwz r31,12(r1) 384: 38 21 00 10 addi r1,r1,16 388: 4e 80 00 20 blr 38c: 48 00 00 00 b 38c <interrupt_exit_kernel_prepare+0x44> 390: 7c 08 02 a6 mflr r0 394: 90 01 00 14 stw r0,20(r1) 398: 48 00 00 01 bl 398 <interrupt_exit_kernel_prepare+0x50> 398: R_PPC_REL24 unrecoverable_exception 39c: 80 01 00 14 lwz r0,20(r1) 3a0: 81 3f 00 84 lwz r9,132(r31) 3a4: 7c 08 03 a6 mtlr r0 3a8: 4b ff ff b8 b 360 <interrupt_exit_kernel_prepare+0x18> 3ac: 39 02 00 70 addi r8,r2,112 3b0: 3d 40 00 01 lis r10,1 3b4: 7c e0 40 28 lwarx r7,0,r8 3b8: 7c e7 50 78 andc r7,r7,r10 3bc: 7c e0 41 2d stwcx. r7,0,r8 3c0: 40 a2 ff f4 bne 3b4 <interrupt_exit_kernel_prepare+0x6c> 3c4: 38 60 00 01 li r3,1 3c8: 4b ff ff ac b 374 <interrupt_exit_kernel_prepare+0x2c> Signed-off-by:
-
Laurent Dufour authored
It is better to rely on the API provided by the MM layer instead of directly manipulating the mm_users field. Signed-off-by:
Laurent Dufour <ldufour@linux.ibm.com> Acked-by:
Frederic Barrat <fbarrat@linux.ibm.com> Acked-by:
Andrew Donnellan <ajd@linux.ibm.com> Signed-off-by:
-
Ravi Bangoria authored
As per ISA 3.1, prefixed instruction should not cross 64-byte boundary. So don't allow Uprobe on such prefixed instruction. There are two ways probed instruction is changed in mapped pages. First, when Uprobe is activated, it searches for all the relevant pages and replace instruction in them. In this case, if that probe is on the 64-byte unaligned prefixed instruction, error out directly. Second, when Uprobe is already active and user maps a relevant page via mmap(), instruction is replaced via mmap() code path. But because Uprobe is invalid, entire mmap() operation can not be stopped. In this case just print an error and continue. Signed-off-by:
Ravi Bangoria <ravi.bangoria@linux.ibm.com> Acked-by:
Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> Acked-by:
Sandipan Das <sandipan@linux.ibm.com> Signed-off-by:
-
Christopher M. Riedl authored
Usually sigset_t is exactly 8B which is a "trivial" size and does not warrant using __copy_from_user(). Use __get_user() directly in anticipation of future work to remove the trivial size optimizations from __copy_from_user(). The ppc32 implementation of get_sigset_t() previously called copy_from_user() which, unlike __copy_from_user(), calls access_ok(). Replacing this w/ __get_user() (no access_ok()) is fine here since both callsites in signal_32.c are preceded by an earlier access_ok(). Signed-off-by:
Christopher M. Riedl <cmr@codefail.de> Signed-off-by:
-
Daniel Axtens authored
Add uaccess blocks and use the 'unsafe' versions of functions doing user access where possible to reduce the number of times uaccess has to be opened/closed. Co-developed-by:
Daniel Axtens <dja@axtens.net> Signed-off-by:
-
Daniel Axtens authored
Add uaccess blocks and use the 'unsafe' versions of functions doing user access where possible to reduce the number of times uaccess has to be opened/closed. There is no 'unsafe' version of copy_siginfo_to_user, so move it slightly to allow for a "longer" uaccess block. Co-developed-by:
-
Christopher M. Riedl authored
Previously restore_sigcontext() performed a costly KUAP switch on every uaccess operation. These repeated uaccess switches cause a significant drop in signal handling performance. Rewrite restore_sigcontext() to assume that a userspace read access window is open by replacing all uaccess functions with their 'unsafe' versions. Modify the callers to first open, call unsafe_restore_sigcontext(), and then close the uaccess window. Signed-off-by:
-
Christopher M. Riedl authored
Previously setup_sigcontext() performed a costly KUAP switch on every uaccess operation. These repeated uaccess switches cause a significant drop in signal handling performance. Rewrite setup_sigcontext() to assume that a userspace write access window is open by replacing all uaccess functions with their 'unsafe' versions. Modify the callers to first open, call unsafe_setup_sigcontext() and then close the uaccess window. Signed-off-by:
-
Christopher M. Riedl authored
Both rt_sigreturn() and handle_rt_signal_64() contain TM-related ifdefs which break-up an if/else block. Provide stubs for the ifdef-guarded TM functions and remove the need for an ifdef in rt_sigreturn(). Rework the remaining TM ifdef in handle_rt_signal64() similar to commit f1cf4f93 ("powerpc/signal32: Remove ifdefery in middle of if/else"). Unlike in the commit for ppc32, the ifdef can't be removed entirely since uc_transact in sigframe depends on CONFIG_PPC_TRANSACTIONAL_MEM. Signed-off-by:
-
Christopher M. Riedl authored
Unlike the other MSR_TM_* macros, MSR_TM_ACTIVE does not reference or use its parameter unless CONFIG_PPC_TRANSACTIONAL_MEM is defined. This causes an 'unused variable' compile warning unless the variable is also guarded with CONFIG_PPC_TRANSACTIONAL_MEM. Reference but do nothing with the argument in the macro to avoid a potential compile warning. Signed-off-by:
-
Christopher M. Riedl authored
The majority of setup_sigcontext() can be refactored to execute in an "unsafe" context assuming an open uaccess window except for some non-inline function calls. Move these out into a separate prepare_setup_sigcontext() function which must be called first and before opening up a uaccess window. Non-inline function calls should be avoided during a uaccess window for a few reasons: - KUAP should be enabled for as much kernel code as possible. Opening a uaccess window disables KUAP which means any code executed during this time contributes to a potential attack surface. - Non-inline functions default to traceable which means they are instrumented for ftrace. This adds more code which could run with KUAP disabled. - Powerpc does not currently support the objtool UACCESS checks. All code running with uaccess must be audited manually which means: less code -> less work -> fewer problems (in theory). A follow-up commit converts setup_sigcontext() to be "unsafe". Signed-off-by:
-
