- 25 Jun, 2019 6 commits
-
-
Aaro Koskinen authored
Currently arm64 uses the default implementation of panic_smp_self_stop() where the CPU runs in a cpu_relax() loop unable to receive IPIs anymore. As a result, when two CPUs panic() simultaneously we get "SMP: failed to stop secondary CPUs" warnings and extra delays before a reset, because smp_send_stop() still tries to stop the other paniced CPU. Provide an implementation of panic_smp_self_stop() that is identical to the IPI CPU stop handler, so that the online status of stopped CPUs gets properly updated. Acked-by:
Will Deacon <will.deacon@arm.com> Signed-off-by:
Aaro Koskinen <aaro.koskinen@nokia.com> Signed-off-by:
Catalin Marinas <catalin.marinas@arm.com>
-
Jayachandran C authored
The current code puts the stopped cpus in an 'yield' instruction loop. Using a busy loop here is unnecessary, we can use the cpu_park_loop() function here to do a wfi/wfe. Signed-off-by:
Jayachandran C <jnair@caviumnetworks.com> Signed-off-by:
-
Mark Brown authored
ARMv8.5 introduces the FRINT series of instructions for rounding floating point numbers to integers. Provide a capability to userspace in order to allow applications to determine if the system supports these instructions. Signed-off-by:
Mark Brown <broonie@kernel.org> Acked-by:
-
Mark Brown authored
ARMv8.5 adds new instructions XAFLAG and AXFLAG to translate the representation of the results of floating point comparisons between the native ARM format and an alternative format used by some software. Add a hwcap allowing userspace to determine if they are present, since we referred to earlier CondM extensions as FLAGM call these extensions FLAGM2. Signed-off-by:
-
Nick Desaulniers authored
For testing coverage and improved defense in depth, enable KASLR by default. Acked-by:
Ard Biesheuvel <ard.biesheuvel@linaro.org> Acked-by:
Will Deacon <will@kernel.org> Reviewed-by:
Kees Cook <keescook@chromium.org> Suggested-by:
Arnd Bergmann <arnd@arndb.de> Suggested-by:
Olof Johansson <olof@lixom.net> Signed-off-by:
Nick Desaulniers <ndesaulniers@google.com> Signed-off-by:
-
Catalin Marinas authored
Otherwise, selecting it without MODULES leads to build failures. Fixes: 58557e48 ("arm64: Allow user selection of ARM64_MODULE_PLTS") Signed-off-by:
-
- 24 Jun, 2019 6 commits
-
-
Ard Biesheuvel authored
The BPF code now takes care of mapping the code pages executable after mapping them read-only, to ensure that no RWX mapped regions are needed, even transiently. This means we can drop the executable permissions from the mapping at allocation time. Acked-by:
Ard Biesheuvel <ard.biesheuvel@arm.com> Signed-off-by:
-
Ard Biesheuvel authored
In order to avoid transient inconsistencies where freed code pages are remapped writable while stale TLB entries still exist on other cores, mark the kprobes text pages with the VM_FLUSH_RESET_PERMS attribute. This instructs the core vmalloc code not to defer the TLB flush when this region is unmapped and returned to the page allocator. Acked-by:
-
Ard Biesheuvel authored
Wire up the special helper functions to manipulate aliases of vmalloc regions in the linear map. Acked-by:
-
Ard Biesheuvel authored
Now that the core code manages the executable permissions of code regions of modules explicitly, it is no longer necessary to create the module vmalloc regions with RWX permissions, and we can create them with RW- permissions instead, which is preferred from a security perspective. Acked-by:
-
Florian Fainelli authored
Make ARM64_MODULE_PLTS a selectable Kconfig symbol, since some people might have very big modules spilling out of the dedicated module area into vmalloc. Help text is copied from the ARM 32-bit counterpart and modified to a mention of KASLR and specific ARM errata workaround(s). Acked-by:
Florian Fainelli <f.fainelli@gmail.com> Signed-off-by:
-
Ard Biesheuvel authored
Some Qualcomm Snapdragon based laptops built to run Microsoft Windows are clearly ACPI 5.1 based, given that that is the first ACPI revision that supports ARM, and introduced the FADT 'arm_boot_flags' field, which has a non-zero field on those systems. So in these cases, infer from the ARM boot flags that the FADT must be 5.1 or later, and treat it as 5.1. Acked-by:
Sudeep Holla <sudeep.holla@arm.com> Tested-by:
Lee Jones <lee.jones@linaro.org> Reviewed-by:
Graeme Gregory <graeme.gregory@linaro.org> Acked-by:
Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> Acked-by:
Hanjun Guo <guohanjun@huawei.com> Signed-off-by:
-
- 21 Jun, 2019 8 commits
-
-
Julien Thierry authored
Now that Pseudo-NMI are fixed, allow the use of that option again This reverts commit 96a13f57 ("arm64: Kconfig: Make ARM64_PSEUDO_NMI depend on BROKEN for now"). Cc: Will Deacon <will.deacon@arm.com> Reviewed-by:
Marc Zyngier <marc.zyngier@arm.com> Signed-off-by:
Julien Thierry <julien.thierry@arm.com> Signed-off-by:
-
Wei Li authored
When enabling ARM64_PSEUDO_NMI feature in kdump capture kernel, it will report a kernel stack overflow exception: [ 0.000000] CPU features: detected: IRQ priority masking [ 0.000000] alternatives: patching kernel code [ 0.000000] Insufficient stack space to handle exception! [ 0.000000] ESR: 0x96000044 -- DABT (current EL) [ 0.000000] FAR: 0x0000000000000040 [ 0.000000] Task stack: [0xffff0000097f0000..0xffff0000097f4000] [ 0.000000] IRQ stack: [0x0000000000000000..0x0000000000004000] [ 0.000000] Overflow stack: [0xffff80002b7cf290..0xffff80002b7d0290] [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.19.34-lw+ #3 [ 0.000000] pstate: 400003c5 (nZcv DAIF -PAN -UAO) [ 0.000000] pc : el1_sync+0x0/0xb8 [ 0.000000] lr : el1_irq+0xb8/0x140 [ 0.000000] sp : 0000000000000040 [ 0.000000] pmr_save: 00000070 [ 0.000000] x29: ffff0000097f3f60 x28: ffff000009806240 [ 0.000000] x27: 0000000080000000 x26: 0000000000004000 [ 0.000000] x25: 0000000000000000 x24: ffff000009329028 [ 0.000000] x23: 0000000040000005 x22: ffff000008095c6c [ 0.000000] x21: ffff0000097f3f70 x20: 0000000000000070 [ 0.000000] x19: ffff0000097f3e30 x18: ffffffffffffffff [ 0.000000] x17: 0000000000000000 x16: 0000000000000000 [ 0.000000] x15: ffff0000097f9708 x14: ffff000089a382ef [ 0.000000] x13: ffff000009a382fd x12: ffff000009824000 [ 0.000000] x11: ffff0000097fb7b0 x10: ffff000008730028 [ 0.000000] x9 : ffff000009440018 x8 : 000000000000000d [ 0.000000] x7 : 6b20676e69686374 x6 : 000000000000003b [ 0.000000] x5 : 0000000000000000 x4 : ffff000008093600 [ 0.000000] x3 : 0000000400000008 x2 : 7db2e689fc2b8e00 [ 0.000000] x1 : 0000000000000000 x0 : ffff0000097f3e30 [ 0.000000] Kernel panic - not syncing: kernel stack overflow [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.19.34-lw+ #3 [ 0.000000] Call trace: [ 0.000000] dump_backtrace+0x0/0x1b8 [ 0.000000] show_stack+0x24/0x30 [ 0.000000] dump_stack+0xa8/0xcc [ 0.000000] panic+0x134/0x30c [ 0.000000] __stack_chk_fail+0x0/0x28 [ 0.000000] handle_bad_stack+0xfc/0x108 [ 0.000000] __bad_stack+0x90/0x94 [ 0.000000] el1_sync+0x0/0xb8 [ 0.000000] init_gic_priority_masking+0x4c/0x70 [ 0.000000] smp_prepare_boot_cpu+0x60/0x68 [ 0.000000] start_kernel+0x1e8/0x53c [ 0.000000] ---[ end Kernel panic - not syncing: kernel stack overflow ]--- The reason is init_gic_priority_masking() may unmask PSR.I while the irq stacks are not inited yet. Some "NMI" could be raised unfortunately and it will just go into this exception. In this patch, we just write the PMR in smp_prepare_boot_cpu(), and delay unmasking PSR.I after irq stacks inited in init_IRQ(). Fixes: e7932188 ("arm64: Switch to PMR masking when starting CPUs") Cc: Will Deacon <will.deacon@arm.com> Reviewed-by:
Wei Li <liwei391@huawei.com> [JT: make init_gic_priority_masking() not modify daif, rebase on other priority masking fixes] Signed-off-by:
-
Julien Thierry authored
Using IRQ priority masking to enable/disable interrupts is a bit sensitive as it requires to deal with both ICC_PMR_EL1 and PSR.I. Introduce some validity checks to both highlight the states in which functions dealing with IRQ enabling/disabling can (not) be called, and bark a warning when called in an unexpected state. Since these checks are done on hotpaths, introduce a build option to choose whether to do the checking. Cc: Will Deacon <will.deacon@arm.com> Reviewed-by:
-
Julien Thierry authored
When using IRQ priority masking to disable interrupts, in order to deal with the PSR.I state, local_irq_save() would convert the I bit into a PMR value (GIC_PRIO_IRQOFF). This resulted in local_irq_restore() potentially modifying the value of PMR in undesired location due to the state of PSR.I upon flag saving [1]. In an attempt to solve this issue in a less hackish manner, introduce a bit (GIC_PRIO_IGNORE_PMR) for the PMR values that can represent whether PSR.I is being used to disable interrupts, in which case it takes precedence of the status of interrupt masking via PMR. GIC_PRIO_PSR_I_SET is chosen such that (<pmr_value> | GIC_PRIO_PSR_I_SET) does not mask more interrupts than <pmr_value> as some sections (e.g. arch_cpu_idle(), interrupt acknowledge path) requires PMR not to mask interrupts that could be signaled to the CPU when using only PSR.I. [1] https://www.spinics.net/lists/arm-kernel/msg716956.html Fixes: 4a503217 ("arm64: irqflags: Use ICC_PMR_EL1 for interrupt masking") Cc: <stable@vger.kernel.org> # 5.1.x- Reported-by:
Zenghui Yu <yuzenghui@huawei.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Wei Li <liwei391@huawei.com> Cc: Will Deacon <will.deacon@arm.com> Cc: Christoffer Dall <christoffer.dall@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Suzuki K Pouloze <suzuki.poulose@arm.com> Cc: Oleg Nesterov <oleg@redhat.com> Reviewed-by:
-
Julien Thierry authored
In the presence of any form of instrumentation, nmi_enter() should be done before calling any traceable code and any instrumentation code. Currently, nmi_enter() is done in handle_domain_nmi(), which is much too late as instrumentation code might get called before. Move the nmi_enter/exit() calls to the arch IRQ vector handler. On arm64, it is not possible to know if the IRQ vector handler was called because of an NMI before acknowledging the interrupt. However, It is possible to know whether normal interrupts could be taken in the interrupted context (i.e. if taking an NMI in that context could introduce a potential race condition). When interrupting a context with IRQs disabled, call nmi_enter() as soon as possible. In contexts with IRQs enabled, defer this to the interrupt controller, which is in a better position to know if an interrupt taken is an NMI. Fixes: bc3c03cc ("arm64: Enable the support of pseudo-NMIs") Cc: <stable@vger.kernel.org> # 5.1.x- Cc: Will Deacon <will.deacon@arm.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Jason Cooper <jason@lakedaemon.net> Cc: Mark Rutland <mark.rutland@arm.com> Reviewed-by:
-
Julien Thierry authored
Some of the inline assembly instruction use the condition flags and need to include "cc" in the clobber list. Fixes: 4a503217 ("arm64: irqflags: Use ICC_PMR_EL1 for interrupt masking") Cc: <stable@vger.kernel.org> # 5.1.x- Suggested-by:
Mark Rutland <mark.rutland@arm.com> Signed-off-by:
-
Julien Thierry authored
Flags are only read by the instructions doing the irqflags restore operation. Pass the operand as read only to the asm inline instead of read-write. Cc: Will Deacon <will.deacon@arm.com> Reviewed-by:
Mark Rutland <mark.rutland@ar.com> Signed-off-by:
-
Julien Thierry authored
For el0_dbg and el0_error, DAIF bits get explicitly cleared before calling ct_user_exit. When context tracking is disabled, DAIF gets set (almost) immediately after. When context tracking is enabled, among the first things done is disabling IRQs. What is actually needed is: - PSR.D = 0 so the system can be debugged (should be already the case) - PSR.A = 0 so async error can be handled during context tracking Do not clear PSR.I in those two locations. Reviewed-by:
James Morse <james.morse@arm.com> Cc: Will Deacon <will.deacon@arm.com> Signed-off-by:
-
- 17 Jun, 2019 1 commit
-
-
Masayoshi Mizuma authored
If the cache line size is greater than ARCH_DMA_MINALIGN (128), the warning shows and it's tainted as TAINT_CPU_OUT_OF_SPEC. However, it's not good because as discussed in the thread [1], the cpu cache line size will be problem only on non-coherent devices. Since the coherent flag is already introduced to struct device, show the warning only if the device is non-coherent device and ARCH_DMA_MINALIGN is smaller than the cpu cache size. [1] https://lore.kernel.org/linux-arm-kernel/20180514145703.celnlobzn3uh5tc2@localhost/Signed-off-by:
Masayoshi Mizuma <m.mizuma@jp.fujitsu.com> Reviewed-by:
Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com> Tested-by:
Zhang Lei <zhang.lei@jp.fujitsu.com> [catalin.marinas@arm.com: removed 'if' block for WARN_TAINT] Signed-off-by:
-
- 14 Jun, 2019 1 commit
-
-
Bartlomiej Zolnierkiewicz authored
'default n' is the default value for any bool or tristate Kconfig setting so there is no need to write it explicitly. Also since commit f467c564 ("kconfig: only write '# CONFIG_FOO is not set' for visible symbols") the Kconfig behavior is the same regardless of 'default n' being present or not: ... One side effect of (and the main motivation for) this change is making the following two definitions behave exactly the same: config FOO bool config FOO bool default n With this change, neither of these will generate a '# CONFIG_FOO is not set' line (assuming FOO isn't selected/implied). That might make it clearer to people that a bare 'default n' is redundant. ... Signed-off-by:
Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com> Signed-off-by:
-
- 10 Jun, 2019 1 commit
-
-
Mark Rutland authored
In set_pte_at(), we read the old pte value so that it can be passed into checks for racy hw updates. These checks are only performed for CONFIG_DEBUG_VM, and the value is not used otherwise. Since we read the pte value with READ_ONCE(), the compiler cannot elide the redundant read for !CONFIG_DEBUG_VM kernels. Let's ameliorate matters by moving the read and the checks into a helper, __check_racy_pte_update(), which only performs the read when the value will be used. This also allows us to reformat the conditions for clarity. Acked-by:
-
- 07 Jun, 2019 3 commits
-
-
Anshuman Khandual authored
__do_page_fault() is over complicated with multiple goto statements. This cleans up the code flow and while there drops local variable vm_fault_t. Reviewed-by:
Anshuman Khandual <anshuman.khandual@arm.com> Cc: Will Deacon <will.deacon@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Christoph Hellwig <hch@infradead.org> Signed-off-by:
-
Anshuman Khandual authored
This patch adds an is_write_abort() wrapper and documents the detection of the abort type on cache maintenance operations. Cc: Will Deacon <will.deacon@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Andrey Konovalov <andreyknvl@google.com> Acked-by:
-
Odin Ugedal authored
The config value used in the if was changed in b433dce0, but the comment on the corresponding end was not changed. Signed-off-by:
Odin Ugedal <odin@ugedal.com> Signed-off-by:
-
- 05 Jun, 2019 3 commits
-
-
Sudeep Holla authored
Add PTRACE_SYSEMU and PTRACE_SYSEMU_SINGLESTEP support on arm64. We don't need any special handling for PTRACE_SYSEMU_SINGLESTEP. It's quite difficult to generalize handling PTRACE_SYSEMU cross architectures and avoid calls to tracehook_report_syscall_entry twice. Different architecture have different mechanism to indicate NO_SYSCALL and trying to generalise adds more code for no gain. Cc: Will Deacon <will.deacon@arm.com> Signed-off-by:
-
Sudeep Holla authored
x86 and um use 31 and 32 for PTRACE_SYSEMU and PTRACE_SYSEMU_SINGLESTEP while powerpc uses different value maybe for legacy reasons. Though handling of PTRACE_SYSEMU can be made architecture independent, it's hard to make these definations generic. To add to this existing mess few architectures like arm, c6x and sh use 31 for PTRACE_GETFDPIC (get the ELF fdpic loadmap address). It's not possible to move the definations to generic headers. So we unfortunately have to duplicate the same defination to ARM64 if we need to support PTRACE_SYSEMU and PTRACE_SYSEMU_SINGLESTEP. Cc: Will Deacon <will.deacon@arm.com> Signed-off-by:
-
Sudeep Holla authored
While the TIF_SYSCALL_EMU is set in ptrace_resume independent of any architecture, currently only powerpc and x86 unset the TIF_SYSCALL_EMU flag in ptrace_disable which gets called from ptrace_detach. Let's move the clearing of TIF_SYSCALL_EMU flag to __ptrace_unlink which gets executed from ptrace_detach and also keep it along with or close to clearing of TIF_SYSCALL_TRACE. Cc: Paul Mackerras <paulus@samba.org> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Ingo Molnar <mingo@redhat.com> Acked-by:
Oleg Nesterov <oleg@redhat.com> Signed-off-by:
-
- 04 Jun, 2019 11 commits
-
-
Anshuman Khandual authored
The task_struct argument is not getting used in __do_page_fault(). Hence just drop it and use current or cuurent->mm instead where ever required. This does not change any functionality. Signed-off-by:
-
Anshuman Khandual authored
There is an inconsistency between down_read_trylock() success and failure paths while dealing with kernel access for non exception table areas where it calls __do_kernel_fault(). In case of failure it just bails out without holding mmap_sem but when it succeeds it does so while holding mmap_sem. Fix this inconsistency by just dropping mmap_sem in success path as well. __do_kernel_fault() calls die_kernel_fault() which then calls show_pte(). show_pte() in this path might become bit more unreliable without holding mmap_sem. But there are already instances [1] in do_page_fault() where die_kernel_fault() gets called without holding mmap_sem. show_pte() can be made more robust independently but in a later patch. [1] Conditional block for (is_ttbr0_addr && is_el1_permission_fault) Signed-off-by:
-
Anshuman Khandual authored
We don't currently set the FAULT_FLAG_INSTRUCTION mm flag for EL0 instruction aborts. This has no functional impact, as we don't override arch_vma_access_permitted(), and the default implementation always returns true. However, it would be helpful to provide the flag so that it can be consumed by tracepoints such as dax_pmd_fault. This patch sets the FAULT_FLAG_INSTRUCTION flag for EL0 instruction aborts. Signed-off-by:
-
Anshuman Khandual authored
There are no callers for the functions which will pass unaligned physical addresses. Hence just change these BUG_ON() checks into VM_BUG_ON() which gets compiled out unless CONFIG_VM_DEBUG is enabled. Signed-off-by:
-
Liu Song authored
Should use aff3 instead of aff2 in comment. Signed-off-by:
Liu Song <liu.song11@zte.com.cn> Signed-off-by:
-
Julien Grall authored
cpu_enable_ssbs() is called via stop_machine() as part of the cpu_enable callback. A spin lock is used to ensure the hook is registered before the rest of the callback is executed. On -RT spin_lock() may sleep. However, all the callees in stop_machine() are expected to not sleep. Therefore a raw_spin_lock() is required here. Given this is already done under stop_machine() and the work done under the lock is quite small, the latency should not increase too much. Signed-off-by:
Julien Grall <julien.grall@arm.com> Signed-off-by:
-
Miles Chen authored
This change makes CONFIG_ZONE_DMA32 defuly y and allows users to overwrite it only when CONFIG_EXPERT=y. For the SoCs that do not need CONFIG_ZONE_DMA32, this is the first step to manage all available memory by a single zone(normal zone) to reduce the overhead of multiple zones. The change also fixes a build error when CONFIG_NUMA=y and CONFIG_ZONE_DMA32=n. arch/arm64/mm/init.c:195:17: error: use of undeclared identifier 'ZONE_DMA32' max_zone_pfns[ZONE_DMA32] = PFN_DOWN(max_zone_dma_phys()); Change since v1: 1. only expose CONFIG_ZONE_DMA32 when CONFIG_EXPERT=y 2. remove redundant IS_ENABLED(CONFIG_ZONE_DMA32) Cc: Robin Murphy <robin.murphy@arm.com> Signed-off-by:Miles Chen <miles.chen@mediatek.com> Signed-off-by:
-
Anshuman Khandual authored
Even though they have got the same value, PMD_TYPE_SECT and PUD_TYPE_SECT get used for kernel huge mappings. But before that first the table bit gets cleared using leaf level PTE_TABLE_BIT. Though functionally they are same, we should use page table level specific macros to be consistent as per the MMU specifications. Create page table level specific wrappers for kernel huge mapping entries and just drop mk_sect_prot() which does not have any other user. Signed-off-by:
-
Shaokun Zhang authored
cache_line_size is derived from CTR_EL0.CWG field and is called mostly for I/O device drivers. For some platforms like the HiSilicon Kunpeng920 server SoC, cache line sizes are different between L1/2 cache and L3 cache while L1 cache line size is 64-byte and L3 is 128-byte, but CTR_EL0.CWG is misreporting using L1 cache line size. We shall correct the right value which is important for I/O performance. Let's update the cache line size if it is detected from DT or PPTT information. Cc: Will Deacon <will.deacon@arm.com> Cc: Jeremy Linton <jeremy.linton@arm.com> Cc: Zhenfa Qiu <qiuzhenfa@hisilicon.com> Reported-by:
Zhenfa Qiu <qiuzhenfa@hisilicon.com> Suggested-by:
Shaokun Zhang <zhangshaokun@hisilicon.com> Signed-off-by:
-
Shaokun Zhang authored
Add coherency_max_size variable to record the maximum cache line size for different cache levels. If it is available, we will synchronize it as cache line size, otherwise we will use CTR_EL0.CWG reporting in cache_line_size() for arm64. Cc: "Rafael J. Wysocki" <rafael@kernel.org> Cc: Jeremy Linton <jeremy.linton@arm.com> Cc: Will Deacon <will.deacon@arm.com> Reviewed-by:
Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by:
-
Julien Grall authored
When the kernel is compiled with CONFIG_KERNEL_MODE_NEON, some part of the kernel may be able to use FPSIMD/SVE. This is for instance the case for crypto code. Any use of FPSIMD/SVE in the kernel are clearly marked by using the function kernel_neon_{begin, end}. Furthermore, this can only be used when may_use_simd() returns true. The current implementation of may_use_simd() allows softirq to use FPSIMD/SVE unless it is currently in use (i.e kernel_neon_busy is true). When in use, softirqs usually fall back to a software method. At the moment, as a softirq may use FPSIMD/SVE, softirqs are disabled when touching the FPSIMD/SVE context. This has the drawback to disable all softirqs even if they are not using FPSIMD/SVE. Since a softirq is supposed to check may_use_simd() anyway before attempting to use FPSIMD/SVE, there is limited reason to keep softirq disabled when touching the FPSIMD/SVE context. Instead, we can simply disable preemption and mark the FPSIMD/SVE context as in use by setting CPU's fpsimd_context_busy flag. Two new helpers {get, put}_cpu_fpsimd_context are introduced to mark the area using FPSIMD/SVE context and they are used to replace local_bh_{disable, enable}. The functions kernel_neon_{begin, end} are also re-implemented to use the new helpers. Additionally, double-underscored versions of the helpers are provided to called when preemption is already disabled. These are only relevant on paths where irqs are disabled anyway, so they are not needed for correctness in the current code. Let's use them anyway though: this marks critical sections clearly and will help to avoid mistakes during future maintenance. The change has been benchmarked on Linux 5.1-rc4 with defconfig. On Juno2: * hackbench 100 process 1000 (10 times) * .7% quicker On ThunderX 2: * hackbench 1000 process 1000 (20 times) * 3.4% quicker Reviewed-by:Dave Martin <dave.martin@arm.com> Acked-by:
-
