Skip to content

L
linux
Commit 7af8a0f8 authored by Linus Torvalds's avatar Linus Torvalds
Browse files
Options

Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux 

Pull arm64 updates from Will Deacon:
 "It's a bit all over the place this time with no "killer feature" to
  speak of.  Support for mismatched cache line sizes should help people
  seeing whacky JIT failures on some SoCs, and the big.LITTLE perf
  updates have been a long time coming, but a lot of the changes here
  are cleanups.

  We stray outside arch/arm64 in a few areas: the arch/arm/ arch_timer
  workaround is acked by Russell, the DT/OF bits are acked by Rob, the
  arch_timer clocksource changes acked by Marc, CPU hotplug by tglx and
  jump_label by Peter (all CC'd).

  Summary:

   - Support for execute-only page permissions
   - Support for hibernate and DEBUG_PAGEALLOC
   - Support for heterogeneous systems with mismatches cache line sizes
   - Errata workarounds (A53 843419 update and QorIQ A-008585 timer bug)
   - arm64 PMU perf updates, including cpumasks for heterogeneous systems
   - Set UTS_MACHINE for building rpm packages
   - Yet another head.S tidy-up
   - Some cleanups and refactoring, particularly in the NUMA code
   - Lots of random, non-critical fixes across the board"

* tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (100 commits)
  arm64: tlbflush.h: add __tlbi() macro
  arm64: Kconfig: remove SMP dependence for NUMA
  arm64: Kconfig: select OF/ACPI_NUMA under NUMA config
  arm64: fix dump_backtrace/unwind_frame with NULL tsk
  arm/arm64: arch_timer: Use archdata to indicate vdso suitability
  arm64: arch_timer: Work around QorIQ Erratum A-008585
  arm64: arch_timer: Add device tree binding for A-008585 erratum
  arm64: Correctly bounds check virt_addr_valid
  arm64: migrate exception table users off module.h and onto extable.h
  arm64: pmu: Hoist pmu platform device name
  arm64: pmu: Probe default hw/cache counters
  arm64: pmu: add fallback probe table
  MAINTAINERS: Update ARM PMU PROFILING AND DEBUGGING entry
  arm64: Improve kprobes test for atomic sequence
  arm64/kvm: use alternative auto-nop
  arm64: use alternative auto-nop
  arm64: alternative: add auto-nop infrastructure
  arm64: lse: convert lse alternatives NOP padding to use __nops
  arm64: barriers: introduce nops and __nops macros for NOP sequences
  arm64: sysreg: replace open-coded mrs_s/msr_s with {read,write}_sysreg_s
  ...
parents c8d2bc9b db68f3e7
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1699 additions and 974 deletions
Documentation/arm64/silicon-errata.txt
View file @ 7af8a0f8
......@@ -61,3 +61,5 @@ stable kernels.
| Cavium | ThunderX GICv3 | #23154 | CAVIUM_ERRATUM_23154 |
| Cavium | ThunderX Core | #27456 | CAVIUM_ERRATUM_27456 |
| Cavium | ThunderX SMMUv2 | #27704 | N/A |
| | | | |
| Freescale/NXP | LS2080A/LS1043A | A-008585 | FSL_ERRATUM_A008585 |
Documentation/devicetree/bindings/arm/arch_timer.txt
View file @ 7af8a0f8
......@@ -25,6 +25,12 @@ to deliver its interrupts via SPIs.
- always-on : a boolean property. If present, the timer is powered through an
always-on power domain, therefore it never loses context.
- fsl,erratum-a008585 : A boolean property. Indicates the presence of
QorIQ erratum A-008585, which says that reading the counter is
unreliable unless the same value is returned by back-to-back reads.
This also affects writes to the tval register, due to the implicit
counter read.
** Optional properties:
- arm,cpu-registers-not-fw-configured : Firmware does not initialize
......
Documentation/kernel-parameters.txt
View file @ 7af8a0f8
......@@ -698,6 +698,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
loops can be debugged more effectively on production
systems.
clocksource.arm_arch_timer.fsl-a008585=
[ARM64]
Format: <bool>
Enable/disable the workaround of Freescale/NXP
erratum A-008585. This can be useful for KVM
guests, if the guest device tree doesn't show the
erratum. If unspecified, the workaround is
enabled based on the device tree.
clearcpuid=BITNUM [X86]
Disable CPUID feature X for the kernel. See
arch/x86/include/asm/cpufeatures.h for the valid bit
......
Documentation/static-keys.txt
View file @ 7af8a0f8
......@@ -15,6 +15,8 @@ The updated API replacements are:
DEFINE_STATIC_KEY_TRUE(key);
DEFINE_STATIC_KEY_FALSE(key);
DEFINE_STATIC_KEY_ARRAY_TRUE(keys, count);
DEFINE_STATIC_KEY_ARRAY_FALSE(keys, count);
static_branch_likely()
static_branch_unlikely()
......@@ -140,6 +142,13 @@ static_branch_inc(), will change the branch back to true. Likewise, if the
key is initialized false, a 'static_branch_inc()', will change the branch to
true. And then a 'static_branch_dec()', will again make the branch false.
Where an array of keys is required, it can be defined as:
DEFINE_STATIC_KEY_ARRAY_TRUE(keys, count);
or:
DEFINE_STATIC_KEY_ARRAY_FALSE(keys, count);
4) Architecture level code patching interface, 'jump labels'
......
MAINTAINERS
View file @ 7af8a0f8
......@@ -913,15 +913,17 @@ F: arch/arm/include/asm/floppy.h
ARM PMU PROFILING AND DEBUGGING
M: Will Deacon <will.deacon@arm.com>
R: Mark Rutland <mark.rutland@arm.com>
M: Mark Rutland <mark.rutland@arm.com>
S: Maintained
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
F: arch/arm*/kernel/perf_*
F: arch/arm/oprofile/common.c
F: arch/arm*/kernel/hw_breakpoint.c
F: arch/arm*/include/asm/hw_breakpoint.h
F: arch/arm*/include/asm/perf_event.h
F: drivers/perf/arm_pmu.c
F: drivers/perf/*
F: include/linux/perf/arm_pmu.h
F: Documentation/devicetree/bindings/arm/pmu.txt
ARM PORT
M: Russell King <linux@armlinux.org.uk>
......
arch/arm/Kconfig
View file @ 7af8a0f8
config ARM
bool
default y
select ARCH_CLOCKSOURCE_DATA
select ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE
select ARCH_HAS_DEVMEM_IS_ALLOWED
select ARCH_HAS_ELF_RANDOMIZE
......
arch/arm/include/asm/clocksource.h 0 → 100644
View file @ 7af8a0f8
#ifndef _ASM_CLOCKSOURCE_H
#define _ASM_CLOCKSOURCE_H
struct arch_clocksource_data {
bool vdso_direct; /* Usable for direct VDSO access? */
};
#endif
arch/arm/kernel/perf_event_v7.c
View file @ 7af8a0f8
......@@ -596,12 +596,6 @@ static struct attribute_group armv7_pmuv1_events_attr_group = {
.attrs = armv7_pmuv1_event_attrs,
};
static const struct attribute_group *armv7_pmuv1_attr_groups[] = {
&armv7_pmuv1_events_attr_group,
&armv7_pmu_format_attr_group,
NULL,
};
ARMV7_EVENT_ATTR(mem_access, ARMV7_PERFCTR_MEM_ACCESS);
ARMV7_EVENT_ATTR(l1i_cache, ARMV7_PERFCTR_L1_ICACHE_ACCESS);
ARMV7_EVENT_ATTR(l1d_cache_wb, ARMV7_PERFCTR_L1_DCACHE_WB);
......@@ -653,12 +647,6 @@ static struct attribute_group armv7_pmuv2_events_attr_group = {
.attrs = armv7_pmuv2_event_attrs,
};
static const struct attribute_group *armv7_pmuv2_attr_groups[] = {
&armv7_pmuv2_events_attr_group,
&armv7_pmu_format_attr_group,
NULL,
};
/*
* Perf Events' indices
*/
......@@ -1208,7 +1196,10 @@ static int armv7_a8_pmu_init(struct arm_pmu *cpu_pmu)
armv7pmu_init(cpu_pmu);
cpu_pmu->name = "armv7_cortex_a8";
cpu_pmu->map_event = armv7_a8_map_event;
cpu_pmu->pmu.attr_groups = armv7_pmuv1_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv7_pmuv1_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv7_pmu_format_attr_group;
return armv7_probe_num_events(cpu_pmu);
}
......@@ -1217,7 +1208,10 @@ static int armv7_a9_pmu_init(struct arm_pmu *cpu_pmu)
armv7pmu_init(cpu_pmu);
cpu_pmu->name = "armv7_cortex_a9";
cpu_pmu->map_event = armv7_a9_map_event;
cpu_pmu->pmu.attr_groups = armv7_pmuv1_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv7_pmuv1_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv7_pmu_format_attr_group;
return armv7_probe_num_events(cpu_pmu);
}
......@@ -1226,7 +1220,10 @@ static int armv7_a5_pmu_init(struct arm_pmu *cpu_pmu)
armv7pmu_init(cpu_pmu);
cpu_pmu->name = "armv7_cortex_a5";
cpu_pmu->map_event = armv7_a5_map_event;
cpu_pmu->pmu.attr_groups = armv7_pmuv1_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv7_pmuv1_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv7_pmu_format_attr_group;
return armv7_probe_num_events(cpu_pmu);
}
......@@ -1236,7 +1233,10 @@ static int armv7_a15_pmu_init(struct arm_pmu *cpu_pmu)
cpu_pmu->name = "armv7_cortex_a15";
cpu_pmu->map_event = armv7_a15_map_event;
cpu_pmu->set_event_filter = armv7pmu_set_event_filter;
cpu_pmu->pmu.attr_groups = armv7_pmuv2_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv7_pmuv2_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv7_pmu_format_attr_group;
return armv7_probe_num_events(cpu_pmu);
}
......@@ -1246,7 +1246,10 @@ static int armv7_a7_pmu_init(struct arm_pmu *cpu_pmu)
cpu_pmu->name = "armv7_cortex_a7";
cpu_pmu->map_event = armv7_a7_map_event;
cpu_pmu->set_event_filter = armv7pmu_set_event_filter;
cpu_pmu->pmu.attr_groups = armv7_pmuv2_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv7_pmuv2_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv7_pmu_format_attr_group;
return armv7_probe_num_events(cpu_pmu);
}
......@@ -1256,7 +1259,10 @@ static int armv7_a12_pmu_init(struct arm_pmu *cpu_pmu)
cpu_pmu->name = "armv7_cortex_a12";
cpu_pmu->map_event = armv7_a12_map_event;
cpu_pmu->set_event_filter = armv7pmu_set_event_filter;
cpu_pmu->pmu.attr_groups = armv7_pmuv2_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv7_pmuv2_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv7_pmu_format_attr_group;
return armv7_probe_num_events(cpu_pmu);
}
......@@ -1264,7 +1270,10 @@ static int armv7_a17_pmu_init(struct arm_pmu *cpu_pmu)
{
int ret = armv7_a12_pmu_init(cpu_pmu);
cpu_pmu->name = "armv7_cortex_a17";
cpu_pmu->pmu.attr_groups = armv7_pmuv2_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv7_pmuv2_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv7_pmu_format_attr_group;
return ret;
}
......
arch/arm/kernel/vdso.c
View file @ 7af8a0f8
......@@ -270,7 +270,7 @@ static bool tk_is_cntvct(const struct timekeeper *tk)
if (!IS_ENABLED(CONFIG_ARM_ARCH_TIMER))
return false;
if (strcmp(tk->tkr_mono.clock->name, "arch_sys_counter") != 0)
if (!tk->tkr_mono.clock->archdata.vdso_direct)
return false;
return true;
......
arch/arm64/Kconfig
View file @ 7af8a0f8
......@@ -4,6 +4,7 @@ config ARM64
select ACPI_GENERIC_GSI if ACPI
select ACPI_REDUCED_HARDWARE_ONLY if ACPI
select ACPI_MCFG if ACPI
select ARCH_CLOCKSOURCE_DATA
select ARCH_HAS_DEVMEM_IS_ALLOWED
select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI
select ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE
......@@ -102,10 +103,8 @@ config ARM64
select NO_BOOTMEM
select OF
select OF_EARLY_FLATTREE
select OF_NUMA if NUMA && OF
select OF_RESERVED_MEM
select PCI_ECAM if ACPI
select PERF_USE_VMALLOC
select POWER_RESET
select POWER_SUPPLY
select SPARSE_IRQ
......@@ -122,6 +121,9 @@ config ARCH_PHYS_ADDR_T_64BIT
config MMU
def_bool y
config DEBUG_RODATA
def_bool y
config ARM64_PAGE_SHIFT
int
default 16 if ARM64_64K_PAGES
......@@ -415,18 +417,13 @@ config ARM64_ERRATUM_845719
config ARM64_ERRATUM_843419
bool "Cortex-A53: 843419: A load or store might access an incorrect address"
depends on MODULES
default y
select ARM64_MODULE_CMODEL_LARGE
select ARM64_MODULE_CMODEL_LARGE if MODULES
help
This option builds kernel modules using the large memory model in
order to avoid the use of the ADRP instruction, which can cause
a subsequent memory access to use an incorrect address on Cortex-A53
parts up to r0p4.
Note that the kernel itself must be linked with a version of ld
which fixes potentially affected ADRP instructions through the
use of veneers.
This option links the kernel with '--fix-cortex-a53-843419' and
builds modules using the large memory model in order to avoid the use
of the ADRP instruction, which can cause a subsequent memory access
to use an incorrect address on Cortex-A53 parts up to r0p4.
If unsure, say Y.
......@@ -582,7 +579,8 @@ config HOTPLUG_CPU
# Common NUMA Features
config NUMA
bool "Numa Memory Allocation and Scheduler Support"
depends on SMP
select ACPI_NUMA if ACPI
select OF_NUMA
help
Enable NUMA (Non Uniform Memory Access) support.
......@@ -603,11 +601,18 @@ config USE_PERCPU_NUMA_NODE_ID
def_bool y
depends on NUMA
config HAVE_SETUP_PER_CPU_AREA
def_bool y
depends on NUMA
config NEED_PER_CPU_EMBED_FIRST_CHUNK
def_bool y
depends on NUMA
source kernel/Kconfig.preempt
source kernel/Kconfig.hz
config ARCH_SUPPORTS_DEBUG_PAGEALLOC
depends on !HIBERNATION
def_bool y
config ARCH_HAS_HOLES_MEMORYMODEL
......
arch/arm64/Kconfig.debug
View file @ 7af8a0f8
......@@ -49,16 +49,6 @@ config DEBUG_SET_MODULE_RONX
If in doubt, say Y.
config DEBUG_RODATA
bool "Make kernel text and rodata read-only"
default y
help
If this is set, kernel text and rodata will be made read-only. This
is to help catch accidental or malicious attempts to change the
kernel's executable code.
If in doubt, say Y.
config DEBUG_ALIGN_RODATA
depends on DEBUG_RODATA
bool "Align linker sections up to SECTION_SIZE"
......
arch/arm64/Kconfig.platforms
View file @ 7af8a0f8
......@@ -159,7 +159,6 @@ config ARCH_TEGRA
select CLKSRC_MMIO
select CLKSRC_OF
select GENERIC_CLOCKEVENTS
select HAVE_CLK
select PINCTRL
select RESET_CONTROLLER
help
......
arch/arm64/Makefile
View file @ 7af8a0f8
......@@ -18,6 +18,14 @@ ifneq ($(CONFIG_RELOCATABLE),)
LDFLAGS_vmlinux += -pie -Bsymbolic
endif
ifeq ($(CONFIG_ARM64_ERRATUM_843419),y)
ifeq ($(call ld-option, --fix-cortex-a53-843419),)
$(warning ld does not support --fix-cortex-a53-843419; kernel may be susceptible to erratum)
else
LDFLAGS_vmlinux += --fix-cortex-a53-843419
endif
endif
KBUILD_DEFCONFIG := defconfig
# Check for binutils support for specific extensions
......@@ -38,10 +46,12 @@ ifeq ($(CONFIG_CPU_BIG_ENDIAN), y)
KBUILD_CPPFLAGS += -mbig-endian
AS += -EB
LD += -EB
UTS_MACHINE := aarch64_be
else
KBUILD_CPPFLAGS += -mlittle-endian
AS += -EL
LD += -EL
UTS_MACHINE := aarch64
endif
CHECKFLAGS += -D__aarch64__
......
arch/arm64/include/asm/Kbuild
View file @ 7af8a0f8
generic-y += bug.h
generic-y += bugs.h
generic-y += clkdev.h
generic-y += cputime.h
......@@ -10,7 +9,6 @@ generic-y += dma-contiguous.h
generic-y += early_ioremap.h
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += ftrace.h
generic-y += hw_irq.h
generic-y += ioctl.h
generic-y += ioctls.h
......@@ -27,12 +25,10 @@ generic-y += mman.h
generic-y += msgbuf.h
generic-y += msi.h
generic-y += mutex.h
generic-y += pci.h
generic-y += poll.h
generic-y += preempt.h
generic-y += resource.h
generic-y += rwsem.h
generic-y += sections.h
generic-y += segment.h
generic-y += sembuf.h
generic-y += serial.h
......@@ -45,7 +41,6 @@ generic-y += swab.h
generic-y += switch_to.h
generic-y += termbits.h
generic-y += termios.h
generic-y += topology.h
generic-y += trace_clock.h
generic-y += types.h
generic-y += unaligned.h
......
arch/arm64/include/asm/acpi.h
View file @ 7af8a0f8
......@@ -12,7 +12,7 @@
#ifndef _ASM_ACPI_H
#define _ASM_ACPI_H
#include <linux/mm.h>
#include <linux/memblock.h>
#include <linux/psci.h>
#include <asm/cputype.h>
......@@ -32,7 +32,11 @@
static inline void __iomem *acpi_os_ioremap(acpi_physical_address phys,
acpi_size size)
{
if (!page_is_ram(phys >> PAGE_SHIFT))
/*
* EFI's reserve_regions() call adds memory with the WB attribute
* to memblock via early_init_dt_add_memory_arch().
*/
if (!memblock_is_memory(phys))
return ioremap(phys, size);
return ioremap_cache(phys, size);
......
arch/arm64/include/asm/alternative.h
View file @ 7af8a0f8
......@@ -2,6 +2,7 @@
#define __ASM_ALTERNATIVE_H
#include <asm/cpufeature.h>
#include <asm/insn.h>
#ifndef __ASSEMBLY__
......@@ -90,34 +91,55 @@ void apply_alternatives(void *start, size_t length);
.endm
/*
* Begin an alternative code sequence.
* Alternative sequences
*
* The code for the case where the capability is not present will be
* assembled and linked as normal. There are no restrictions on this
* code.
*
* The code for the case where the capability is present will be
* assembled into a special section to be used for dynamic patching.
* Code for that case must:
*
* 1. Be exactly the same length (in bytes) as the default code
* sequence.
*
* The code that follows this macro will be assembled and linked as
* normal. There are no restrictions on this code.
* 2. Not contain a branch target that is used outside of the
* alternative sequence it is defined in (branches into an
* alternative sequence are not fixed up).
*/
/*
* Begin an alternative code sequence.
*/
.macro alternative_if_not cap
.set .Lasm_alt_mode, 0
.pushsection .altinstructions, "a"
altinstruction_entry 661f, 663f, \cap, 662f-661f, 664f-663f
.popsection
661:
.endm
.macro alternative_if cap
.set .Lasm_alt_mode, 1
.pushsection .altinstructions, "a"
altinstruction_entry 663f, 661f, \cap, 664f-663f, 662f-661f
.popsection
.pushsection .altinstr_replacement, "ax"
.align 2 /* So GAS knows label 661 is suitably aligned */
661:
.endm
/*
* Provide the alternative code sequence.
*
* The code that follows this macro is assembled into a special
* section to be used for dynamic patching. Code that follows this
* macro must:
*
* 1. Be exactly the same length (in bytes) as the default code
* sequence.
*
* 2. Not contain a branch target that is used outside of the
* alternative sequence it is defined in (branches into an
* alternative sequence are not fixed up).
* Provide the other half of the alternative code sequence.
*/
.macro alternative_else
662: .pushsection .altinstr_replacement, "ax"
662:
.if .Lasm_alt_mode==0
.pushsection .altinstr_replacement, "ax"
.else
.popsection
.endif
663:
.endm
......@@ -125,11 +147,25 @@ void apply_alternatives(void *start, size_t length);
* Complete an alternative code sequence.
*/
.macro alternative_endif
664: .popsection
664:
.if .Lasm_alt_mode==0
.popsection
.endif
.org . - (664b-663b) + (662b-661b)
.org . - (662b-661b) + (664b-663b)
.endm
/*
* Provides a trivial alternative or default sequence consisting solely
* of NOPs. The number of NOPs is chosen automatically to match the
* previous case.
*/
.macro alternative_else_nop_endif
alternative_else
nops (662b-661b) / AARCH64_INSN_SIZE
alternative_endif
.endm
#define _ALTERNATIVE_CFG(insn1, insn2, cap, cfg, ...) \
alternative_insn insn1, insn2, cap, IS_ENABLED(cfg)
......
arch/arm64/include/asm/arch_timer.h
View file @ 7af8a0f8
......@@ -20,13 +20,55 @@
#define __ASM_ARCH_TIMER_H
#include <asm/barrier.h>
#include <asm/sysreg.h>
#include <linux/bug.h>
#include <linux/init.h>
#include <linux/jump_label.h>
#include <linux/types.h>
#include <clocksource/arm_arch_timer.h>
#if IS_ENABLED(CONFIG_FSL_ERRATUM_A008585)
extern struct static_key_false arch_timer_read_ool_enabled;
#define needs_fsl_a008585_workaround() \
static_branch_unlikely(&arch_timer_read_ool_enabled)
#else
#define needs_fsl_a008585_workaround() false
#endif
u32 __fsl_a008585_read_cntp_tval_el0(void);
u32 __fsl_a008585_read_cntv_tval_el0(void);
u64 __fsl_a008585_read_cntvct_el0(void);
/*
* The number of retries is an arbitrary value well beyond the highest number
* of iterations the loop has been observed to take.
*/
#define __fsl_a008585_read_reg(reg) ({ \
u64 _old, _new; \
int _retries = 200; \
\
do { \
_old = read_sysreg(reg); \
_new = read_sysreg(reg); \
_retries--; \
} while (unlikely(_old != _new) && _retries); \
\
WARN_ON_ONCE(!_retries); \
_new; \
})
#define arch_timer_reg_read_stable(reg) \
({ \
u64 _val; \
if (needs_fsl_a008585_workaround()) \
_val = __fsl_a008585_read_##reg(); \
else \
_val = read_sysreg(reg); \
_val; \
})
/*
* These register accessors are marked inline so the compiler can
* nicely work out which register we want, and chuck away the rest of
......@@ -38,19 +80,19 @@ void arch_timer_reg_write_cp15(int access, enum arch_timer_reg reg, u32 val)
if (access == ARCH_TIMER_PHYS_ACCESS) {
switch (reg) {
case ARCH_TIMER_REG_CTRL:
asm volatile("msr cntp_ctl_el0, %0" : : "r" (val));
write_sysreg(val, cntp_ctl_el0);
break;
case ARCH_TIMER_REG_TVAL:
asm volatile("msr cntp_tval_el0, %0" : : "r" (val));
write_sysreg(val, cntp_tval_el0);
break;
}
} else if (access == ARCH_TIMER_VIRT_ACCESS) {
switch (reg) {
case ARCH_TIMER_REG_CTRL:
asm volatile("msr cntv_ctl_el0, %0" : : "r" (val));
write_sysreg(val, cntv_ctl_el0);
break;
case ARCH_TIMER_REG_TVAL:
asm volatile("msr cntv_tval_el0, %0" : : "r" (val));
write_sysreg(val, cntv_tval_el0);
break;
}
}
......@@ -61,48 +103,38 @@ void arch_timer_reg_write_cp15(int access, enum arch_timer_reg reg, u32 val)
static __always_inline
u32 arch_timer_reg_read_cp15(int access, enum arch_timer_reg reg)
{
u32 val;
if (access == ARCH_TIMER_PHYS_ACCESS) {
switch (reg) {
case ARCH_TIMER_REG_CTRL:
asm volatile("mrs %0, cntp_ctl_el0" : "=r" (val));
break;
return read_sysreg(cntp_ctl_el0);
case ARCH_TIMER_REG_TVAL:
asm volatile("mrs %0, cntp_tval_el0" : "=r" (val));
break;
return arch_timer_reg_read_stable(cntp_tval_el0);
}
} else if (access == ARCH_TIMER_VIRT_ACCESS) {
switch (reg) {
case ARCH_TIMER_REG_CTRL:
asm volatile("mrs %0, cntv_ctl_el0" : "=r" (val));
break;
return read_sysreg(cntv_ctl_el0);
case ARCH_TIMER_REG_TVAL:
asm volatile("mrs %0, cntv_tval_el0" : "=r" (val));
break;
return arch_timer_reg_read_stable(cntv_tval_el0);
}
}
return val;
BUG();
}
static inline u32 arch_timer_get_cntfrq(void)
{
u32 val;
asm volatile("mrs %0, cntfrq_el0" : "=r" (val));
return val;
return read_sysreg(cntfrq_el0);
}
static inline u32 arch_timer_get_cntkctl(void)
{
u32 cntkctl;
asm volatile("mrs %0, cntkctl_el1" : "=r" (cntkctl));
return cntkctl;
return read_sysreg(cntkctl_el1);
}
static inline void arch_timer_set_cntkctl(u32 cntkctl)
{
asm volatile("msr cntkctl_el1, %0" : : "r" (cntkctl));
write_sysreg(cntkctl, cntkctl_el1);
}
static inline u64 arch_counter_get_cntpct(void)
......@@ -116,12 +148,8 @@ static inline u64 arch_counter_get_cntpct(void)
static inline u64 arch_counter_get_cntvct(void)
{
u64 cval;
isb();
asm volatile("mrs %0, cntvct_el0" : "=r" (cval));
return cval;
return arch_timer_reg_read_stable(cntvct_el0);
}
static inline int arch_timer_arch_init(void)
......
arch/arm64/include/asm/assembler.h
View file @ 7af8a0f8
......@@ -86,6 +86,15 @@
dmb \opt
.endm
/*
* NOP sequence
*/
.macro nops, num
.rept \num
nop
.endr
.endm
/*
* Emit an entry into the exception table
*/
......@@ -216,11 +225,26 @@ lr .req x30 // link register
.macro mmid, rd, rn
ldr \rd, [\rn, #MM_CONTEXT_ID]
.endm
/*
* read_ctr - read CTR_EL0. If the system has mismatched
* cache line sizes, provide the system wide safe value
* from arm64_ftr_reg_ctrel0.sys_val
*/
.macro read_ctr, reg
alternative_if_not ARM64_MISMATCHED_CACHE_LINE_SIZE
mrs \reg, ctr_el0 // read CTR
nop
alternative_else
ldr_l \reg, arm64_ftr_reg_ctrel0 + ARM64_FTR_SYSVAL
alternative_endif
.endm
/*
* dcache_line_size - get the minimum D-cache line size from the CTR register.
* raw_dcache_line_size - get the minimum D-cache line size on this CPU
* from the CTR register.
*/
.macro dcache_line_size, reg, tmp
.macro raw_dcache_line_size, reg, tmp
mrs \tmp, ctr_el0 // read CTR
ubfm \tmp, \tmp, #16, #19 // cache line size encoding
mov \reg, #4 // bytes per word
......@@ -228,15 +252,36 @@ lr .req x30 // link register
.endm
/*
* icache_line_size - get the minimum I-cache line size from the CTR register.
* dcache_line_size - get the safe D-cache line size across all CPUs
*/
.macro icache_line_size, reg, tmp
.macro dcache_line_size, reg, tmp
read_ctr \tmp
ubfm \tmp, \tmp, #16, #19 // cache line size encoding
mov \reg, #4 // bytes per word
lsl \reg, \reg, \tmp // actual cache line size
.endm
/*
* raw_icache_line_size - get the minimum I-cache line size on this CPU
* from the CTR register.
*/
.macro raw_icache_line_size, reg, tmp
mrs \tmp, ctr_el0 // read CTR
and \tmp, \tmp, #0xf // cache line size encoding
mov \reg, #4 // bytes per word
lsl \reg, \reg, \tmp // actual cache line size
.endm
/*
* icache_line_size - get the safe I-cache line size across all CPUs
*/
.macro icache_line_size, reg, tmp
read_ctr \tmp
and \tmp, \tmp, #0xf // cache line size encoding
mov \reg, #4 // bytes per word
lsl \reg, \reg, \tmp // actual cache line size
.endm
/*
* tcr_set_idmap_t0sz - update TCR.T0SZ so that we can load the ID map
*/
......
arch/arm64/include/asm/atomic_lse.h
View file @ 7af8a0f8
......@@ -86,8 +86,8 @@ static inline int atomic_add_return##name(int i, atomic_t *v) \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
__LL_SC_ATOMIC(add_return##name), \
__LL_SC_ATOMIC(add_return##name) \
__nops(1), \
/* LSE atomics */ \
" ldadd" #mb " %w[i], w30, %[v]\n" \
" add %w[i], %w[i], w30") \
......@@ -112,8 +112,8 @@ static inline void atomic_and(int i, atomic_t *v)
asm volatile(ARM64_LSE_ATOMIC_INSN(
/* LL/SC */
" nop\n"
__LL_SC_ATOMIC(and),
__LL_SC_ATOMIC(and)
__nops(1),
/* LSE atomics */
" mvn %w[i], %w[i]\n"
" stclr %w[i], %[v]")
......@@ -130,8 +130,8 @@ static inline int atomic_fetch_and##name(int i, atomic_t *v) \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
__LL_SC_ATOMIC(fetch_and##name), \
__LL_SC_ATOMIC(fetch_and##name) \
__nops(1), \
/* LSE atomics */ \
" mvn %w[i], %w[i]\n" \
" ldclr" #mb " %w[i], %w[i], %[v]") \
......@@ -156,8 +156,8 @@ static inline void atomic_sub(int i, atomic_t *v)
asm volatile(ARM64_LSE_ATOMIC_INSN(
/* LL/SC */
" nop\n"
__LL_SC_ATOMIC(sub),
__LL_SC_ATOMIC(sub)
__nops(1),
/* LSE atomics */
" neg %w[i], %w[i]\n"
" stadd %w[i], %[v]")
......@@ -174,9 +174,8 @@ static inline int atomic_sub_return##name(int i, atomic_t *v) \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
__LL_SC_ATOMIC(sub_return##name) \
" nop", \
__nops(2), \
/* LSE atomics */ \
" neg %w[i], %w[i]\n" \
" ldadd" #mb " %w[i], w30, %[v]\n" \
......@@ -203,8 +202,8 @@ static inline int atomic_fetch_sub##name(int i, atomic_t *v) \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
__LL_SC_ATOMIC(fetch_sub##name), \
__LL_SC_ATOMIC(fetch_sub##name) \
__nops(1), \
/* LSE atomics */ \
" neg %w[i], %w[i]\n" \
" ldadd" #mb " %w[i], %w[i], %[v]") \
......@@ -284,8 +283,8 @@ static inline long atomic64_add_return##name(long i, atomic64_t *v) \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
__LL_SC_ATOMIC64(add_return##name), \
__LL_SC_ATOMIC64(add_return##name) \
__nops(1), \
/* LSE atomics */ \
" ldadd" #mb " %[i], x30, %[v]\n" \
" add %[i], %[i], x30") \
......@@ -310,8 +309,8 @@ static inline void atomic64_and(long i, atomic64_t *v)
asm volatile(ARM64_LSE_ATOMIC_INSN(
/* LL/SC */
" nop\n"
__LL_SC_ATOMIC64(and),
__LL_SC_ATOMIC64(and)
__nops(1),
/* LSE atomics */
" mvn %[i], %[i]\n"
" stclr %[i], %[v]")
......@@ -328,8 +327,8 @@ static inline long atomic64_fetch_and##name(long i, atomic64_t *v) \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
__LL_SC_ATOMIC64(fetch_and##name), \
__LL_SC_ATOMIC64(fetch_and##name) \
__nops(1), \
/* LSE atomics */ \
" mvn %[i], %[i]\n" \
" ldclr" #mb " %[i], %[i], %[v]") \
......@@ -354,8 +353,8 @@ static inline void atomic64_sub(long i, atomic64_t *v)
asm volatile(ARM64_LSE_ATOMIC_INSN(
/* LL/SC */
" nop\n"
__LL_SC_ATOMIC64(sub),
__LL_SC_ATOMIC64(sub)
__nops(1),
/* LSE atomics */
" neg %[i], %[i]\n"
" stadd %[i], %[v]")
......@@ -372,9 +371,8 @@ static inline long atomic64_sub_return##name(long i, atomic64_t *v) \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
__LL_SC_ATOMIC64(sub_return##name) \
" nop", \
__nops(2), \
/* LSE atomics */ \
" neg %[i], %[i]\n" \
" ldadd" #mb " %[i], x30, %[v]\n" \
......@@ -401,8 +399,8 @@ static inline long atomic64_fetch_sub##name(long i, atomic64_t *v) \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
__LL_SC_ATOMIC64(fetch_sub##name), \
__LL_SC_ATOMIC64(fetch_sub##name) \
__nops(1), \
/* LSE atomics */ \
" neg %[i], %[i]\n" \
" ldadd" #mb " %[i], %[i], %[v]") \
......@@ -426,13 +424,8 @@ static inline long atomic64_dec_if_positive(atomic64_t *v)
asm volatile(ARM64_LSE_ATOMIC_INSN(
/* LL/SC */
" nop\n"
__LL_SC_ATOMIC64(dec_if_positive)
" nop\n"
" nop\n"
" nop\n"
" nop\n"
" nop",
__nops(6),
/* LSE atomics */
"1: ldr x30, %[v]\n"
" subs %[ret], x30, #1\n"
......@@ -464,9 +457,8 @@ static inline unsigned long __cmpxchg_case_##name(volatile void *ptr, \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
__LL_SC_CMPXCHG(name) \
" nop", \
__LL_SC_CMPXCHG(name) \
__nops(2), \
/* LSE atomics */ \
" mov " #w "30, %" #w "[old]\n" \
" cas" #mb #sz "\t" #w "30, %" #w "[new], %[v]\n" \
......@@ -517,10 +509,8 @@ static inline long __cmpxchg_double##name(unsigned long old1, \
\
asm volatile(ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
" nop\n" \
" nop\n" \
" nop\n" \
__LL_SC_CMPXCHG_DBL(name), \
__LL_SC_CMPXCHG_DBL(name) \
__nops(3), \
/* LSE atomics */ \
" casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\
" eor %[old1], %[old1], %[oldval1]\n" \
......
arch/arm64/include/asm/barrier.h
View file @ 7af8a0f8
......@@ -20,6 +20,9 @@
#ifndef __ASSEMBLY__
#define __nops(n) ".rept " #n "\nnop\n.endr\n"
#define nops(n) asm volatile(__nops(n))
#define sev() asm volatile("sev" : : : "memory")
#define wfe() asm volatile("wfe" : : : "memory")
#define wfi() asm volatile("wfi" : : : "memory")
......
arch/arm64/include/asm/cacheflush.h
View file @ 7af8a0f8
......@@ -68,6 +68,7 @@
extern void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end);
extern void flush_icache_range(unsigned long start, unsigned long end);
extern void __flush_dcache_area(void *addr, size_t len);
extern void __clean_dcache_area_poc(void *addr, size_t len);
extern void __clean_dcache_area_pou(void *addr, size_t len);
extern long __flush_cache_user_range(unsigned long start, unsigned long end);
......@@ -85,7 +86,7 @@ static inline void flush_cache_page(struct vm_area_struct *vma,
*/
extern void __dma_map_area(const void *, size_t, int);
extern void __dma_unmap_area(const void *, size_t, int);
extern void __dma_flush_range(const void *, const void *);
extern void __dma_flush_area(const void *, size_t);
/*
* Copy user data from/to a page which is mapped into a different
......
arch/arm64/include/asm/clocksource.h 0 → 100644
View file @ 7af8a0f8
#ifndef _ASM_CLOCKSOURCE_H
#define _ASM_CLOCKSOURCE_H
struct arch_clocksource_data {
bool vdso_direct; /* Usable for direct VDSO access? */
};
#endif
arch/arm64/include/asm/cmpxchg.h
View file @ 7af8a0f8
......@@ -43,10 +43,8 @@ static inline unsigned long __xchg_case_##name(unsigned long x, \
" cbnz %w1, 1b\n" \
" " #mb, \
/* LSE atomics */ \
" nop\n" \
" nop\n" \
" swp" #acq_lse #rel #sz "\t%" #w "3, %" #w "0, %2\n" \
" nop\n" \
__nops(3) \
" " #nop_lse) \
: "=&r" (ret), "=&r" (tmp), "+Q" (*(u8 *)ptr) \
: "r" (x) \
......
arch/arm64/include/asm/cpufeature.h
View file @ 7af8a0f8
......@@ -9,6 +9,8 @@
#ifndef __ASM_CPUFEATURE_H
#define __ASM_CPUFEATURE_H
#include <linux/jump_label.h>
#include <asm/hwcap.h>
#include <asm/sysreg.h>
......@@ -37,8 +39,9 @@
#define ARM64_WORKAROUND_CAVIUM_27456 12
#define ARM64_HAS_32BIT_EL0 13
#define ARM64_HYP_OFFSET_LOW 14
#define ARM64_MISMATCHED_CACHE_LINE_SIZE 15
#define ARM64_NCAPS 15
#define ARM64_NCAPS 16
#ifndef __ASSEMBLY__
......@@ -63,7 +66,7 @@ struct arm64_ftr_bits {
enum ftr_type type;
u8 shift;
u8 width;
s64 safe_val; /* safe value for discrete features */
s64 safe_val; /* safe value for FTR_EXACT features */
};
/*
......@@ -72,13 +75,14 @@ struct arm64_ftr_bits {
* @sys_val Safe value across the CPUs (system view)
*/
struct arm64_ftr_reg {
u32 sys_id;
const char *name;
u64 strict_mask;
u64 sys_val;
struct arm64_ftr_bits *ftr_bits;
const char *name;
u64 strict_mask;
u64 sys_val;
const struct arm64_ftr_bits *ftr_bits;
};
extern struct arm64_ftr_reg arm64_ftr_reg_ctrel0;
/* scope of capability check */
enum {
SCOPE_SYSTEM,
......@@ -109,6 +113,7 @@ struct arm64_cpu_capabilities {
};
extern DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS);
extern struct static_key_false cpu_hwcap_keys[ARM64_NCAPS];
bool this_cpu_has_cap(unsigned int cap);
......@@ -121,16 +126,21 @@ static inline bool cpus_have_cap(unsigned int num)
{
if (num >= ARM64_NCAPS)
return false;
return test_bit(num, cpu_hwcaps);
if (__builtin_constant_p(num))
return static_branch_unlikely(&cpu_hwcap_keys[num]);
else
return test_bit(num, cpu_hwcaps);
}
static inline void cpus_set_cap(unsigned int num)
{
if (num >= ARM64_NCAPS)
if (num >= ARM64_NCAPS) {
pr_warn("Attempt to set an illegal CPU capability (%d >= %d)\n",
num, ARM64_NCAPS);
else
} else {
__set_bit(num, cpu_hwcaps);
static_branch_enable(&cpu_hwcap_keys[num]);
}
}
static inline int __attribute_const__
......@@ -157,7 +167,7 @@ cpuid_feature_extract_unsigned_field(u64 features, int field)
return cpuid_feature_extract_unsigned_field_width(features, field, 4);
}
static inline u64 arm64_ftr_mask(struct arm64_ftr_bits *ftrp)
static inline u64 arm64_ftr_mask(const struct arm64_ftr_bits *ftrp)
{
return (u64)GENMASK(ftrp->shift + ftrp->width - 1, ftrp->shift);
}
......@@ -170,7 +180,7 @@ cpuid_feature_extract_field(u64 features, int field, bool sign)
cpuid_feature_extract_unsigned_field(features, field);
}
static inline s64 arm64_ftr_value(struct arm64_ftr_bits *ftrp, u64 val)
static inline s64 arm64_ftr_value(const struct arm64_ftr_bits *ftrp, u64 val)
{
return (s64)cpuid_feature_extract_field(val, ftrp->shift, ftrp->sign);
}
......@@ -193,11 +203,11 @@ void __init setup_cpu_features(void);
void update_cpu_capabilities(const struct arm64_cpu_capabilities *caps,
const char *info);
void enable_cpu_capabilities(const struct arm64_cpu_capabilities *caps);
void check_local_cpu_errata(void);
void __init enable_errata_workarounds(void);
void check_local_cpu_capabilities(void);
void verify_local_cpu_errata(void);
void verify_local_cpu_capabilities(void);
void update_cpu_errata_workarounds(void);
void __init enable_errata_workarounds(void);
void verify_local_cpu_errata_workarounds(void);
u64 read_system_reg(u32 id);
......
arch/arm64/include/asm/cputype.h
View file @ 7af8a0f8
......@@ -93,11 +93,7 @@
#include <asm/sysreg.h>
#define read_cpuid(reg) ({ \
u64 __val; \
asm("mrs_s %0, " __stringify(SYS_ ## reg) : "=r" (__val)); \
__val; \
})
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
/*
* The CPU ID never changes at run time, so we might as well tell the
......
arch/arm64/include/asm/dcc.h
View file @ 7af8a0f8
......@@ -21,21 +21,16 @@
#define __ASM_DCC_H
#include <asm/barrier.h>
#include <asm/sysreg.h>
static inline u32 __dcc_getstatus(void)
{
u32 ret;
asm volatile("mrs %0, mdccsr_el0" : "=r" (ret));
return ret;
return read_sysreg(mdccsr_el0);
}
static inline char __dcc_getchar(void)
{
char c;
asm volatile("mrs %0, dbgdtrrx_el0" : "=r" (c));
char c = read_sysreg(dbgdtrrx_el0);
isb();
return c;
......@@ -47,8 +42,7 @@ static inline void __dcc_putchar(char c)
* The typecast is to make absolutely certain that 'c' is
* zero-extended.
*/
asm volatile("msr dbgdtrtx_el0, %0"
: : "r" ((unsigned long)(unsigned char)c));
write_sysreg((unsigned char)c, dbgdtrtx_el0);
isb();
}
......
arch/arm64/include/asm/esr.h
View file @ 7af8a0f8
......@@ -78,6 +78,23 @@
#define ESR_ELx_IL (UL(1) << 25)
#define ESR_ELx_ISS_MASK (ESR_ELx_IL - 1)
/* ISS field definitions shared by different classes */
#define ESR_ELx_WNR (UL(1) << 6)
/* Shared ISS field definitions for Data/Instruction aborts */
#define ESR_ELx_EA (UL(1) << 9)
#define ESR_ELx_S1PTW (UL(1) << 7)
/* Shared ISS fault status code(IFSC/DFSC) for Data/Instruction aborts */
#define ESR_ELx_FSC (0x3F)
#define ESR_ELx_FSC_TYPE (0x3C)
#define ESR_ELx_FSC_EXTABT (0x10)
#define ESR_ELx_FSC_ACCESS (0x08)
#define ESR_ELx_FSC_FAULT (0x04)
#define ESR_ELx_FSC_PERM (0x0C)
/* ISS field definitions for Data Aborts */
#define ESR_ELx_ISV (UL(1) << 24)
#define ESR_ELx_SAS_SHIFT (22)
#define ESR_ELx_SAS (UL(3) << ESR_ELx_SAS_SHIFT)
......@@ -86,16 +103,9 @@
#define ESR_ELx_SRT_MASK (UL(0x1F) << ESR_ELx_SRT_SHIFT)
#define ESR_ELx_SF (UL(1) << 15)
#define ESR_ELx_AR (UL(1) << 14)
#define ESR_ELx_EA (UL(1) << 9)
#define ESR_ELx_CM (UL(1) << 8)
#define ESR_ELx_S1PTW (UL(1) << 7)
#define ESR_ELx_WNR (UL(1) << 6)
#define ESR_ELx_FSC (0x3F)
#define ESR_ELx_FSC_TYPE (0x3C)
#define ESR_ELx_FSC_EXTABT (0x10)
#define ESR_ELx_FSC_ACCESS (0x08)
#define ESR_ELx_FSC_FAULT (0x04)
#define ESR_ELx_FSC_PERM (0x0C)
/* ISS field definitions for exceptions taken in to Hyp */
#define ESR_ELx_CV (UL(1) << 24)
#define ESR_ELx_COND_SHIFT (20)
#define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT)
......@@ -109,6 +119,62 @@
((ESR_ELx_EC_BRK64 << ESR_ELx_EC_SHIFT) | ESR_ELx_IL | \
((imm) & 0xffff))
/* ISS field definitions for System instruction traps */
#define ESR_ELx_SYS64_ISS_RES0_SHIFT 22
#define ESR_ELx_SYS64_ISS_RES0_MASK (UL(0x7) << ESR_ELx_SYS64_ISS_RES0_SHIFT)
#define ESR_ELx_SYS64_ISS_DIR_MASK 0x1
#define ESR_ELx_SYS64_ISS_DIR_READ 0x1
#define ESR_ELx_SYS64_ISS_DIR_WRITE 0x0
#define ESR_ELx_SYS64_ISS_RT_SHIFT 5
#define ESR_ELx_SYS64_ISS_RT_MASK (UL(0x1f) << ESR_ELx_SYS64_ISS_RT_SHIFT)
#define ESR_ELx_SYS64_ISS_CRM_SHIFT 1
#define ESR_ELx_SYS64_ISS_CRM_MASK (UL(0xf) << ESR_ELx_SYS64_ISS_CRM_SHIFT)
#define ESR_ELx_SYS64_ISS_CRN_SHIFT 10
#define ESR_ELx_SYS64_ISS_CRN_MASK (UL(0xf) << ESR_ELx_SYS64_ISS_CRN_SHIFT)
#define ESR_ELx_SYS64_ISS_OP1_SHIFT 14
#define ESR_ELx_SYS64_ISS_OP1_MASK (UL(0x7) << ESR_ELx_SYS64_ISS_OP1_SHIFT)
#define ESR_ELx_SYS64_ISS_OP2_SHIFT 17
#define ESR_ELx_SYS64_ISS_OP2_MASK (UL(0x7) << ESR_ELx_SYS64_ISS_OP2_SHIFT)
#define ESR_ELx_SYS64_ISS_OP0_SHIFT 20
#define ESR_ELx_SYS64_ISS_OP0_MASK (UL(0x3) << ESR_ELx_SYS64_ISS_OP0_SHIFT)
#define ESR_ELx_SYS64_ISS_SYS_MASK (ESR_ELx_SYS64_ISS_OP0_MASK | \
ESR_ELx_SYS64_ISS_OP1_MASK | \
ESR_ELx_SYS64_ISS_OP2_MASK | \
ESR_ELx_SYS64_ISS_CRN_MASK | \
ESR_ELx_SYS64_ISS_CRM_MASK)
#define ESR_ELx_SYS64_ISS_SYS_VAL(op0, op1, op2, crn, crm) \
(((op0) << ESR_ELx_SYS64_ISS_OP0_SHIFT) | \
((op1) << ESR_ELx_SYS64_ISS_OP1_SHIFT) | \
((op2) << ESR_ELx_SYS64_ISS_OP2_SHIFT) | \
((crn) << ESR_ELx_SYS64_ISS_CRN_SHIFT) | \
((crm) << ESR_ELx_SYS64_ISS_CRM_SHIFT))
#define ESR_ELx_SYS64_ISS_SYS_OP_MASK (ESR_ELx_SYS64_ISS_SYS_MASK | \
ESR_ELx_SYS64_ISS_DIR_MASK)
/*
* User space cache operations have the following sysreg encoding
* in System instructions.
* op0=1, op1=3, op2=1, crn=7, crm={ 5, 10, 11, 14 }, WRITE (L=0)
*/
#define ESR_ELx_SYS64_ISS_CRM_DC_CIVAC 14
#define ESR_ELx_SYS64_ISS_CRM_DC_CVAU 11
#define ESR_ELx_SYS64_ISS_CRM_DC_CVAC 10
#define ESR_ELx_SYS64_ISS_CRM_IC_IVAU 5
#define ESR_ELx_SYS64_ISS_EL0_CACHE_OP_MASK (ESR_ELx_SYS64_ISS_OP0_MASK | \
ESR_ELx_SYS64_ISS_OP1_MASK | \
ESR_ELx_SYS64_ISS_OP2_MASK | \
ESR_ELx_SYS64_ISS_CRN_MASK | \
ESR_ELx_SYS64_ISS_DIR_MASK)
#define ESR_ELx_SYS64_ISS_EL0_CACHE_OP_VAL \
(ESR_ELx_SYS64_ISS_SYS_VAL(1, 3, 1, 7, 0) | \
ESR_ELx_SYS64_ISS_DIR_WRITE)
#define ESR_ELx_SYS64_ISS_SYS_CTR ESR_ELx_SYS64_ISS_SYS_VAL(3, 3, 1, 0, 0)
#define ESR_ELx_SYS64_ISS_SYS_CTR_READ (ESR_ELx_SYS64_ISS_SYS_CTR | \
ESR_ELx_SYS64_ISS_DIR_READ)
#ifndef __ASSEMBLY__
#include <asm/types.h>
......
arch/arm64/include/asm/hw_breakpoint.h
View file @ 7af8a0f8
......@@ -18,6 +18,7 @@
#include <asm/cputype.h>
#include <asm/cpufeature.h>
#include <asm/sysreg.h>
#include <asm/virt.h>
#ifdef __KERNEL__
......@@ -98,18 +99,18 @@ static inline void decode_ctrl_reg(u32 reg,
#define AARCH64_DBG_REG_WCR (AARCH64_DBG_REG_WVR + ARM_MAX_WRP)
/* Debug register names. */
#define AARCH64_DBG_REG_NAME_BVR "bvr"
#define AARCH64_DBG_REG_NAME_BCR "bcr"
#define AARCH64_DBG_REG_NAME_WVR "wvr"
#define AARCH64_DBG_REG_NAME_WCR "wcr"
#define AARCH64_DBG_REG_NAME_BVR bvr
#define AARCH64_DBG_REG_NAME_BCR bcr
#define AARCH64_DBG_REG_NAME_WVR wvr
#define AARCH64_DBG_REG_NAME_WCR wcr
/* Accessor macros for the debug registers. */
#define AARCH64_DBG_READ(N, REG, VAL) do {\
asm volatile("mrs %0, dbg" REG #N "_el1" : "=r" (VAL));\
VAL = read_sysreg(dbg##REG##N##_el1);\
} while (0)
#define AARCH64_DBG_WRITE(N, REG, VAL) do {\
asm volatile("msr dbg" REG #N "_el1, %0" :: "r" (VAL));\
write_sysreg(VAL, dbg##REG##N##_el1);\
} while (0)
struct task_struct;
......@@ -141,8 +142,6 @@ static inline void ptrace_hw_copy_thread(struct task_struct *task)
}
#endif
extern struct pmu perf_ops_bp;
/* Determine number of BRP registers available. */
static inline int get_num_brps(void)
{
......
arch/arm64/include/asm/insn.h
View file @ 7af8a0f8
......@@ -246,7 +246,8 @@ static __always_inline bool aarch64_insn_is_##abbr(u32 code) \
static __always_inline u32 aarch64_insn_get_##abbr##_value(void) \
{ return (val); }
__AARCH64_INSN_FUNCS(adr_adrp, 0x1F000000, 0x10000000)
__AARCH64_INSN_FUNCS(adr, 0x9F000000, 0x10000000)
__AARCH64_INSN_FUNCS(adrp, 0x9F000000, 0x90000000)
__AARCH64_INSN_FUNCS(prfm_lit, 0xFF000000, 0xD8000000)
__AARCH64_INSN_FUNCS(str_reg, 0x3FE0EC00, 0x38206800)
__AARCH64_INSN_FUNCS(ldr_reg, 0x3FE0EC00, 0x38606800)
......@@ -318,6 +319,11 @@ __AARCH64_INSN_FUNCS(msr_reg, 0xFFF00000, 0xD5100000)
bool aarch64_insn_is_nop(u32 insn);
bool aarch64_insn_is_branch_imm(u32 insn);
static inline bool aarch64_insn_is_adr_adrp(u32 insn)
{
return aarch64_insn_is_adr(insn) || aarch64_insn_is_adrp(insn);
}
int aarch64_insn_read(void *addr, u32 *insnp);
int aarch64_insn_write(void *addr, u32 insn);
enum aarch64_insn_encoding_class aarch64_get_insn_class(u32 insn);
......@@ -398,6 +404,9 @@ int aarch64_insn_patch_text_nosync(void *addr, u32 insn);
int aarch64_insn_patch_text_sync(void *addrs[], u32 insns[], int cnt);
int aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt);
s32 aarch64_insn_adrp_get_offset(u32 insn);
u32 aarch64_insn_adrp_set_offset(u32 insn, s32 offset);
bool aarch32_insn_is_wide(u32 insn);
#define A32_RN_OFFSET 16
......
arch/arm64/include/asm/io.h
View file @ 7af8a0f8
......@@ -40,25 +40,25 @@
#define __raw_writeb __raw_writeb
static inline void __raw_writeb(u8 val, volatile void __iomem *addr)
{
asm volatile("strb %w0, [%1]" : : "r" (val), "r" (addr));
asm volatile("strb %w0, [%1]" : : "rZ" (val), "r" (addr));
}
#define __raw_writew __raw_writew
static inline void __raw_writew(u16 val, volatile void __iomem *addr)
{
asm volatile("strh %w0, [%1]" : : "r" (val), "r" (addr));
asm volatile("strh %w0, [%1]" : : "rZ" (val), "r" (addr));
}
#define __raw_writel __raw_writel
static inline void __raw_writel(u32 val, volatile void __iomem *addr)
{
asm volatile("str %w0, [%1]" : : "r" (val), "r" (addr));
asm volatile("str %w0, [%1]" : : "rZ" (val), "r" (addr));
}
#define __raw_writeq __raw_writeq
static inline void __raw_writeq(u64 val, volatile void __iomem *addr)
{
asm volatile("str %0, [%1]" : : "r" (val), "r" (addr));
asm volatile("str %x0, [%1]" : : "rZ" (val), "r" (addr));
}
#define __raw_readb __raw_readb
......@@ -184,17 +184,6 @@ extern void __iomem *ioremap_cache(phys_addr_t phys_addr, size_t size);
#define iowrite32be(v,p) ({ __iowmb(); __raw_writel((__force __u32)cpu_to_be32(v), p); })
#define iowrite64be(v,p) ({ __iowmb(); __raw_writeq((__force __u64)cpu_to_be64(v), p); })
/*
* Convert a physical pointer to a virtual kernel pointer for /dev/mem
* access
*/
#define xlate_dev_mem_ptr(p) __va(p)
/*
* Convert a virtual cached pointer to an uncached pointer
*/
#define xlate_dev_kmem_ptr(p) p
#include <asm-generic/io.h>
/*
......
arch/arm64/include/asm/kvm_mmu.h
View file @ 7af8a0f8
......@@ -99,14 +99,10 @@
.macro kern_hyp_va reg
alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
and \reg, \reg, #HYP_PAGE_OFFSET_HIGH_MASK
alternative_else
nop
alternative_endif
alternative_if_not ARM64_HYP_OFFSET_LOW
nop
alternative_else
alternative_else_nop_endif
alternative_if ARM64_HYP_OFFSET_LOW
and \reg, \reg, #HYP_PAGE_OFFSET_LOW_MASK
alternative_endif
alternative_else_nop_endif
.endm
#else
......
arch/arm64/include/asm/memory.h
View file @ 7af8a0f8
......@@ -214,7 +214,7 @@ static inline void *phys_to_virt(phys_addr_t x)
#ifndef CONFIG_SPARSEMEM_VMEMMAP
#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
#define _virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
#else
#define __virt_to_pgoff(kaddr) (((u64)(kaddr) & ~PAGE_OFFSET) / PAGE_SIZE * sizeof(struct page))
#define __page_to_voff(kaddr) (((u64)(page) & ~VMEMMAP_START) * PAGE_SIZE / sizeof(struct page))
......@@ -222,11 +222,15 @@ static inline void *phys_to_virt(phys_addr_t x)
#define page_to_virt(page) ((void *)((__page_to_voff(page)) | PAGE_OFFSET))
#define virt_to_page(vaddr) ((struct page *)((__virt_to_pgoff(vaddr)) | VMEMMAP_START))
#define virt_addr_valid(kaddr) pfn_valid((((u64)(kaddr) & ~PAGE_OFFSET) \
#define _virt_addr_valid(kaddr) pfn_valid((((u64)(kaddr) & ~PAGE_OFFSET) \
+ PHYS_OFFSET) >> PAGE_SHIFT)
#endif
#endif
#define _virt_addr_is_linear(kaddr) (((u64)(kaddr)) >= PAGE_OFFSET)
#define virt_addr_valid(kaddr) (_virt_addr_is_linear(kaddr) && \
_virt_addr_valid(kaddr))
#include <asm-generic/memory_model.h>
#endif
arch/arm64/include/asm/mmu_context.h
View file @ 7af8a0f8
......@@ -27,22 +27,17 @@
#include <asm-generic/mm_hooks.h>
#include <asm/cputype.h>
#include <asm/pgtable.h>
#include <asm/sysreg.h>
#include <asm/tlbflush.h>
#ifdef CONFIG_PID_IN_CONTEXTIDR
static inline void contextidr_thread_switch(struct task_struct *next)
{
asm(
" msr contextidr_el1, %0\n"
" isb"
:
: "r" (task_pid_nr(next)));
}
#else
static inline void contextidr_thread_switch(struct task_struct *next)
{
if (!IS_ENABLED(CONFIG_PID_IN_CONTEXTIDR))
return;
write_sysreg(task_pid_nr(next), contextidr_el1);
isb();
}
#endif
/*
* Set TTBR0 to empty_zero_page. No translations will be possible via TTBR0.
......@@ -51,11 +46,8 @@ static inline void cpu_set_reserved_ttbr0(void)
{
unsigned long ttbr = virt_to_phys(empty_zero_page);
asm(
" msr ttbr0_el1, %0 // set TTBR0\n"
" isb"
:
: "r" (ttbr));
write_sysreg(ttbr, ttbr0_el1);
isb();
}
/*
......@@ -81,13 +73,11 @@ static inline void __cpu_set_tcr_t0sz(unsigned long t0sz)
if (!__cpu_uses_extended_idmap())
return;
asm volatile (
" mrs %0, tcr_el1 ;"
" bfi %0, %1, %2, %3 ;"
" msr tcr_el1, %0 ;"
" isb"
: "=&r" (tcr)
: "r"(t0sz), "I"(TCR_T0SZ_OFFSET), "I"(TCR_TxSZ_WIDTH));
tcr = read_sysreg(tcr_el1);
tcr &= ~TCR_T0SZ_MASK;
tcr |= t0sz << TCR_T0SZ_OFFSET;
write_sysreg(tcr, tcr_el1);
isb();
}
#define cpu_set_default_tcr_t0sz() __cpu_set_tcr_t0sz(TCR_T0SZ(VA_BITS))
......
arch/arm64/include/asm/pgtable-hwdef.h
View file @ 7af8a0f8
......@@ -208,6 +208,7 @@
#define TCR_T1SZ(x) ((UL(64) - (x)) << TCR_T1SZ_OFFSET)
#define TCR_TxSZ(x) (TCR_T0SZ(x) | TCR_T1SZ(x))
#define TCR_TxSZ_WIDTH 6
#define TCR_T0SZ_MASK (((UL(1) << TCR_TxSZ_WIDTH) - 1) << TCR_T0SZ_OFFSET)
#define TCR_IRGN0_SHIFT 8
#define TCR_IRGN0_MASK (UL(3) << TCR_IRGN0_SHIFT)
......
arch/arm64/include/asm/pgtable-prot.h
View file @ 7af8a0f8
......@@ -70,12 +70,13 @@
#define PAGE_COPY_EXEC __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN)
#define PAGE_READONLY __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN)
#define PAGE_READONLY_EXEC __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN)
#define PAGE_EXECONLY __pgprot(_PAGE_DEFAULT | PTE_NG | PTE_PXN)
#define __P000 PAGE_NONE
#define __P001 PAGE_READONLY
#define __P010 PAGE_COPY
#define __P011 PAGE_COPY
#define __P100 PAGE_READONLY_EXEC
#define __P100 PAGE_EXECONLY
#define __P101 PAGE_READONLY_EXEC
#define __P110 PAGE_COPY_EXEC
#define __P111 PAGE_COPY_EXEC
......@@ -84,7 +85,7 @@
#define __S001 PAGE_READONLY
#define __S010 PAGE_SHARED
#define __S011 PAGE_SHARED
#define __S100 PAGE_READONLY_EXEC
#define __S100 PAGE_EXECONLY
#define __S101 PAGE_READONLY_EXEC
#define __S110 PAGE_SHARED_EXEC
#define __S111 PAGE_SHARED_EXEC
......
arch/arm64/include/asm/pgtable.h
View file @ 7af8a0f8
......@@ -73,7 +73,7 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
#define pte_write(pte) (!!(pte_val(pte) & PTE_WRITE))
#define pte_exec(pte) (!(pte_val(pte) & PTE_UXN))
#define pte_cont(pte) (!!(pte_val(pte) & PTE_CONT))
#define pte_user(pte) (!!(pte_val(pte) & PTE_USER))
#define pte_ng(pte) (!!(pte_val(pte) & PTE_NG))
#ifdef CONFIG_ARM64_HW_AFDBM
#define pte_hw_dirty(pte) (pte_write(pte) && !(pte_val(pte) & PTE_RDONLY))
......@@ -84,8 +84,8 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
#define pte_dirty(pte) (pte_sw_dirty(pte) || pte_hw_dirty(pte))
#define pte_valid(pte) (!!(pte_val(pte) & PTE_VALID))
#define pte_valid_not_user(pte) \
((pte_val(pte) & (PTE_VALID | PTE_USER)) == PTE_VALID)
#define pte_valid_global(pte) \
((pte_val(pte) & (PTE_VALID | PTE_NG)) == PTE_VALID)
#define pte_valid_young(pte) \
((pte_val(pte) & (PTE_VALID | PTE_AF)) == (PTE_VALID | PTE_AF))
......@@ -155,6 +155,16 @@ static inline pte_t pte_mknoncont(pte_t pte)
return clear_pte_bit(pte, __pgprot(PTE_CONT));
}
static inline pte_t pte_clear_rdonly(pte_t pte)
{
return clear_pte_bit(pte, __pgprot(PTE_RDONLY));
}
static inline pte_t pte_mkpresent(pte_t pte)
{
return set_pte_bit(pte, __pgprot(PTE_VALID));
}
static inline pmd_t pmd_mkcont(pmd_t pmd)
{
return __pmd(pmd_val(pmd) | PMD_SECT_CONT);
......@@ -168,7 +178,7 @@ static inline void set_pte(pte_t *ptep, pte_t pte)
* Only if the new pte is valid and kernel, otherwise TLB maintenance
* or update_mmu_cache() have the necessary barriers.
*/
if (pte_valid_not_user(pte)) {
if (pte_valid_global(pte)) {
dsb(ishst);
isb();
}
......@@ -202,7 +212,7 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,
pte_val(pte) &= ~PTE_RDONLY;
else
pte_val(pte) |= PTE_RDONLY;
if (pte_user(pte) && pte_exec(pte) && !pte_special(pte))
if (pte_ng(pte) && pte_exec(pte) && !pte_special(pte))
__sync_icache_dcache(pte, addr);
}
......
arch/arm64/include/asm/processor.h
View file @ 7af8a0f8
......@@ -37,7 +37,6 @@
#include <asm/ptrace.h>
#include <asm/types.h>
#ifdef __KERNEL__
#define STACK_TOP_MAX TASK_SIZE_64
#ifdef CONFIG_COMPAT
#define AARCH32_VECTORS_BASE 0xffff0000
......@@ -49,7 +48,6 @@
extern phys_addr_t arm64_dma_phys_limit;
#define ARCH_LOW_ADDRESS_LIMIT (arm64_dma_phys_limit - 1)
#endif /* __KERNEL__ */
struct debug_info {
/* Have we suspended stepping by a debugger? */
......
arch/arm64/include/asm/sections.h 0 → 100644
View file @ 7af8a0f8
/*
* Copyright (C) 2016 ARM Limited
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __ASM_SECTIONS_H
#define __ASM_SECTIONS_H
#include <asm-generic/sections.h>
extern char __alt_instructions[], __alt_instructions_end[];
extern char __exception_text_start[], __exception_text_end[];
extern char __hibernate_exit_text_start[], __hibernate_exit_text_end[];
extern char __hyp_idmap_text_start[], __hyp_idmap_text_end[];
extern char __hyp_text_start[], __hyp_text_end[];
extern char __idmap_text_start[], __idmap_text_end[];
extern char __irqentry_text_start[], __irqentry_text_end[];
extern char __mmuoff_data_start[], __mmuoff_data_end[];
#endif /* __ASM_SECTIONS_H */
arch/arm64/include/asm/spinlock.h
View file @ 7af8a0f8
......@@ -66,8 +66,7 @@ static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
ARM64_LSE_ATOMIC_INSN(
/* LL/SC */
" stxr %w1, %w0, %2\n"
" nop\n"
" nop\n",
__nops(2),
/* LSE atomics */
" mov %w1, %w0\n"
" cas %w0, %w0, %2\n"
......@@ -99,9 +98,7 @@ static inline void arch_spin_lock(arch_spinlock_t *lock)
/* LSE atomics */
" mov %w2, %w5\n"
" ldadda %w2, %w0, %3\n"
" nop\n"
" nop\n"
" nop\n"
__nops(3)
)
/* Did we get the lock? */
......@@ -165,8 +162,8 @@ static inline void arch_spin_unlock(arch_spinlock_t *lock)
" stlrh %w1, %0",
/* LSE atomics */
" mov %w1, #1\n"
" nop\n"
" staddlh %w1, %0")
" staddlh %w1, %0\n"
__nops(1))
: "=Q" (lock->owner), "=&r" (tmp)
:
: "memory");
......@@ -212,7 +209,7 @@ static inline void arch_write_lock(arch_rwlock_t *rw)
" cbnz %w0, 1b\n"
" stxr %w0, %w2, %1\n"
" cbnz %w0, 2b\n"
" nop",
__nops(1),
/* LSE atomics */
"1: mov %w0, wzr\n"
"2: casa %w0, %w2, %1\n"
......@@ -241,8 +238,7 @@ static inline int arch_write_trylock(arch_rwlock_t *rw)
/* LSE atomics */
" mov %w0, wzr\n"
" casa %w0, %w2, %1\n"
" nop\n"
" nop")
__nops(2))
: "=&r" (tmp), "+Q" (rw->lock)
: "r" (0x80000000)
: "memory");
......@@ -290,8 +286,8 @@ static inline void arch_read_lock(arch_rwlock_t *rw)
" add %w0, %w0, #1\n"
" tbnz %w0, #31, 1b\n"
" stxr %w1, %w0, %2\n"
" nop\n"
" cbnz %w1, 2b",
" cbnz %w1, 2b\n"
__nops(1),
/* LSE atomics */
"1: wfe\n"
"2: ldxr %w0, %2\n"
......@@ -317,9 +313,8 @@ static inline void arch_read_unlock(arch_rwlock_t *rw)
" cbnz %w1, 1b",
/* LSE atomics */
" movn %w0, #0\n"
" nop\n"
" nop\n"
" staddl %w0, %2")
" staddl %w0, %2\n"
__nops(2))
: "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock)
:
: "memory");
......@@ -344,7 +339,7 @@ static inline int arch_read_trylock(arch_rwlock_t *rw)
" tbnz %w1, #31, 1f\n"
" casa %w0, %w1, %2\n"
" sbc %w1, %w1, %w0\n"
" nop\n"
__nops(1)
"1:")
: "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock)
:
......
arch/arm64/include/asm/suspend.h
View file @ 7af8a0f8
......@@ -47,4 +47,7 @@ int swsusp_arch_resume(void);
int arch_hibernation_header_save(void *addr, unsigned int max_size);
int arch_hibernation_header_restore(void *addr);
/* Used to resume on the CPU we hibernated on */
int hibernate_resume_nonboot_cpu_disable(void);
#endif
arch/arm64/include/asm/sysreg.h
View file @ 7af8a0f8
......@@ -100,6 +100,7 @@
/* SCTLR_EL1 specific flags. */
#define SCTLR_EL1_UCI (1 << 26)
#define SCTLR_EL1_SPAN (1 << 23)
#define SCTLR_EL1_UCT (1 << 15)
#define SCTLR_EL1_SED (1 << 8)
#define SCTLR_EL1_CP15BEN (1 << 5)
......@@ -253,16 +254,6 @@ asm(
" .endm\n"
);
static inline void config_sctlr_el1(u32 clear, u32 set)
{
u32 val;
asm volatile("mrs %0, sctlr_el1" : "=r" (val));
val &= ~clear;
val |= set;
asm volatile("msr sctlr_el1, %0" : : "r" (val));
}
/*
* Unlike read_cpuid, calls to read_sysreg are never expected to be
* optimized away or replaced with synthetic values.
......@@ -273,12 +264,41 @@ static inline void config_sctlr_el1(u32 clear, u32 set)
__val; \
})
/*
* The "Z" constraint normally means a zero immediate, but when combined with
* the "%x0" template means XZR.
*/
#define write_sysreg(v, r) do { \
u64 __val = (u64)v; \
asm volatile("msr " __stringify(r) ", %0" \
: : "r" (__val)); \
asm volatile("msr " __stringify(r) ", %x0" \
: : "rZ" (__val)); \
} while (0)
/*
* For registers without architectural names, or simply unsupported by
* GAS.
*/
#define read_sysreg_s(r) ({ \
u64 __val; \
asm volatile("mrs_s %0, " __stringify(r) : "=r" (__val)); \
__val; \
})
#define write_sysreg_s(v, r) do { \
u64 __val = (u64)v; \
asm volatile("msr_s " __stringify(r) ", %0" : : "rZ" (__val)); \
} while (0)
static inline void config_sctlr_el1(u32 clear, u32 set)
{
u32 val;
val = read_sysreg(sctlr_el1);
val &= ~clear;
val |= set;
write_sysreg(val, sctlr_el1);
}
#endif
#endif /* __ASM_SYSREG_H */
arch/arm64/include/asm/system_misc.h
View file @ 7af8a0f8
......@@ -56,12 +56,6 @@ extern void (*arm_pm_restart)(enum reboot_mode reboot_mode, const char *cmd);
__show_ratelimited; \
})
#define UDBG_UNDEFINED (1 << 0)
#define UDBG_SYSCALL (1 << 1)
#define UDBG_BADABORT (1 << 2)
#define UDBG_SEGV (1 << 3)
#define UDBG_BUS (1 << 4)
#endif /* __ASSEMBLY__ */
#endif /* __ASM_SYSTEM_MISC_H */
arch/arm64/include/asm/thread_info.h
View file @ 7af8a0f8
......@@ -75,6 +75,9 @@ static inline struct thread_info *current_thread_info(void) __attribute_const__;
/*
* struct thread_info can be accessed directly via sp_el0.
*
* We don't use read_sysreg() as we want the compiler to cache the value where
* possible.
*/
static inline struct thread_info *current_thread_info(void)
{
......
arch/arm64/include/asm/tlbflush.h
View file @ 7af8a0f8
......@@ -24,6 +24,24 @@
#include <linux/sched.h>
#include <asm/cputype.h>
/*
* Raw TLBI operations.
*
* Where necessary, use the __tlbi() macro to avoid asm()
* boilerplate. Drivers and most kernel code should use the TLB
* management routines in preference to the macro below.
*
* The macro can be used as __tlbi(op) or __tlbi(op, arg), depending
* on whether a particular TLBI operation takes an argument or
* not. The macros handles invoking the asm with or without the
* register argument as appropriate.
*/
#define __TLBI_0(op, arg) asm ("tlbi " #op)
#define __TLBI_1(op, arg) asm ("tlbi " #op ", %0" : : "r" (arg))
#define __TLBI_N(op, arg, n, ...) __TLBI_##n(op, arg)
#define __tlbi(op, ...) __TLBI_N(op, ##__VA_ARGS__, 1, 0)
/*
* TLB Management
* ==============
......@@ -66,7 +84,7 @@
static inline void local_flush_tlb_all(void)
{
dsb(nshst);
asm("tlbi vmalle1");
__tlbi(vmalle1);
dsb(nsh);
isb();
}
......@@ -74,7 +92,7 @@ static inline void local_flush_tlb_all(void)
static inline void flush_tlb_all(void)
{
dsb(ishst);
asm("tlbi vmalle1is");
__tlbi(vmalle1is);
dsb(ish);
isb();
}
......@@ -84,7 +102,7 @@ static inline void flush_tlb_mm(struct mm_struct *mm)
unsigned long asid = ASID(mm) << 48;
dsb(ishst);
asm("tlbi aside1is, %0" : : "r" (asid));
__tlbi(aside1is, asid);
dsb(ish);
}
......@@ -94,7 +112,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma,
unsigned long addr = uaddr >> 12 | (ASID(vma->vm_mm) << 48);
dsb(ishst);
asm("tlbi vale1is, %0" : : "r" (addr));
__tlbi(vale1is, addr);
dsb(ish);
}
......@@ -122,9 +140,9 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma,
dsb(ishst);
for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) {
if (last_level)
asm("tlbi vale1is, %0" : : "r"(addr));
__tlbi(vale1is, addr);
else
asm("tlbi vae1is, %0" : : "r"(addr));
__tlbi(vae1is, addr);
}
dsb(ish);
}
......@@ -149,7 +167,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
dsb(ishst);
for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12))
asm("tlbi vaae1is, %0" : : "r"(addr));
__tlbi(vaae1is, addr);
dsb(ish);
isb();
}
......@@ -163,7 +181,7 @@ static inline void __flush_tlb_pgtable(struct mm_struct *mm,
{
unsigned long addr = uaddr >> 12 | (ASID(mm) << 48);
asm("tlbi vae1is, %0" : : "r" (addr));
__tlbi(vae1is, addr);
dsb(ish);
}
......
arch/arm64/include/asm/traps.h
View file @ 7af8a0f8
......@@ -19,6 +19,7 @@
#define __ASM_TRAP_H
#include <linux/list.h>
#include <asm/sections.h>
struct pt_regs;
......@@ -39,9 +40,6 @@ void arm64_notify_segfault(struct pt_regs *regs, unsigned long addr);
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
static inline int __in_irqentry_text(unsigned long ptr)
{
extern char __irqentry_text_start[];
extern char __irqentry_text_end[];
return ptr >= (unsigned long)&__irqentry_text_start &&
ptr < (unsigned long)&__irqentry_text_end;
}
......@@ -54,8 +52,6 @@ static inline int __in_irqentry_text(unsigned long ptr)
static inline int in_exception_text(unsigned long ptr)
{
extern char __exception_text_start[];
extern char __exception_text_end[];
int in;
in = ptr >= (unsigned long)&__exception_text_start &&
......
arch/arm64/include/asm/virt.h
View file @ 7af8a0f8
......@@ -45,6 +45,8 @@
#ifndef __ASSEMBLY__
#include <asm/ptrace.h>
#include <asm/sections.h>
#include <asm/sysreg.h>
/*
* __boot_cpu_mode records what mode CPUs were booted in.
......@@ -75,10 +77,7 @@ static inline bool is_hyp_mode_mismatched(void)
static inline bool is_kernel_in_hyp_mode(void)
{
u64 el;
asm("mrs %0, CurrentEL" : "=r" (el));
return el == CurrentEL_EL2;
return read_sysreg(CurrentEL) == CurrentEL_EL2;
}
#ifdef CONFIG_ARM64_VHE
......@@ -87,14 +86,6 @@ extern void verify_cpu_run_el(void);
static inline void verify_cpu_run_el(void) {}
#endif
/* The section containing the hypervisor idmap text */
extern char __hyp_idmap_text_start[];
extern char __hyp_idmap_text_end[];
/* The section containing the hypervisor text */
extern char __hyp_text_start[];
extern char __hyp_text_end[];
#endif /* __ASSEMBLY__ */
#endif /* ! __ASM__VIRT_H */
arch/arm64/kernel/Makefile
View file @ 7af8a0f8
......@@ -10,6 +10,8 @@ CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_insn.o = -pg
CFLAGS_REMOVE_return_address.o = -pg
CFLAGS_setup.o = -DUTS_MACHINE='"$(UTS_MACHINE)"'
# Object file lists.
arm64-obj-y := debug-monitors.o entry.o irq.o fpsimd.o \
entry-fpsimd.o process.o ptrace.o setup.o signal.o \
......
arch/arm64/kernel/acpi_numa.c
View file @ 7af8a0f8
......@@ -105,8 +105,10 @@ int __init arm64_acpi_numa_init(void)
int ret;
ret = acpi_numa_init();
if (ret)
if (ret) {
pr_info("Failed to initialise from firmware\n");
return ret;
}
return srat_disabled() ? -EINVAL : 0;
}
arch/arm64/kernel/alternative.c
View file @ 7af8a0f8
......@@ -25,14 +25,13 @@
#include <asm/alternative.h>
#include <asm/cpufeature.h>
#include <asm/insn.h>
#include <asm/sections.h>
#include <linux/stop_machine.h>
#define __ALT_PTR(a,f) (u32 *)((void *)&(a)->f + (a)->f)
#define ALT_ORIG_PTR(a) __ALT_PTR(a, orig_offset)
#define ALT_REPL_PTR(a) __ALT_PTR(a, alt_offset)
extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
struct alt_region {
struct alt_instr *begin;
struct alt_instr *end;
......@@ -59,6 +58,8 @@ static bool branch_insn_requires_update(struct alt_instr *alt, unsigned long pc)
BUG();
}
#define align_down(x, a) ((unsigned long)(x) & ~(((unsigned long)(a)) - 1))
static u32 get_alt_insn(struct alt_instr *alt, u32 *insnptr, u32 *altinsnptr)
{
u32 insn;
......@@ -80,6 +81,25 @@ static u32 get_alt_insn(struct alt_instr *alt, u32 *insnptr, u32 *altinsnptr)
offset = target - (unsigned long)insnptr;
insn = aarch64_set_branch_offset(insn, offset);
}
} else if (aarch64_insn_is_adrp(insn)) {
s32 orig_offset, new_offset;
unsigned long target;
/*
* If we're replacing an adrp instruction, which uses PC-relative
* immediate addressing, adjust the offset to reflect the new
* PC. adrp operates on 4K aligned addresses.
*/
orig_offset = aarch64_insn_adrp_get_offset(insn);
target = align_down(altinsnptr, SZ_4K) + orig_offset;
new_offset = target - align_down(insnptr, SZ_4K);
insn = aarch64_insn_adrp_set_offset(insn, new_offset);
} else if (aarch64_insn_uses_literal(insn)) {
/*
* Disallow patching unhandled instructions using PC relative
* literal addresses
*/
BUG();
}
return insn;
......@@ -124,8 +144,8 @@ static int __apply_alternatives_multi_stop(void *unused)
{
static int patched = 0;
struct alt_region region = {
.begin = __alt_instructions,
.end = __alt_instructions_end,
.begin = (struct alt_instr *)__alt_instructions,
.end = (struct alt_instr *)__alt_instructions_end,
};
/* We always have a CPU 0 at this point (__init) */
......
arch/arm64/kernel/asm-offsets.c
View file @ 7af8a0f8
......@@ -23,6 +23,7 @@
#include <linux/dma-mapping.h>
#include <linux/kvm_host.h>
#include <linux/suspend.h>
#include <asm/cpufeature.h>
#include <asm/thread_info.h>
#include <asm/memory.h>
#include <asm/smp_plat.h>
......@@ -145,5 +146,6 @@ int main(void)
DEFINE(HIBERN_PBE_ORIG, offsetof(struct pbe, orig_address));
DEFINE(HIBERN_PBE_ADDR, offsetof(struct pbe, address));
DEFINE(HIBERN_PBE_NEXT, offsetof(struct pbe, next));
DEFINE(ARM64_FTR_SYSVAL, offsetof(struct arm64_ftr_reg, sys_val));
return 0;
}
arch/arm64/kernel/cacheinfo.c
View file @ 7af8a0f8
......@@ -39,7 +39,7 @@ static inline enum cache_type get_cache_type(int level)
if (level > MAX_CACHE_LEVEL)
return CACHE_TYPE_NOCACHE;
asm volatile ("mrs %x0, clidr_el1" : "=r" (clidr));
clidr = read_sysreg(clidr_el1);
return CLIDR_CTYPE(clidr, level);
}
......@@ -55,11 +55,9 @@ u64 __attribute_const__ cache_get_ccsidr(u64 csselr)
WARN_ON(preemptible());
/* Put value into CSSELR */
asm volatile("msr csselr_el1, %x0" : : "r" (csselr));
write_sysreg(csselr, csselr_el1);
isb();
/* Read result out of CCSIDR */
asm volatile("mrs %x0, ccsidr_el1" : "=r" (ccsidr));
ccsidr = read_sysreg(ccsidr_el1);
return ccsidr;
}
......
arch/arm64/kernel/cpu_errata.c
View file @ 7af8a0f8
......@@ -30,6 +30,21 @@ is_affected_midr_range(const struct arm64_cpu_capabilities *entry, int scope)
entry->midr_range_max);
}
static bool
has_mismatched_cache_line_size(const struct arm64_cpu_capabilities *entry,
int scope)
{
WARN_ON(scope != SCOPE_LOCAL_CPU || preemptible());
return (read_cpuid_cachetype() & arm64_ftr_reg_ctrel0.strict_mask) !=
(arm64_ftr_reg_ctrel0.sys_val & arm64_ftr_reg_ctrel0.strict_mask);
}
static void cpu_enable_trap_ctr_access(void *__unused)
{
/* Clear SCTLR_EL1.UCT */
config_sctlr_el1(SCTLR_EL1_UCT, 0);
}
#define MIDR_RANGE(model, min, max) \
.def_scope = SCOPE_LOCAL_CPU, \
.matches = is_affected_midr_range, \
......@@ -107,6 +122,13 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
MIDR_RANGE(MIDR_THUNDERX_81XX, 0x00, 0x00),
},
#endif
{
.desc = "Mismatched cache line size",
.capability = ARM64_MISMATCHED_CACHE_LINE_SIZE,
.matches = has_mismatched_cache_line_size,
.def_scope = SCOPE_LOCAL_CPU,
.enable = cpu_enable_trap_ctr_access,
},
{
}
};
......@@ -116,7 +138,7 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
* and the related information is freed soon after. If the new CPU requires
* an errata not detected at boot, fail this CPU.
*/
void verify_local_cpu_errata(void)
void verify_local_cpu_errata_workarounds(void)
{
const struct arm64_cpu_capabilities *caps = arm64_errata;
......@@ -131,7 +153,7 @@ void verify_local_cpu_errata(void)
}
}
void check_local_cpu_errata(void)
void update_cpu_errata_workarounds(void)
{
update_cpu_capabilities(arm64_errata, "enabling workaround for");
}
......
arch/arm64/kernel/cpu_ops.c
View file @ 7af8a0f8
......@@ -17,6 +17,7 @@
*/
#include <linux/acpi.h>
#include <linux/cache.h>
#include <linux/errno.h>
#include <linux/of.h>
#include <linux/string.h>
......@@ -28,7 +29,7 @@ extern const struct cpu_operations smp_spin_table_ops;
extern const struct cpu_operations acpi_parking_protocol_ops;
extern const struct cpu_operations cpu_psci_ops;
const struct cpu_operations *cpu_ops[NR_CPUS];
const struct cpu_operations *cpu_ops[NR_CPUS] __ro_after_init;
static const struct cpu_operations *dt_supported_cpu_ops[] __initconst = {
&smp_spin_table_ops,
......
arch/arm64/kernel/cpufeature.c
View file @ 7af8a0f8
This diff is collapsed.
arch/arm64/kernel/cpuinfo.c
View file @ 7af8a0f8
......@@ -363,8 +363,6 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info)
}
cpuinfo_detect_icache_policy(info);
check_local_cpu_errata();
}
void cpuinfo_store_cpu(void)
......
arch/arm64/kernel/debug-monitors.c
View file @ 7af8a0f8
......@@ -46,16 +46,14 @@ static void mdscr_write(u32 mdscr)
{
unsigned long flags;
local_dbg_save(flags);
asm volatile("msr mdscr_el1, %0" :: "r" (mdscr));
write_sysreg(mdscr, mdscr_el1);
local_dbg_restore(flags);
}
NOKPROBE_SYMBOL(mdscr_write);
static u32 mdscr_read(void)
{
u32 mdscr;
asm volatile("mrs %0, mdscr_el1" : "=r" (mdscr));
return mdscr;
return read_sysreg(mdscr_el1);
}
NOKPROBE_SYMBOL(mdscr_read);
......@@ -132,36 +130,18 @@ NOKPROBE_SYMBOL(disable_debug_monitors);
/*
* OS lock clearing.
*/
static void clear_os_lock(void *unused)
static int clear_os_lock(unsigned int cpu)
{
asm volatile("msr oslar_el1, %0" : : "r" (0));
}
static int os_lock_notify(struct notifier_block *self,
unsigned long action, void *data)
{
if ((action & ~CPU_TASKS_FROZEN) == CPU_ONLINE)
clear_os_lock(NULL);
return NOTIFY_OK;
write_sysreg(0, oslar_el1);
isb();
return 0;
}
static struct notifier_block os_lock_nb = {
.notifier_call = os_lock_notify,
};
static int debug_monitors_init(void)
{
cpu_notifier_register_begin();
/* Clear the OS lock. */
on_each_cpu(clear_os_lock, NULL, 1);
isb();
/* Register hotplug handler. */
__register_cpu_notifier(&os_lock_nb);
cpu_notifier_register_done();
return 0;
return cpuhp_setup_state(CPUHP_AP_ARM64_DEBUG_MONITORS_STARTING,
"CPUHP_AP_ARM64_DEBUG_MONITORS_STARTING",
clear_os_lock, NULL);
}
postcore_initcall(debug_monitors_init);
......@@ -254,7 +234,7 @@ static int single_step_handler(unsigned long addr, unsigned int esr,
return 0;
if (user_mode(regs)) {
send_user_sigtrap(TRAP_HWBKPT);
send_user_sigtrap(TRAP_TRACE);
/*
* ptrace will disable single step unless explicitly
......@@ -382,7 +362,7 @@ NOKPROBE_SYMBOL(aarch32_break_handler);
static int __init debug_traps_init(void)
{
hook_debug_fault_code(DBG_ESR_EVT_HWSS, single_step_handler, SIGTRAP,
TRAP_HWBKPT, "single-step handler");
TRAP_TRACE, "single-step handler");
hook_debug_fault_code(DBG_ESR_EVT_BRK, brk_handler, SIGTRAP,
TRAP_BRKPT, "ptrace BRK handler");
return 0;
......@@ -435,8 +415,10 @@ NOKPROBE_SYMBOL(kernel_active_single_step);
/* ptrace API */
void user_enable_single_step(struct task_struct *task)
{
set_ti_thread_flag(task_thread_info(task), TIF_SINGLESTEP);
set_regs_spsr_ss(task_pt_regs(task));
struct thread_info *ti = task_thread_info(task);
if (!test_and_set_ti_thread_flag(ti, TIF_SINGLESTEP))
set_regs_spsr_ss(task_pt_regs(task));
}
NOKPROBE_SYMBOL(user_enable_single_step);
......
arch/arm64/kernel/entry.S
View file @ 7af8a0f8
......@@ -104,7 +104,7 @@
str x20, [sp, #S_ORIG_ADDR_LIMIT]
mov x20, #TASK_SIZE_64
str x20, [tsk, #TI_ADDR_LIMIT]
ALTERNATIVE(nop, SET_PSTATE_UAO(0), ARM64_HAS_UAO, CONFIG_ARM64_UAO)
/* No need to reset PSTATE.UAO, hardware's already set it to 0 for us */
.endif /* \el == 0 */
mrs x22, elr_el1
mrs x23, spsr_el1
......@@ -150,13 +150,7 @@
ldr x23, [sp, #S_SP] // load return stack pointer
msr sp_el0, x23
#ifdef CONFIG_ARM64_ERRATUM_845719
alternative_if_not ARM64_WORKAROUND_845719
nop
nop
#ifdef CONFIG_PID_IN_CONTEXTIDR
nop
#endif
alternative_else
alternative_if ARM64_WORKAROUND_845719
tbz x22, #4, 1f
#ifdef CONFIG_PID_IN_CONTEXTIDR
mrs x29, contextidr_el1
......@@ -165,7 +159,7 @@ alternative_else
msr contextidr_el1, xzr
#endif
1:
alternative_endif
alternative_else_nop_endif
#endif
.endif
msr elr_el1, x21 // set up the return data
......@@ -707,18 +701,13 @@ ret_fast_syscall_trace:
* Ok, we need to do extra processing, enter the slow path.
*/
work_pending:
tbnz x1, #TIF_NEED_RESCHED, work_resched
/* TIF_SIGPENDING, TIF_NOTIFY_RESUME or TIF_FOREIGN_FPSTATE case */
mov x0, sp // 'regs'
enable_irq // enable interrupts for do_notify_resume()
bl do_notify_resume
b ret_to_user
work_resched:
#ifdef CONFIG_TRACE_IRQFLAGS
bl trace_hardirqs_off // the IRQs are off here, inform the tracing code
bl trace_hardirqs_on // enabled while in userspace
#endif
bl schedule
ldr x1, [tsk, #TI_FLAGS] // re-check for single-step
b finish_ret_to_user
/*
* "slow" syscall return path.
*/
......@@ -727,6 +716,7 @@ ret_to_user:
ldr x1, [tsk, #TI_FLAGS]
and x2, x1, #_TIF_WORK_MASK
cbnz x2, work_pending
finish_ret_to_user:
enable_step_tsk x1, x2
kernel_exit 0
ENDPROC(ret_to_user)
......
arch/arm64/kernel/head.S
View file @ 7af8a0f8
This diff is collapsed.
arch/arm64/kernel/hibernate-asm.S
View file @ 7af8a0f8
......@@ -36,8 +36,8 @@
.macro break_before_make_ttbr_switch zero_page, page_table
msr ttbr1_el1, \zero_page
isb
tlbi vmalle1is
dsb ish
tlbi vmalle1
dsb nsh
msr ttbr1_el1, \page_table
isb
.endm
......@@ -96,7 +96,7 @@ ENTRY(swsusp_arch_suspend_exit)
add x1, x10, #PAGE_SIZE
/* Clean the copied page to PoU - based on flush_icache_range() */
dcache_line_size x2, x3
raw_dcache_line_size x2, x3
sub x3, x2, #1
bic x4, x10, x3
2: dc cvau, x4 /* clean D line / unified line */
......
arch/arm64/kernel/hibernate.c
View file @ 7af8a0f8
......@@ -15,9 +15,9 @@
* License terms: GNU General Public License (GPL) version 2
*/
#define pr_fmt(x) "hibernate: " x
#include <linux/cpu.h>
#include <linux/kvm_host.h>
#include <linux/mm.h>
#include <linux/notifier.h>
#include <linux/pm.h>
#include <linux/sched.h>
#include <linux/suspend.h>
......@@ -26,6 +26,7 @@
#include <asm/barrier.h>
#include <asm/cacheflush.h>
#include <asm/cputype.h>
#include <asm/irqflags.h>
#include <asm/memory.h>
#include <asm/mmu_context.h>
......@@ -34,6 +35,7 @@
#include <asm/pgtable-hwdef.h>
#include <asm/sections.h>
#include <asm/smp.h>
#include <asm/smp_plat.h>
#include <asm/suspend.h>
#include <asm/sysreg.h>
#include <asm/virt.h>
......@@ -54,18 +56,18 @@ extern int in_suspend;
/* Do we need to reset el2? */
#define el2_reset_needed() (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
/*
* Start/end of the hibernate exit code, this must be copied to a 'safe'
* location in memory, and executed from there.
*/
extern char __hibernate_exit_text_start[], __hibernate_exit_text_end[];
/* temporary el2 vectors in the __hibernate_exit_text section. */
extern char hibernate_el2_vectors[];
/* hyp-stub vectors, used to restore el2 during resume from hibernate. */
extern char __hyp_stub_vectors[];
/*
* The logical cpu number we should resume on, initialised to a non-cpu
* number.
*/
static int sleep_cpu = -EINVAL;
/*
* Values that may not change over hibernate/resume. We put the build number
* and date in here so that we guarantee not to resume with a different
......@@ -88,6 +90,8 @@ static struct arch_hibernate_hdr {
* re-configure el2.
*/
phys_addr_t __hyp_stub_vectors;
u64 sleep_cpu_mpidr;
} resume_hdr;
static inline void arch_hdr_invariants(struct arch_hibernate_hdr_invariants *i)
......@@ -130,12 +134,22 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size)
else
hdr->__hyp_stub_vectors = 0;
/* Save the mpidr of the cpu we called cpu_suspend() on... */
if (sleep_cpu < 0) {
pr_err("Failing to hibernate on an unkown CPU.\n");
return -ENODEV;
}
hdr->sleep_cpu_mpidr = cpu_logical_map(sleep_cpu);
pr_info("Hibernating on CPU %d [mpidr:0x%llx]\n", sleep_cpu,
hdr->sleep_cpu_mpidr);
return 0;
}
EXPORT_SYMBOL(arch_hibernation_header_save);
int arch_hibernation_header_restore(void *addr)
{
int ret;
struct arch_hibernate_hdr_invariants invariants;
struct arch_hibernate_hdr *hdr = addr;
......@@ -145,6 +159,24 @@ int arch_hibernation_header_restore(void *addr)
return -EINVAL;
}
sleep_cpu = get_logical_index(hdr->sleep_cpu_mpidr);
pr_info("Hibernated on CPU %d [mpidr:0x%llx]\n", sleep_cpu,
hdr->sleep_cpu_mpidr);
if (sleep_cpu < 0) {
pr_crit("Hibernated on a CPU not known to this kernel!\n");
sleep_cpu = -EINVAL;
return -EINVAL;
}
if (!cpu_online(sleep_cpu)) {
pr_info("Hibernated on a CPU that is offline! Bringing CPU up.\n");
ret = cpu_up(sleep_cpu);
if (ret) {
pr_err("Failed to bring hibernate-CPU up!\n");
sleep_cpu = -EINVAL;
return ret;
}
}
resume_hdr = *hdr;
return 0;
......@@ -241,6 +273,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
return rc;
}
#define dcache_clean_range(start, end) __flush_dcache_area(start, (end - start))
int swsusp_arch_suspend(void)
{
......@@ -256,10 +289,16 @@ int swsusp_arch_suspend(void)
local_dbg_save(flags);
if (__cpu_suspend_enter(&state)) {
sleep_cpu = smp_processor_id();
ret = swsusp_save();
} else {
/* Clean kernel to PoC for secondary core startup */
__flush_dcache_area(LMADDR(KERNEL_START), KERNEL_END - KERNEL_START);
/* Clean kernel core startup/idle code to PoC*/
dcache_clean_range(__mmuoff_data_start, __mmuoff_data_end);
dcache_clean_range(__idmap_text_start, __idmap_text_end);
/* Clean kvm setup code to PoC? */
if (el2_reset_needed())
dcache_clean_range(__hyp_idmap_text_start, __hyp_idmap_text_end);
/*
* Tell the hibernation core that we've just restored
......@@ -267,6 +306,7 @@ int swsusp_arch_suspend(void)
*/
in_suspend = 0;
sleep_cpu = -EINVAL;
__cpu_suspend_exit();
}
......@@ -275,6 +315,33 @@ int swsusp_arch_suspend(void)
return ret;
}
static void _copy_pte(pte_t *dst_pte, pte_t *src_pte, unsigned long addr)
{
pte_t pte = *src_pte;
if (pte_valid(pte)) {
/*
* Resume will overwrite areas that may be marked
* read only (code, rodata). Clear the RDONLY bit from
* the temporary mappings we use during restore.
*/
set_pte(dst_pte, pte_clear_rdonly(pte));
} else if (debug_pagealloc_enabled() && !pte_none(pte)) {
/*
* debug_pagealloc will removed the PTE_VALID bit if
* the page isn't in use by the resume kernel. It may have
* been in use by the original kernel, in which case we need
* to put it back in our copy to do the restore.
*
* Before marking this entry valid, check the pfn should
* be mapped.
*/
BUG_ON(!pfn_valid(pte_pfn(pte)));
set_pte(dst_pte, pte_mkpresent(pte_clear_rdonly(pte)));
}
}
static int copy_pte(pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long start,
unsigned long end)
{
......@@ -290,13 +357,7 @@ static int copy_pte(pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long start,
src_pte = pte_offset_kernel(src_pmd, start);
do {
if (!pte_none(*src_pte))
/*
* Resume will overwrite areas that may be marked
* read only (code, rodata). Clear the RDONLY bit from
* the temporary mappings we use during restore.
*/
set_pte(dst_pte, __pte(pte_val(*src_pte) & ~PTE_RDONLY));
_copy_pte(dst_pte, src_pte, addr);
} while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end);
return 0;
......@@ -483,27 +544,12 @@ int swsusp_arch_resume(void)
return rc;
}
static int check_boot_cpu_online_pm_callback(struct notifier_block *nb,
unsigned long action, void *ptr)
int hibernate_resume_nonboot_cpu_disable(void)
{
if (action == PM_HIBERNATION_PREPARE &&
cpumask_first(cpu_online_mask) != 0) {
pr_warn("CPU0 is offline.\n");
return notifier_from_errno(-ENODEV);
if (sleep_cpu < 0) {
pr_err("Failing to resume from hibernate on an unkown CPU.\n");
return -ENODEV;
}
return NOTIFY_OK;
}
static int __init check_boot_cpu_online_init(void)
{
/*
* Set this pm_notifier callback with a lower priority than
* cpu_hotplug_pm_callback, so that cpu_hotplug_pm_callback will be
* called earlier to disable cpu hotplug before the cpu online check.
*/
pm_notifier(check_boot_cpu_online_pm_callback, -INT_MAX);
return 0;
return freeze_secondary_cpus(sleep_cpu);
}
core_initcall(check_boot_cpu_online_init);
arch/arm64/kernel/hw_breakpoint.c
View file @ 7af8a0f8
......@@ -857,7 +857,7 @@ void hw_breakpoint_thread_switch(struct task_struct *next)
/*
* CPU initialisation.
*/
static void hw_breakpoint_reset(void *unused)
static int hw_breakpoint_reset(unsigned int cpu)
{
int i;
struct perf_event **slots;
......@@ -888,28 +888,14 @@ static void hw_breakpoint_reset(void *unused)
write_wb_reg(AARCH64_DBG_REG_WVR, i, 0UL);
}
}
}
static int hw_breakpoint_reset_notify(struct notifier_block *self,
unsigned long action,
void *hcpu)
{
if ((action & ~CPU_TASKS_FROZEN) == CPU_ONLINE) {
local_irq_disable();
hw_breakpoint_reset(NULL);
local_irq_enable();
}
return NOTIFY_OK;
return 0;
}
static struct notifier_block hw_breakpoint_reset_nb = {
.notifier_call = hw_breakpoint_reset_notify,
};
#ifdef CONFIG_CPU_PM
extern void cpu_suspend_set_dbg_restorer(void (*hw_bp_restore)(void *));
extern void cpu_suspend_set_dbg_restorer(int (*hw_bp_restore)(unsigned int));
#else
static inline void cpu_suspend_set_dbg_restorer(void (*hw_bp_restore)(void *))
static inline void cpu_suspend_set_dbg_restorer(int (*hw_bp_restore)(unsigned int))
{
}
#endif
......@@ -919,36 +905,34 @@ static inline void cpu_suspend_set_dbg_restorer(void (*hw_bp_restore)(void *))
*/
static int __init arch_hw_breakpoint_init(void)
{
int ret;
core_num_brps = get_num_brps();
core_num_wrps = get_num_wrps();
pr_info("found %d breakpoint and %d watchpoint registers.\n",
core_num_brps, core_num_wrps);
cpu_notifier_register_begin();
/*
* Reset the breakpoint resources. We assume that a halting
* debugger will leave the world in a nice state for us.
*/
smp_call_function(hw_breakpoint_reset, NULL, 1);
hw_breakpoint_reset(NULL);
/* Register debug fault handlers. */
hook_debug_fault_code(DBG_ESR_EVT_HWBP, breakpoint_handler, SIGTRAP,
TRAP_HWBKPT, "hw-breakpoint handler");
hook_debug_fault_code(DBG_ESR_EVT_HWWP, watchpoint_handler, SIGTRAP,
TRAP_HWBKPT, "hw-watchpoint handler");
/* Register hotplug notifier. */
__register_cpu_notifier(&hw_breakpoint_reset_nb);
cpu_notifier_register_done();
/*
* Reset the breakpoint resources. We assume that a halting
* debugger will leave the world in a nice state for us.
*/
ret = cpuhp_setup_state(CPUHP_AP_PERF_ARM_HW_BREAKPOINT_STARTING,
"CPUHP_AP_PERF_ARM_HW_BREAKPOINT_STARTING",
hw_breakpoint_reset, NULL);
if (ret)
pr_err("failed to register CPU hotplug notifier: %d\n", ret);
/* Register cpu_suspend hw breakpoint restore hook */
cpu_suspend_set_dbg_restorer(hw_breakpoint_reset);
return 0;
return ret;
}
arch_initcall(arch_hw_breakpoint_init);
......
arch/arm64/kernel/insn.c
View file @ 7af8a0f8
......@@ -96,7 +96,7 @@ static void __kprobes *patch_map(void *addr, int fixmap)
if (module && IS_ENABLED(CONFIG_DEBUG_SET_MODULE_RONX))
page = vmalloc_to_page(addr);
else if (!module && IS_ENABLED(CONFIG_DEBUG_RODATA))
else if (!module)
page = pfn_to_page(PHYS_PFN(__pa(addr)));
else
return addr;
......@@ -1202,6 +1202,19 @@ u32 aarch64_set_branch_offset(u32 insn, s32 offset)
BUG();
}
s32 aarch64_insn_adrp_get_offset(u32 insn)
{
BUG_ON(!aarch64_insn_is_adrp(insn));
return aarch64_insn_decode_immediate(AARCH64_INSN_IMM_ADR, insn) << 12;
}
u32 aarch64_insn_adrp_set_offset(u32 insn, s32 offset)
{
BUG_ON(!aarch64_insn_is_adrp(insn));
return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_ADR, insn,
offset >> 12);
}
/*
* Extract the Op/CR data from a msr/mrs instruction.
*/
......
arch/arm64/kernel/kaslr.c
View file @ 7af8a0f8
......@@ -6,6 +6,7 @@
* published by the Free Software Foundation.
*/
#include <linux/cache.h>
#include <linux/crc32.h>
#include <linux/init.h>
#include <linux/libfdt.h>
......@@ -20,7 +21,7 @@
#include <asm/pgtable.h>
#include <asm/sections.h>
u64 __read_mostly module_alloc_base;
u64 __ro_after_init module_alloc_base;
u16 __initdata memstart_offset_seed;
static __init u64 get_kaslr_seed(void *fdt)
......
arch/arm64/kernel/perf_event.c
View file @ 7af8a0f8
......@@ -24,6 +24,7 @@
#include <asm/sysreg.h>
#include <asm/virt.h>
#include <linux/acpi.h>
#include <linux/of.h>
#include <linux/perf/arm_pmu.h>
#include <linux/platform_device.h>
......@@ -190,13 +191,23 @@
#define ARMV8_THUNDER_PERFCTR_L1I_CACHE_PREF_MISS 0xED
/* PMUv3 HW events mapping. */
/*
* ARMv8 Architectural defined events, not all of these may
* be supported on any given implementation. Undefined events will
* be disabled at run-time.
*/
static const unsigned armv8_pmuv3_perf_map[PERF_COUNT_HW_MAX] = {
PERF_MAP_ALL_UNSUPPORTED,
[PERF_COUNT_HW_CPU_CYCLES] = ARMV8_PMUV3_PERFCTR_CPU_CYCLES,
[PERF_COUNT_HW_INSTRUCTIONS] = ARMV8_PMUV3_PERFCTR_INST_RETIRED,
[PERF_COUNT_HW_CACHE_REFERENCES] = ARMV8_PMUV3_PERFCTR_L1D_CACHE,
[PERF_COUNT_HW_CACHE_MISSES] = ARMV8_PMUV3_PERFCTR_L1D_CACHE_REFILL,
[PERF_COUNT_HW_BRANCH_INSTRUCTIONS] = ARMV8_PMUV3_PERFCTR_PC_WRITE_RETIRED,
[PERF_COUNT_HW_BRANCH_MISSES] = ARMV8_PMUV3_PERFCTR_BR_MIS_PRED,
[PERF_COUNT_HW_BUS_CYCLES] = ARMV8_PMUV3_PERFCTR_BUS_CYCLES,
[PERF_COUNT_HW_STALLED_CYCLES_FRONTEND] = ARMV8_PMUV3_PERFCTR_STALL_FRONTEND,
[PERF_COUNT_HW_STALLED_CYCLES_BACKEND] = ARMV8_PMUV3_PERFCTR_STALL_BACKEND,
};
/* ARM Cortex-A53 HW events mapping. */
......@@ -258,6 +269,15 @@ static const unsigned armv8_pmuv3_perf_cache_map[PERF_COUNT_HW_CACHE_MAX]
[C(L1D)][C(OP_WRITE)][C(RESULT_ACCESS)] = ARMV8_PMUV3_PERFCTR_L1D_CACHE,
[C(L1D)][C(OP_WRITE)][C(RESULT_MISS)] = ARMV8_PMUV3_PERFCTR_L1D_CACHE_REFILL,
[C(L1I)][C(OP_READ)][C(RESULT_ACCESS)] = ARMV8_PMUV3_PERFCTR_L1I_CACHE,
[C(L1I)][C(OP_READ)][C(RESULT_MISS)] = ARMV8_PMUV3_PERFCTR_L1I_CACHE_REFILL,
[C(DTLB)][C(OP_READ)][C(RESULT_MISS)] = ARMV8_PMUV3_PERFCTR_L1D_TLB_REFILL,
[C(DTLB)][C(OP_READ)][C(RESULT_ACCESS)] = ARMV8_PMUV3_PERFCTR_L1D_TLB,
[C(ITLB)][C(OP_READ)][C(RESULT_MISS)] = ARMV8_PMUV3_PERFCTR_L1I_TLB_REFILL,
[C(ITLB)][C(OP_READ)][C(RESULT_ACCESS)] = ARMV8_PMUV3_PERFCTR_L1I_TLB,
[C(BPU)][C(OP_READ)][C(RESULT_ACCESS)] = ARMV8_PMUV3_PERFCTR_BR_PRED,
[C(BPU)][C(OP_READ)][C(RESULT_MISS)] = ARMV8_PMUV3_PERFCTR_BR_MIS_PRED,
[C(BPU)][C(OP_WRITE)][C(RESULT_ACCESS)] = ARMV8_PMUV3_PERFCTR_BR_PRED,
......@@ -523,12 +543,6 @@ static struct attribute_group armv8_pmuv3_format_attr_group = {
.attrs = armv8_pmuv3_format_attrs,
};
static const struct attribute_group *armv8_pmuv3_attr_groups[] = {
&armv8_pmuv3_events_attr_group,
&armv8_pmuv3_format_attr_group,
NULL,
};
/*
* Perf Events' indices
*/
......@@ -905,9 +919,22 @@ static void armv8pmu_reset(void *info)
static int armv8_pmuv3_map_event(struct perf_event *event)
{
return armpmu_map_event(event, &armv8_pmuv3_perf_map,
&armv8_pmuv3_perf_cache_map,
ARMV8_PMU_EVTYPE_EVENT);
int hw_event_id;
struct arm_pmu *armpmu = to_arm_pmu(event->pmu);
hw_event_id = armpmu_map_event(event, &armv8_pmuv3_perf_map,
&armv8_pmuv3_perf_cache_map,
ARMV8_PMU_EVTYPE_EVENT);
if (hw_event_id < 0)
return hw_event_id;
/* disable micro/arch events not supported by this PMU */
if ((hw_event_id < ARMV8_PMUV3_MAX_COMMON_EVENTS) &&
!test_bit(hw_event_id, armpmu->pmceid_bitmap)) {
return -EOPNOTSUPP;
}
return hw_event_id;
}
static int armv8_a53_map_event(struct perf_event *event)
......@@ -985,7 +1012,10 @@ static int armv8_pmuv3_init(struct arm_pmu *cpu_pmu)
armv8_pmu_init(cpu_pmu);
cpu_pmu->name = "armv8_pmuv3";
cpu_pmu->map_event = armv8_pmuv3_map_event;
cpu_pmu->pmu.attr_groups = armv8_pmuv3_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv8_pmuv3_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv8_pmuv3_format_attr_group;
return armv8pmu_probe_pmu(cpu_pmu);
}
......@@ -994,7 +1024,10 @@ static int armv8_a53_pmu_init(struct arm_pmu *cpu_pmu)
armv8_pmu_init(cpu_pmu);
cpu_pmu->name = "armv8_cortex_a53";
cpu_pmu->map_event = armv8_a53_map_event;
cpu_pmu->pmu.attr_groups = armv8_pmuv3_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv8_pmuv3_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv8_pmuv3_format_attr_group;
return armv8pmu_probe_pmu(cpu_pmu);
}
......@@ -1003,7 +1036,10 @@ static int armv8_a57_pmu_init(struct arm_pmu *cpu_pmu)
armv8_pmu_init(cpu_pmu);
cpu_pmu->name = "armv8_cortex_a57";
cpu_pmu->map_event = armv8_a57_map_event;
cpu_pmu->pmu.attr_groups = armv8_pmuv3_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv8_pmuv3_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv8_pmuv3_format_attr_group;
return armv8pmu_probe_pmu(cpu_pmu);
}
......@@ -1012,7 +1048,10 @@ static int armv8_a72_pmu_init(struct arm_pmu *cpu_pmu)
armv8_pmu_init(cpu_pmu);
cpu_pmu->name = "armv8_cortex_a72";
cpu_pmu->map_event = armv8_a57_map_event;
cpu_pmu->pmu.attr_groups = armv8_pmuv3_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv8_pmuv3_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv8_pmuv3_format_attr_group;
return armv8pmu_probe_pmu(cpu_pmu);
}
......@@ -1021,7 +1060,10 @@ static int armv8_thunder_pmu_init(struct arm_pmu *cpu_pmu)
armv8_pmu_init(cpu_pmu);
cpu_pmu->name = "armv8_cavium_thunder";
cpu_pmu->map_event = armv8_thunder_map_event;
cpu_pmu->pmu.attr_groups = armv8_pmuv3_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv8_pmuv3_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv8_pmuv3_format_attr_group;
return armv8pmu_probe_pmu(cpu_pmu);
}
......@@ -1030,7 +1072,10 @@ static int armv8_vulcan_pmu_init(struct arm_pmu *cpu_pmu)
armv8_pmu_init(cpu_pmu);
cpu_pmu->name = "armv8_brcm_vulcan";
cpu_pmu->map_event = armv8_vulcan_map_event;
cpu_pmu->pmu.attr_groups = armv8_pmuv3_attr_groups;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
&armv8_pmuv3_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
&armv8_pmuv3_format_attr_group;
return armv8pmu_probe_pmu(cpu_pmu);
}
......@@ -1044,21 +1089,32 @@ static const struct of_device_id armv8_pmu_of_device_ids[] = {
{},
};
/*
* Non DT systems have their micro/arch events probed at run-time.
* A fairly complete list of generic events are provided and ones that
* aren't supported by the current PMU are disabled.
*/
static const struct pmu_probe_info armv8_pmu_probe_table[] = {
PMU_PROBE(0, 0, armv8_pmuv3_init), /* enable all defined counters */
{ /* sentinel value */ }
};
static int armv8_pmu_device_probe(struct platform_device *pdev)
{
return arm_pmu_device_probe(pdev, armv8_pmu_of_device_ids, NULL);
if (acpi_disabled)
return arm_pmu_device_probe(pdev, armv8_pmu_of_device_ids,
NULL);
return arm_pmu_device_probe(pdev, armv8_pmu_of_device_ids,
armv8_pmu_probe_table);
}
static struct platform_driver armv8_pmu_driver = {
.driver = {
.name = "armv8-pmu",
.name = ARMV8_PMU_PDEV_NAME,
.of_match_table = armv8_pmu_of_device_ids,
},
.probe = armv8_pmu_device_probe,
};
static int __init register_armv8_pmu_driver(void)
{
return platform_driver_register(&armv8_pmu_driver);
}
device_initcall(register_armv8_pmu_driver);
builtin_platform_driver(armv8_pmu_driver);
arch/arm64/kernel/probes/decode-insn.c
View file @ 7af8a0f8
......@@ -16,6 +16,7 @@
#include <linux/kernel.h>
#include <linux/kprobes.h>
#include <linux/module.h>
#include <linux/kallsyms.h>
#include <asm/kprobes.h>
#include <asm/insn.h>
#include <asm/sections.h>
......@@ -122,7 +123,7 @@ arm_probe_decode_insn(kprobe_opcode_t insn, struct arch_specific_insn *asi)
static bool __kprobes
is_probed_address_atomic(kprobe_opcode_t *scan_start, kprobe_opcode_t *scan_end)
{
while (scan_start > scan_end) {
while (scan_start >= scan_end) {
/*
* atomic region starts from exclusive load and ends with
* exclusive store.
......@@ -142,33 +143,30 @@ arm_kprobe_decode_insn(kprobe_opcode_t *addr, struct arch_specific_insn *asi)
{
enum kprobe_insn decoded;
kprobe_opcode_t insn = le32_to_cpu(*addr);
kprobe_opcode_t *scan_start = addr - 1;
kprobe_opcode_t *scan_end = addr - MAX_ATOMIC_CONTEXT_SIZE;
#if defined(CONFIG_MODULES) && defined(MODULES_VADDR)
struct module *mod;
#endif
if (addr >= (kprobe_opcode_t *)_text &&
scan_end < (kprobe_opcode_t *)_text)
scan_end = (kprobe_opcode_t *)_text;
#if defined(CONFIG_MODULES) && defined(MODULES_VADDR)
else {
preempt_disable();
mod = __module_address((unsigned long)addr);
if (mod && within_module_init((unsigned long)addr, mod) &&
!within_module_init((unsigned long)scan_end, mod))
scan_end = (kprobe_opcode_t *)mod->init_layout.base;
else if (mod && within_module_core((unsigned long)addr, mod) &&
!within_module_core((unsigned long)scan_end, mod))
scan_end = (kprobe_opcode_t *)mod->core_layout.base;
preempt_enable();
kprobe_opcode_t *scan_end = NULL;
unsigned long size = 0, offset = 0;
/*
* If there's a symbol defined in front of and near enough to
* the probe address assume it is the entry point to this
* code and use it to further limit how far back we search
* when determining if we're in an atomic sequence. If we could
* not find any symbol skip the atomic test altogether as we
* could otherwise end up searching irrelevant text/literals.
* KPROBES depends on KALLSYMS so this last case should never
* happen.
*/
if (kallsyms_lookup_size_offset((unsigned long) addr, &size, &offset)) {
if (offset < (MAX_ATOMIC_CONTEXT_SIZE*sizeof(kprobe_opcode_t)))
scan_end = addr - (offset / sizeof(kprobe_opcode_t));
else
scan_end = addr - MAX_ATOMIC_CONTEXT_SIZE;
}
#endif
decoded = arm_probe_decode_insn(insn, asi);
if (decoded == INSN_REJECTED ||
is_probed_address_atomic(scan_start, scan_end))
return INSN_REJECTED;
if (decoded != INSN_REJECTED && scan_end)
if (is_probed_address_atomic(addr - 1, scan_end))
return INSN_REJECTED;
return decoded;
}
arch/arm64/kernel/probes/kprobes.c
View file @ 7af8a0f8
......@@ -19,7 +19,7 @@
#include <linux/kasan.h>
#include <linux/kernel.h>
#include <linux/kprobes.h>
#include <linux/module.h>
#include <linux/extable.h>
#include <linux/slab.h>
#include <linux/stop_machine.h>
#include <linux/stringify.h>
......@@ -31,7 +31,7 @@
#include <asm/insn.h>
#include <asm/uaccess.h>
#include <asm/irq.h>
#include <asm-generic/sections.h>
#include <asm/sections.h>
#include "decode-insn.h"
......@@ -166,13 +166,18 @@ static void __kprobes set_current_kprobe(struct kprobe *p)
}
/*
* The D-flag (Debug mask) is set (masked) upon debug exception entry.
* Kprobes needs to clear (unmask) D-flag -ONLY- in case of recursive
* probe i.e. when probe hit from kprobe handler context upon
* executing the pre/post handlers. In this case we return with
* D-flag clear so that single-stepping can be carried-out.
*
* Leave D-flag set in all other cases.
* When PSTATE.D is set (masked), then software step exceptions can not be
* generated.
* SPSR's D bit shows the value of PSTATE.D immediately before the
* exception was taken. PSTATE.D is set while entering into any exception
* mode, however software clears it for any normal (none-debug-exception)
* mode in the exception entry. Therefore, when we are entering into kprobe
* breakpoint handler from any normal mode then SPSR.D bit is already
* cleared, however it is set when we are entering from any debug exception
* mode.
* Since we always need to generate single step exception after a kprobe
* breakpoint exception therefore we need to clear it unconditionally, when
* we become sure that the current breakpoint exception is for kprobe.
*/
static void __kprobes
spsr_set_debug_flag(struct pt_regs *regs, int mask)
......@@ -245,10 +250,7 @@ static void __kprobes setup_singlestep(struct kprobe *p,
set_ss_context(kcb, slot); /* mark pending ss */
if (kcb->kprobe_status == KPROBE_REENTER)
spsr_set_debug_flag(regs, 0);
else
WARN_ON(regs->pstate & PSR_D_BIT);
spsr_set_debug_flag(regs, 0);
/* IRQs and single stepping do not mix well. */
kprobes_save_local_irqflag(kcb, regs);
......@@ -333,8 +335,6 @@ int __kprobes kprobe_fault_handler(struct pt_regs *regs, unsigned int fsr)
BUG();
kernel_disable_single_step();
if (kcb->kprobe_status == KPROBE_REENTER)
spsr_set_debug_flag(regs, 1);
if (kcb->kprobe_status == KPROBE_REENTER)
restore_previous_kprobe(kcb);
......@@ -457,9 +457,6 @@ kprobe_single_step_handler(struct pt_regs *regs, unsigned int esr)
kprobes_restore_local_irqflag(kcb, regs);
kernel_disable_single_step();
if (kcb->kprobe_status == KPROBE_REENTER)
spsr_set_debug_flag(regs, 1);
post_kprobe_handler(kcb, regs);
}
......@@ -543,9 +540,6 @@ int __kprobes longjmp_break_handler(struct kprobe *p, struct pt_regs *regs)
bool arch_within_kprobe_blacklist(unsigned long addr)
{
extern char __idmap_text_start[], __idmap_text_end[];
extern char __hyp_idmap_text_start[], __hyp_idmap_text_end[];
if ((addr >= (unsigned long)__kprobes_text_start &&
addr < (unsigned long)__kprobes_text_end) ||
(addr >= (unsigned long)__entry_text_start &&
......
arch/arm64/kernel/process.c
View file @ 7af8a0f8
......@@ -202,7 +202,7 @@ void show_regs(struct pt_regs * regs)
static void tls_thread_flush(void)
{
asm ("msr tpidr_el0, xzr");
write_sysreg(0, tpidr_el0);
if (is_compat_task()) {
current->thread.tp_value = 0;
......@@ -213,7 +213,7 @@ static void tls_thread_flush(void)
* with a stale shadow state during context switch.
*/
barrier();
asm ("msr tpidrro_el0, xzr");
write_sysreg(0, tpidrro_el0);
}
}
......@@ -253,7 +253,7 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start,
* Read the current TLS pointer from tpidr_el0 as it may be
* out-of-sync with the saved value.
*/
asm("mrs %0, tpidr_el0" : "=r" (*task_user_tls(p)));
*task_user_tls(p) = read_sysreg(tpidr_el0);
if (stack_start) {
if (is_compat_thread(task_thread_info(p)))
......@@ -289,17 +289,15 @@ static void tls_thread_switch(struct task_struct *next)
{
unsigned long tpidr, tpidrro;
asm("mrs %0, tpidr_el0" : "=r" (tpidr));
tpidr = read_sysreg(tpidr_el0);
*task_user_tls(current) = tpidr;
tpidr = *task_user_tls(next);
tpidrro = is_compat_thread(task_thread_info(next)) ?
next->thread.tp_value : 0;
asm(
" msr tpidr_el0, %0\n"
" msr tpidrro_el0, %1"
: : "r" (tpidr), "r" (tpidrro));
write_sysreg(tpidr, tpidr_el0);
write_sysreg(tpidrro, tpidrro_el0);
}
/* Restore the UAO state depending on next's addr_limit */
......
arch/arm64/kernel/relocate_kernel.S
View file @ 7af8a0f8
......@@ -34,7 +34,7 @@ ENTRY(arm64_relocate_new_kernel)
/* Setup the list loop variables. */
mov x17, x1 /* x17 = kimage_start */
mov x16, x0 /* x16 = kimage_head */
dcache_line_size x15, x0 /* x15 = dcache line size */
raw_dcache_line_size x15, x0 /* x15 = dcache line size */
mov x14, xzr /* x14 = entry ptr */
mov x13, xzr /* x13 = copy dest */
......
arch/arm64/kernel/setup.c
View file @ 7af8a0f8
......@@ -206,10 +206,15 @@ static void __init request_standard_resources(void)
for_each_memblock(memory, region) {
res = alloc_bootmem_low(sizeof(*res));
res->name = "System RAM";
if (memblock_is_nomap(region)) {
res->name = "reserved";
res->flags = IORESOURCE_MEM | IORESOURCE_BUSY;
} else {
res->name = "System RAM";
res->flags = IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY;
}
res->start = __pfn_to_phys(memblock_region_memory_base_pfn(region));
res->end = __pfn_to_phys(memblock_region_memory_end_pfn(region)) - 1;
res->flags = IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY;
request_resource(&iomem_resource, res);
......@@ -228,7 +233,7 @@ void __init setup_arch(char **cmdline_p)
{
pr_info("Boot CPU: AArch64 Processor [%08x]\n", read_cpuid_id());
sprintf(init_utsname()->machine, ELF_PLATFORM);
sprintf(init_utsname()->machine, UTS_MACHINE);
init_mm.start_code = (unsigned long) _text;
init_mm.end_code = (unsigned long) _etext;
init_mm.end_data = (unsigned long) _edata;
......
arch/arm64/kernel/signal.c
View file @ 7af8a0f8
......@@ -402,15 +402,31 @@ static void do_signal(struct pt_regs *regs)
asmlinkage void do_notify_resume(struct pt_regs *regs,
unsigned int thread_flags)
{
if (thread_flags & _TIF_SIGPENDING)
do_signal(regs);
if (thread_flags & _TIF_NOTIFY_RESUME) {
clear_thread_flag(TIF_NOTIFY_RESUME);
tracehook_notify_resume(regs);
}
if (thread_flags & _TIF_FOREIGN_FPSTATE)
fpsimd_restore_current_state();
/*
* The assembly code enters us with IRQs off, but it hasn't
* informed the tracing code of that for efficiency reasons.
* Update the trace code with the current status.
*/
trace_hardirqs_off();
do {
if (thread_flags & _TIF_NEED_RESCHED) {
schedule();
} else {
local_irq_enable();
if (thread_flags & _TIF_SIGPENDING)
do_signal(regs);
if (thread_flags & _TIF_NOTIFY_RESUME) {
clear_thread_flag(TIF_NOTIFY_RESUME);
tracehook_notify_resume(regs);
}
if (thread_flags & _TIF_FOREIGN_FPSTATE)
fpsimd_restore_current_state();
}
local_irq_disable();
thread_flags = READ_ONCE(current_thread_info()->flags);
} while (thread_flags & _TIF_WORK_MASK);
}
arch/arm64/kernel/sleep.S
View file @ 7af8a0f8
......@@ -73,10 +73,9 @@ ENTRY(__cpu_suspend_enter)
str x2, [x0, #SLEEP_STACK_DATA_SYSTEM_REGS + CPU_CTX_SP]
/* find the mpidr_hash */
ldr x1, =sleep_save_stash
ldr x1, [x1]
ldr_l x1, sleep_save_stash
mrs x7, mpidr_el1
ldr x9, =mpidr_hash
adr_l x9, mpidr_hash
ldr x10, [x9, #MPIDR_HASH_MASK]
/*
* Following code relies on the struct mpidr_hash
......@@ -95,36 +94,30 @@ ENTRY(__cpu_suspend_enter)
mov x0, #1
ret
ENDPROC(__cpu_suspend_enter)
.ltorg
.pushsection ".idmap.text", "ax"
ENTRY(cpu_resume)
bl el2_setup // if in EL2 drop to EL1 cleanly
bl __cpu_setup
/* enable the MMU early - so we can access sleep_save_stash by va */
adr_l lr, __enable_mmu /* __cpu_setup will return here */
adr_l x27, _resume_switched /* __enable_mmu will branch here */
adrp x25, idmap_pg_dir
adrp x26, swapper_pg_dir
b __cpu_setup
ENDPROC(cpu_resume)
.pushsection ".idmap.text", "ax"
_resume_switched:
bl __enable_mmu
ldr x8, =_cpu_resume
br x8
ENDPROC(_resume_switched)
ENDPROC(cpu_resume)
.ltorg
.popsection
ENTRY(_cpu_resume)
mrs x1, mpidr_el1
adrp x8, mpidr_hash
add x8, x8, #:lo12:mpidr_hash // x8 = struct mpidr_hash phys address
/* retrieve mpidr_hash members to compute the hash */
adr_l x8, mpidr_hash // x8 = struct mpidr_hash virt address
/* retrieve mpidr_hash members to compute the hash */
ldr x2, [x8, #MPIDR_HASH_MASK]
ldp w3, w4, [x8, #MPIDR_HASH_SHIFTS]
ldp w5, w6, [x8, #(MPIDR_HASH_SHIFTS + 8)]
compute_mpidr_hash x7, x3, x4, x5, x6, x1, x2
/* x7 contains hash index, let's use it to grab context pointer */
/* x7 contains hash index, let's use it to grab context pointer */
ldr_l x0, sleep_save_stash
ldr x0, [x0, x7, lsl #3]
add x29, x0, #SLEEP_STACK_DATA_CALLEE_REGS
......
arch/arm64/kernel/smp.c
View file @ 7af8a0f8
......@@ -233,7 +233,7 @@ asmlinkage void secondary_start_kernel(void)
* this CPU ticks all of those. If it doesn't, the CPU will
* fail to come online.
*/
verify_local_cpu_capabilities();
check_local_cpu_capabilities();
if (cpu_ops[cpu]->cpu_postboot)
cpu_ops[cpu]->cpu_postboot();
......@@ -431,8 +431,19 @@ void __init smp_cpus_done(unsigned int max_cpus)
void __init smp_prepare_boot_cpu(void)
{
set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
/*
* Initialise the static keys early as they may be enabled by the
* cpufeature code.
*/
jump_label_init();
cpuinfo_store_boot_cpu();
save_boot_cpu_run_el();
/*
* Run the errata work around checks on the boot CPU, once we have
* initialised the cpu feature infrastructure from
* cpuinfo_store_boot_cpu() above.
*/
update_cpu_errata_workarounds();
}
static u64 __init of_get_cpu_mpidr(struct device_node *dn)
......@@ -613,6 +624,7 @@ static void __init of_parse_and_init_cpus(void)
}
bootcpu_valid = true;
early_map_cpu_to_node(0, of_node_to_nid(dn));
/*
* cpu_logical_map has already been
......
arch/arm64/kernel/smp_spin_table.c
View file @ 7af8a0f8
......@@ -29,7 +29,8 @@
#include <asm/smp_plat.h>
extern void secondary_holding_pen(void);
volatile unsigned long secondary_holding_pen_release = INVALID_HWID;
volatile unsigned long __section(".mmuoff.data.read")
secondary_holding_pen_release = INVALID_HWID;
static phys_addr_t cpu_release_addr[NR_CPUS];
......
arch/arm64/kernel/stacktrace.c
View file @ 7af8a0f8
......@@ -43,6 +43,9 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame)
unsigned long fp = frame->fp;
unsigned long irq_stack_ptr;
if (!tsk)
tsk = current;
/*
* Switching between stacks is valid when tracing current and in
* non-preemptible context.
......@@ -67,7 +70,7 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame)
frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp + 8));
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
if (tsk && tsk->ret_stack &&
if (tsk->ret_stack &&
(frame->pc == (unsigned long)return_to_handler)) {
/*
* This is a case where function graph tracer has
......@@ -152,6 +155,27 @@ static int save_trace(struct stackframe *frame, void *d)
return trace->nr_entries >= trace->max_entries;
}
void save_stack_trace_regs(struct pt_regs *regs, struct stack_trace *trace)
{
struct stack_trace_data data;
struct stackframe frame;
data.trace = trace;
data.skip = trace->skip;
data.no_sched_functions = 0;
frame.fp = regs->regs[29];
frame.sp = regs->sp;
frame.pc = regs->pc;
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
frame.graph = current->curr_ret_stack;
#endif
walk_stackframe(current, &frame, save_trace, &data);
if (trace->nr_entries < trace->max_entries)
trace->entries[trace->nr_entries++] = ULONG_MAX;
}
void save_stack_trace_tsk(struct task_struct *tsk, struct stack_trace *trace)
{
struct stack_trace_data data;
......
arch/arm64/kernel/suspend.c
View file @ 7af8a0f8
......@@ -23,8 +23,8 @@ unsigned long *sleep_save_stash;
* time the notifier runs debug exceptions might have been enabled already,
* with HW breakpoints registers content still in an unknown state.
*/
static void (*hw_breakpoint_restore)(void *);
void __init cpu_suspend_set_dbg_restorer(void (*hw_bp_restore)(void *))
static int (*hw_breakpoint_restore)(unsigned int);
void __init cpu_suspend_set_dbg_restorer(int (*hw_bp_restore)(unsigned int))
{
/* Prevent multiple restore hook initializations */
if (WARN_ON(hw_breakpoint_restore))
......@@ -34,6 +34,8 @@ void __init cpu_suspend_set_dbg_restorer(void (*hw_bp_restore)(void *))
void notrace __cpu_suspend_exit(void)
{
unsigned int cpu = smp_processor_id();
/*
* We are resuming from reset with the idmap active in TTBR0_EL1.
* We must uninstall the idmap and restore the expected MMU
......@@ -45,7 +47,7 @@ void notrace __cpu_suspend_exit(void)
* Restore per-cpu offset before any kernel
* subsystem relying on it has a chance to run.
*/
set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
set_my_cpu_offset(per_cpu_offset(cpu));
/*
* Restore HW breakpoint registers to sane values
......@@ -53,7 +55,7 @@ void notrace __cpu_suspend_exit(void)
* through local_dbg_restore.
*/
if (hw_breakpoint_restore)
hw_breakpoint_restore(NULL);
hw_breakpoint_restore(cpu);
}
/*
......
arch/arm64/kernel/sys_compat.c
View file @ 7af8a0f8
......@@ -94,7 +94,7 @@ long compat_arm_syscall(struct pt_regs *regs)
* See comment in tls_thread_flush.
*/
barrier();
asm ("msr tpidrro_el0, %0" : : "r" (regs->regs[0]));
write_sysreg(regs->regs[0], tpidrro_el0);
return 0;
default:
......
arch/arm64/kernel/traps.c
View file @ 7af8a0f8
......@@ -142,6 +142,11 @@ static void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
unsigned long irq_stack_ptr;
int skip;
pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk);
if (!tsk)
tsk = current;
/*
* Switching between stacks is valid when tracing current and in
* non-preemptible context.
......@@ -151,11 +156,6 @@ static void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
else
irq_stack_ptr = 0;
pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk);
if (!tsk)
tsk = current;
if (tsk == current) {
frame.fp = (unsigned long)__builtin_frame_address(0);
frame.sp = current_stack_pointer;
......@@ -447,36 +447,29 @@ void cpu_enable_cache_maint_trap(void *__unused)
: "=r" (res) \
: "r" (address), "i" (-EFAULT) )
asmlinkage void __exception do_sysinstr(unsigned int esr, struct pt_regs *regs)
static void user_cache_maint_handler(unsigned int esr, struct pt_regs *regs)
{
unsigned long address;
int ret;
/* if this is a write with: Op0=1, Op2=1, Op1=3, CRn=7 */
if ((esr & 0x01fffc01) == 0x0012dc00) {
int rt = (esr >> 5) & 0x1f;
int crm = (esr >> 1) & 0x0f;
int rt = (esr & ESR_ELx_SYS64_ISS_RT_MASK) >> ESR_ELx_SYS64_ISS_RT_SHIFT;
int crm = (esr & ESR_ELx_SYS64_ISS_CRM_MASK) >> ESR_ELx_SYS64_ISS_CRM_SHIFT;
int ret = 0;
address = (rt == 31) ? 0 : regs->regs[rt];
address = (rt == 31) ? 0 : regs->regs[rt];
switch (crm) {
case 11: /* DC CVAU, gets promoted */
__user_cache_maint("dc civac", address, ret);
break;
case 10: /* DC CVAC, gets promoted */
__user_cache_maint("dc civac", address, ret);
break;
case 14: /* DC CIVAC */
__user_cache_maint("dc civac", address, ret);
break;
case 5: /* IC IVAU */
__user_cache_maint("ic ivau", address, ret);
break;
default:
force_signal_inject(SIGILL, ILL_ILLOPC, regs, 0);
return;
}
} else {
switch (crm) {
case ESR_ELx_SYS64_ISS_CRM_DC_CVAU: /* DC CVAU, gets promoted */
__user_cache_maint("dc civac", address, ret);
break;
case ESR_ELx_SYS64_ISS_CRM_DC_CVAC: /* DC CVAC, gets promoted */
__user_cache_maint("dc civac", address, ret);
break;
case ESR_ELx_SYS64_ISS_CRM_DC_CIVAC: /* DC CIVAC */
__user_cache_maint("dc civac", address, ret);
break;
case ESR_ELx_SYS64_ISS_CRM_IC_IVAU: /* IC IVAU */
__user_cache_maint("ic ivau", address, ret);
break;
default:
force_signal_inject(SIGILL, ILL_ILLOPC, regs, 0);
return;
}
......@@ -487,6 +480,48 @@ asmlinkage void __exception do_sysinstr(unsigned int esr, struct pt_regs *regs)
regs->pc += 4;
}
static void ctr_read_handler(unsigned int esr, struct pt_regs *regs)
{
int rt = (esr & ESR_ELx_SYS64_ISS_RT_MASK) >> ESR_ELx_SYS64_ISS_RT_SHIFT;
regs->regs[rt] = arm64_ftr_reg_ctrel0.sys_val;
regs->pc += 4;
}
struct sys64_hook {
unsigned int esr_mask;
unsigned int esr_val;
void (*handler)(unsigned int esr, struct pt_regs *regs);
};
static struct sys64_hook sys64_hooks[] = {
{
.esr_mask = ESR_ELx_SYS64_ISS_EL0_CACHE_OP_MASK,
.esr_val = ESR_ELx_SYS64_ISS_EL0_CACHE_OP_VAL,
.handler = user_cache_maint_handler,
},
{
/* Trap read access to CTR_EL0 */
.esr_mask = ESR_ELx_SYS64_ISS_SYS_OP_MASK,
.esr_val = ESR_ELx_SYS64_ISS_SYS_CTR_READ,
.handler = ctr_read_handler,
},
{},
};
asmlinkage void __exception do_sysinstr(unsigned int esr, struct pt_regs *regs)
{
struct sys64_hook *hook;
for (hook = sys64_hooks; hook->handler; hook++)
if ((hook->esr_mask & esr) == hook->esr_val) {
hook->handler(esr, regs);
return;
}
force_signal_inject(SIGILL, ILL_ILLOPC, regs, 0);
}
long compat_arm_syscall(struct pt_regs *regs);
asmlinkage long do_ni_syscall(struct pt_regs *regs)
......
arch/arm64/kernel/vdso.c
View file @ 7af8a0f8
......@@ -18,12 +18,13 @@
* Author: Will Deacon <will.deacon@arm.com>
*/
#include <linux/kernel.h>
#include <linux/cache.h>
#include <linux/clocksource.h>
#include <linux/elf.h>
#include <linux/err.h>
#include <linux/errno.h>
#include <linux/gfp.h>
#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/sched.h>
#include <linux/signal.h>
......@@ -37,8 +38,7 @@
#include <asm/vdso_datapage.h>
extern char vdso_start, vdso_end;
static unsigned long vdso_pages;
static struct page **vdso_pagelist;
static unsigned long vdso_pages __ro_after_init;
/*
* The vDSO data page.
......@@ -53,9 +53,9 @@ struct vdso_data *vdso_data = &vdso_data_store.data;
/*
* Create and map the vectors page for AArch32 tasks.
*/
static struct page *vectors_page[1];
static struct page *vectors_page[1] __ro_after_init;
static int alloc_vectors_page(void)
static int __init alloc_vectors_page(void)
{
extern char __kuser_helper_start[], __kuser_helper_end[];
extern char __aarch32_sigret_code_start[], __aarch32_sigret_code_end[];
......@@ -88,7 +88,7 @@ int aarch32_setup_vectors_page(struct linux_binprm *bprm, int uses_interp)
{
struct mm_struct *mm = current->mm;
unsigned long addr = AARCH32_VECTORS_BASE;
static struct vm_special_mapping spec = {
static const struct vm_special_mapping spec = {
.name = "[vectors]",
.pages = vectors_page,
......@@ -110,11 +110,19 @@ int aarch32_setup_vectors_page(struct linux_binprm *bprm, int uses_interp)
}
#endif /* CONFIG_COMPAT */
static struct vm_special_mapping vdso_spec[2];
static struct vm_special_mapping vdso_spec[2] __ro_after_init = {
{
.name = "[vvar]",
},
{
.name = "[vdso]",
},
};
static int __init vdso_init(void)
{
int i;
struct page **vdso_pagelist;
if (memcmp(&vdso_start, "\177ELF", 4)) {
pr_err("vDSO is not a valid ELF object!\n");
......@@ -138,16 +146,8 @@ static int __init vdso_init(void)
for (i = 0; i < vdso_pages; i++)
vdso_pagelist[i + 1] = pfn_to_page(PHYS_PFN(__pa(&vdso_start)) + i);
/* Populate the special mapping structures */
vdso_spec[0] = (struct vm_special_mapping) {
.name = "[vvar]",
.pages = vdso_pagelist,
};
vdso_spec[1] = (struct vm_special_mapping) {
.name = "[vdso]",
.pages = &vdso_pagelist[1],
};
vdso_spec[0].pages = &vdso_pagelist[0];
vdso_spec[1].pages = &vdso_pagelist[1];
return 0;
}
......@@ -201,7 +201,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm,
*/
void update_vsyscall(struct timekeeper *tk)
{
u32 use_syscall = strcmp(tk->tkr_mono.clock->name, "arch_sys_counter");
u32 use_syscall = !tk->tkr_mono.clock->archdata.vdso_direct;
++vdso_data->tb_seq_count;
smp_wmb();
......
arch/arm64/kernel/vmlinux.lds.S
View file @ 7af8a0f8
......@@ -185,6 +185,25 @@ SECTIONS
_data = .;
_sdata = .;
RW_DATA_SECTION(L1_CACHE_BYTES, PAGE_SIZE, THREAD_SIZE)
/*
* Data written with the MMU off but read with the MMU on requires
* cache lines to be invalidated, discarding up to a Cache Writeback
* Granule (CWG) of data from the cache. Keep the section that
* requires this type of maintenance to be in its own Cache Writeback
* Granule (CWG) area so the cache maintenance operations don't
* interfere with adjacent data.
*/
.mmuoff.data.write : ALIGN(SZ_2K) {
__mmuoff_data_start = .;
*(.mmuoff.data.write)
}
. = ALIGN(SZ_2K);
.mmuoff.data.read : {
*(.mmuoff.data.read)
__mmuoff_data_end = .;
}
PECOFF_EDATA_PADDING
_edata = .;
......
arch/arm64/kvm/hyp.S
View file @ 7af8a0f8
......@@ -46,10 +46,6 @@ alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
hvc #0
ldr lr, [sp], #16
ret
alternative_else
alternative_else_nop_endif
b __vhe_hyp_call
nop
nop
nop
alternative_endif
ENDPROC(__kvm_call_hyp)
arch/arm64/kvm/sys_regs.c
View file @ 7af8a0f8
......@@ -36,6 +36,7 @@
#include <asm/kvm_host.h>
#include <asm/kvm_mmu.h>
#include <asm/perf_event.h>
#include <asm/sysreg.h>
#include <trace/events/kvm.h>
......@@ -67,11 +68,9 @@ static u32 get_ccsidr(u32 csselr)
/* Make sure noone else changes CSSELR during this! */
local_irq_disable();
/* Put value into CSSELR */
asm volatile("msr csselr_el1, %x0" : : "r" (csselr));
write_sysreg(csselr, csselr_el1);
isb();
/* Read result out of CCSIDR */
asm volatile("mrs %0, ccsidr_el1" : "=r" (ccsidr));
ccsidr = read_sysreg(ccsidr_el1);
local_irq_enable();
return ccsidr;
......@@ -174,9 +173,7 @@ static bool trap_dbgauthstatus_el1(struct kvm_vcpu *vcpu,
if (p->is_write) {
return ignore_write(vcpu, p);
} else {
u32 val;
asm volatile("mrs %0, dbgauthstatus_el1" : "=r" (val));
p->regval = val;
p->regval = read_sysreg(dbgauthstatus_el1);
return true;
}
}
......@@ -429,10 +426,7 @@ static void reset_wcr(struct kvm_vcpu *vcpu,
static void reset_amair_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
{
u64 amair;
asm volatile("mrs %0, amair_el1\n" : "=r" (amair));
vcpu_sys_reg(vcpu, AMAIR_EL1) = amair;
vcpu_sys_reg(vcpu, AMAIR_EL1) = read_sysreg(amair_el1);
}
static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
......@@ -456,8 +450,9 @@ static void reset_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
{
u64 pmcr, val;
asm volatile("mrs %0, pmcr_el0\n" : "=r" (pmcr));
/* Writable bits of PMCR_EL0 (ARMV8_PMU_PMCR_MASK) is reset to UNKNOWN
pmcr = read_sysreg(pmcr_el0);
/*
* Writable bits of PMCR_EL0 (ARMV8_PMU_PMCR_MASK) are reset to UNKNOWN
* except PMCR.E resetting to zero.
*/
val = ((pmcr & ~ARMV8_PMU_PMCR_MASK)
......@@ -557,9 +552,9 @@ static bool access_pmceid(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
return false;
if (!(p->Op2 & 1))
asm volatile("mrs %0, pmceid0_el0\n" : "=r" (pmceid));
pmceid = read_sysreg(pmceid0_el0);
else
asm volatile("mrs %0, pmceid1_el0\n" : "=r" (pmceid));
pmceid = read_sysreg(pmceid1_el0);
p->regval = pmceid;
......@@ -1833,11 +1828,7 @@ static const struct sys_reg_desc *index_to_sys_reg_desc(struct kvm_vcpu *vcpu,
static void get_##reg(struct kvm_vcpu *v, \
const struct sys_reg_desc *r) \
{ \
u64 val; \
\
asm volatile("mrs %0, " __stringify(reg) "\n" \
: "=r" (val)); \
((struct sys_reg_desc *)r)->val = val; \
((struct sys_reg_desc *)r)->val = read_sysreg(reg); \
}
FUNCTION_INVARIANT(midr_el1)
......
arch/arm64/kvm/sys_regs_generic_v8.c
View file @ 7af8a0f8
......@@ -26,6 +26,7 @@
#include <asm/kvm_host.h>
#include <asm/kvm_emulate.h>
#include <asm/kvm_coproc.h>
#include <asm/sysreg.h>
#include <linux/init.h>
#include "sys_regs.h"
......@@ -43,10 +44,7 @@ static bool access_actlr(struct kvm_vcpu *vcpu,
static void reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
{
u64 actlr;
asm volatile("mrs %0, actlr_el1\n" : "=r" (actlr));
vcpu_sys_reg(vcpu, ACTLR_EL1) = actlr;
vcpu_sys_reg(vcpu, ACTLR_EL1) = read_sysreg(actlr_el1);
}
/*
......
arch/arm64/lib/copy_page.S
View file @ 7af8a0f8
......@@ -29,14 +29,11 @@
* x1 - src
*/
ENTRY(copy_page)
alternative_if_not ARM64_HAS_NO_HW_PREFETCH
nop
nop
alternative_else
alternative_if ARM64_HAS_NO_HW_PREFETCH
# Prefetch two cache lines ahead.
prfm pldl1strm, [x1, #128]
prfm pldl1strm, [x1, #256]
alternative_endif
alternative_else_nop_endif
ldp x2, x3, [x1]
ldp x4, x5, [x1, #16]
......@@ -52,11 +49,9 @@ alternative_endif
1:
subs x18, x18, #128
alternative_if_not ARM64_HAS_NO_HW_PREFETCH
nop
alternative_else
alternative_if ARM64_HAS_NO_HW_PREFETCH
prfm pldl1strm, [x1, #384]
alternative_endif
alternative_else_nop_endif
stnp x2, x3, [x0]
ldp x2, x3, [x1]
......
arch/arm64/mm/cache.S
View file @ 7af8a0f8
......@@ -105,19 +105,20 @@ ENTRY(__clean_dcache_area_pou)
ENDPROC(__clean_dcache_area_pou)
/*
* __inval_cache_range(start, end)
* - start - start address of region
* - end - end address of region
* __dma_inv_area(start, size)
* - start - virtual start address of region
* - size - size in question
*/
ENTRY(__inval_cache_range)
__dma_inv_area:
add x1, x1, x0
/* FALLTHROUGH */
/*
* __dma_inv_range(start, end)
* - start - virtual start address of region
* - end - virtual end address of region
* __inval_cache_range(start, end)
* - start - start address of region
* - end - end address of region
*/
__dma_inv_range:
ENTRY(__inval_cache_range)
dcache_line_size x2, x3
sub x3, x2, #1
tst x1, x3 // end cache line aligned?
......@@ -136,46 +137,43 @@ __dma_inv_range:
dsb sy
ret
ENDPIPROC(__inval_cache_range)
ENDPROC(__dma_inv_range)
ENDPROC(__dma_inv_area)
/*
* __clean_dcache_area_poc(kaddr, size)
*
* Ensure that any D-cache lines for the interval [kaddr, kaddr+size)
* are cleaned to the PoC.
*
* - kaddr - kernel address
* - size - size in question
*/
ENTRY(__clean_dcache_area_poc)
/* FALLTHROUGH */
/*
* __dma_clean_range(start, end)
* __dma_clean_area(start, size)
* - start - virtual start address of region
* - end - virtual end address of region
* - size - size in question
*/
__dma_clean_range:
dcache_line_size x2, x3
sub x3, x2, #1
bic x0, x0, x3
1:
alternative_if_not ARM64_WORKAROUND_CLEAN_CACHE
dc cvac, x0
alternative_else
dc civac, x0
alternative_endif
add x0, x0, x2
cmp x0, x1
b.lo 1b
dsb sy
__dma_clean_area:
dcache_by_line_op cvac, sy, x0, x1, x2, x3
ret
ENDPROC(__dma_clean_range)
ENDPIPROC(__clean_dcache_area_poc)
ENDPROC(__dma_clean_area)
/*
* __dma_flush_range(start, end)
* __dma_flush_area(start, size)
*
* clean & invalidate D / U line
*
* - start - virtual start address of region
* - end - virtual end address of region
* - size - size in question
*/
ENTRY(__dma_flush_range)
dcache_line_size x2, x3
sub x3, x2, #1
bic x0, x0, x3
1: dc civac, x0 // clean & invalidate D / U line
add x0, x0, x2
cmp x0, x1
b.lo 1b
dsb sy
ENTRY(__dma_flush_area)
dcache_by_line_op civac, sy, x0, x1, x2, x3
ret
ENDPIPROC(__dma_flush_range)
ENDPIPROC(__dma_flush_area)
/*
* __dma_map_area(start, size, dir)
......@@ -184,10 +182,9 @@ ENDPIPROC(__dma_flush_range)
* - dir - DMA direction
*/
ENTRY(__dma_map_area)
add x1, x1, x0
cmp w2, #DMA_FROM_DEVICE
b.eq __dma_inv_range
b __dma_clean_range
b.eq __dma_inv_area
b __dma_clean_area
ENDPIPROC(__dma_map_area)
/*
......@@ -197,8 +194,7 @@ ENDPIPROC(__dma_map_area)
* - dir - DMA direction
*/
ENTRY(__dma_unmap_area)
add x1, x1, x0
cmp w2, #DMA_TO_DEVICE
b.ne __dma_inv_range
b.ne __dma_inv_area
ret
ENDPIPROC(__dma_unmap_area)
arch/arm64/mm/dma-mapping.c
View file @ 7af8a0f8
......@@ -20,6 +20,7 @@
#include <linux/gfp.h>
#include <linux/acpi.h>
#include <linux/bootmem.h>
#include <linux/cache.h>
#include <linux/export.h>
#include <linux/slab.h>
#include <linux/genalloc.h>
......@@ -30,7 +31,7 @@
#include <asm/cacheflush.h>
static int swiotlb __read_mostly;
static int swiotlb __ro_after_init;
static pgprot_t __get_dma_pgprot(unsigned long attrs, pgprot_t prot,
bool coherent)
......@@ -168,7 +169,7 @@ static void *__dma_alloc(struct device *dev, size_t size,
return ptr;
/* remove any dirty cache lines on the kernel alias */
__dma_flush_range(ptr, ptr + size);
__dma_flush_area(ptr, size);
/* create a coherent mapping */
page = virt_to_page(ptr);
......@@ -387,7 +388,7 @@ static int __init atomic_pool_init(void)
void *page_addr = page_address(page);
memset(page_addr, 0, atomic_pool_size);
__dma_flush_range(page_addr, page_addr + atomic_pool_size);
__dma_flush_area(page_addr, atomic_pool_size);
atomic_pool = gen_pool_create(PAGE_SHIFT, -1);
if (!atomic_pool)
......@@ -548,7 +549,7 @@ fs_initcall(dma_debug_do_init);
/* Thankfully, all cache ops are by VA so we can ignore phys here */
static void flush_page(struct device *dev, const void *virt, phys_addr_t phys)
{
__dma_flush_range(virt, virt + PAGE_SIZE);
__dma_flush_area(virt, PAGE_SIZE);
}
static void *__iommu_alloc_attrs(struct device *dev, size_t size,
......
arch/arm64/mm/extable.c
View file @ 7af8a0f8
......@@ -2,7 +2,7 @@
* Based on arch/arm/mm/extable.c
*/
#include <linux/module.h>
#include <linux/extable.h>
#include <linux/uaccess.h>
int fixup_exception(struct pt_regs *regs)
......
arch/arm64/mm/fault.c
View file @ 7af8a0f8
......@@ -18,7 +18,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <linux/module.h>
#include <linux/extable.h>
#include <linux/signal.h>
#include <linux/mm.h>
#include <linux/hardirq.h>
......@@ -251,8 +251,7 @@ static int __do_page_fault(struct mm_struct *mm, unsigned long addr,
good_area:
/*
* Check that the permissions on the VMA allow for the fault which
* occurred. If we encountered a write or exec fault, we must have
* appropriate permissions, otherwise we allow any permission.
* occurred.
*/
if (!(vma->vm_flags & vm_flags)) {
fault = VM_FAULT_BADACCESS;
......@@ -288,7 +287,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr,
struct task_struct *tsk;
struct mm_struct *mm;
int fault, sig, code;
unsigned long vm_flags = VM_READ | VM_WRITE | VM_EXEC;
unsigned long vm_flags = VM_READ | VM_WRITE;
unsigned int mm_flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
if (notify_page_fault(regs, esr))
......
arch/arm64/mm/flush.c
View file @ 7af8a0f8
......@@ -25,8 +25,6 @@
#include <asm/cachetype.h>
#include <asm/tlbflush.h>
#include "mm.h"
void flush_cache_range(struct vm_area_struct *vma, unsigned long start,
unsigned long end)
{
......
arch/arm64/mm/init.c
View file @ 7af8a0f8
......@@ -23,6 +23,7 @@
#include <linux/swap.h>
#include <linux/init.h>
#include <linux/bootmem.h>
#include <linux/cache.h>
#include <linux/mman.h>
#include <linux/nodemask.h>
#include <linux/initrd.h>
......@@ -34,6 +35,7 @@
#include <linux/dma-contiguous.h>
#include <linux/efi.h>
#include <linux/swiotlb.h>
#include <linux/vmalloc.h>
#include <asm/boot.h>
#include <asm/fixmap.h>
......@@ -47,16 +49,14 @@
#include <asm/tlb.h>
#include <asm/alternative.h>
#include "mm.h"
/*
* We need to be able to catch inadvertent references to memstart_addr
* that occur (potentially in generic code) before arm64_memblock_init()
* executes, which assigns it its actual value. So use a default value
* that cannot be mistaken for a real physical address.
*/
s64 memstart_addr __read_mostly = -1;
phys_addr_t arm64_dma_phys_limit __read_mostly;
s64 memstart_addr __ro_after_init = -1;
phys_addr_t arm64_dma_phys_limit __ro_after_init;
#ifdef CONFIG_BLK_DEV_INITRD
static int __init early_initrd(char *p)
......@@ -485,7 +485,12 @@ void free_initmem(void)
{
free_reserved_area(__va(__pa(__init_begin)), __va(__pa(__init_end)),
0, "unused kernel");
fixup_init();
/*
* Unmap the __init region but leave the VM area in place. This
* prevents the region from being reused for kernel modules, which
* is not supported by kallsyms.
*/
unmap_kernel_range((u64)__init_begin, (u64)(__init_end - __init_begin));
}
#ifdef CONFIG_BLK_DEV_INITRD
......
arch/arm64/mm/mm.h deleted 100644 → 0
View file @ c8d2bc9b
void fixup_init(void);
arch/arm64/mm/mmu.c
View file @ 7af8a0f8
......@@ -17,6 +17,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <linux/cache.h>
#include <linux/export.h>
#include <linux/kernel.h>
#include <linux/errno.h>
......@@ -42,11 +43,9 @@
#include <asm/memblock.h>
#include <asm/mmu_context.h>
#include "mm.h"
u64 idmap_t0sz = TCR_T0SZ(VA_BITS);
u64 kimage_voffset __read_mostly;
u64 kimage_voffset __ro_after_init;
EXPORT_SYMBOL(kimage_voffset);
/*
......@@ -399,16 +398,6 @@ void mark_rodata_ro(void)
section_size, PAGE_KERNEL_RO);
}
void fixup_init(void)
{
/*
* Unmap the __init region but leave the VM area in place. This
* prevents the region from being reused for kernel modules, which
* is not supported by kallsyms.
*/
unmap_kernel_range((u64)__init_begin, (u64)(__init_end - __init_begin));
}
static void __init map_kernel_segment(pgd_t *pgd, void *va_start, void *va_end,
pgprot_t prot, struct vm_struct *vma)
{
......
arch/arm64/mm/numa.c
View file @ 7af8a0f8
......@@ -17,6 +17,8 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#define pr_fmt(fmt) "NUMA: " fmt
#include <linux/acpi.h>
#include <linux/bootmem.h>
#include <linux/memblock.h>
......@@ -24,6 +26,7 @@
#include <linux/of.h>
#include <asm/acpi.h>
#include <asm/sections.h>
struct pglist_data *node_data[MAX_NUMNODES] __read_mostly;
EXPORT_SYMBOL(node_data);
......@@ -38,10 +41,9 @@ static __init int numa_parse_early_param(char *opt)
{
if (!opt)
return -EINVAL;
if (!strncmp(opt, "off", 3)) {
pr_info("%s\n", "NUMA turned off");
if (!strncmp(opt, "off", 3))
numa_off = true;
}
return 0;
}
early_param("numa", numa_parse_early_param);
......@@ -93,7 +95,6 @@ void numa_clear_node(unsigned int cpu)
*/
static void __init setup_node_to_cpumask_map(void)
{
unsigned int cpu;
int node;
/* setup nr_node_ids if not done yet */
......@@ -106,11 +107,8 @@ static void __init setup_node_to_cpumask_map(void)
cpumask_clear(node_to_cpumask_map[node]);
}
for_each_possible_cpu(cpu)
set_cpu_numa_node(cpu, NUMA_NO_NODE);
/* cpumask_of_node() will now work */
pr_debug("NUMA: Node to cpumask map for %d nodes\n", nr_node_ids);
pr_debug("Node to cpumask map for %d nodes\n", nr_node_ids);
}
/*
......@@ -118,18 +116,77 @@ static void __init setup_node_to_cpumask_map(void)
*/
void numa_store_cpu_info(unsigned int cpu)
{
map_cpu_to_node(cpu, numa_off ? 0 : cpu_to_node_map[cpu]);
map_cpu_to_node(cpu, cpu_to_node_map[cpu]);
}
void __init early_map_cpu_to_node(unsigned int cpu, int nid)
{
/* fallback to node 0 */
if (nid < 0 || nid >= MAX_NUMNODES)
if (nid < 0 || nid >= MAX_NUMNODES || numa_off)
nid = 0;
cpu_to_node_map[cpu] = nid;
/*
* We should set the numa node of cpu0 as soon as possible, because it
* has already been set up online before. cpu_to_node(0) will soon be
* called.
*/
if (!cpu)
set_cpu_numa_node(cpu, nid);
}
#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
unsigned long __per_cpu_offset[NR_CPUS] __read_mostly;
EXPORT_SYMBOL(__per_cpu_offset);
static int __init early_cpu_to_node(int cpu)
{
return cpu_to_node_map[cpu];
}
static int __init pcpu_cpu_distance(unsigned int from, unsigned int to)
{
return node_distance(from, to);
}
static void * __init pcpu_fc_alloc(unsigned int cpu, size_t size,
size_t align)
{
int nid = early_cpu_to_node(cpu);
return memblock_virt_alloc_try_nid(size, align,
__pa(MAX_DMA_ADDRESS), MEMBLOCK_ALLOC_ACCESSIBLE, nid);
}
static void __init pcpu_fc_free(void *ptr, size_t size)
{
memblock_free_early(__pa(ptr), size);
}
void __init setup_per_cpu_areas(void)
{
unsigned long delta;
unsigned int cpu;
int rc;
/*
* Always reserve area for module percpu variables. That's
* what the legacy allocator did.
*/
rc = pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE,
PERCPU_DYNAMIC_RESERVE, PAGE_SIZE,
pcpu_cpu_distance,
pcpu_fc_alloc, pcpu_fc_free);
if (rc < 0)
panic("Failed to initialize percpu areas.");
delta = (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start;
for_each_possible_cpu(cpu)
__per_cpu_offset[cpu] = delta + pcpu_unit_offsets[cpu];
}
#endif
/**
* numa_add_memblk - Set node id to memblk
* @nid: NUMA node ID of the new memblk
......@@ -145,13 +202,13 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
ret = memblock_set_node(start, (end - start), &memblock.memory, nid);
if (ret < 0) {
pr_err("NUMA: memblock [0x%llx - 0x%llx] failed to add on node %d\n",
pr_err("memblock [0x%llx - 0x%llx] failed to add on node %d\n",
start, (end - 1), nid);
return ret;
}
node_set(nid, numa_nodes_parsed);
pr_info("NUMA: Adding memblock [0x%llx - 0x%llx] on node %d\n",
pr_info("Adding memblock [0x%llx - 0x%llx] on node %d\n",
start, (end - 1), nid);
return ret;
}
......@@ -166,19 +223,18 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
void *nd;
int tnid;
pr_info("NUMA: Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
nid, start_pfn << PAGE_SHIFT,
(end_pfn << PAGE_SHIFT) - 1);
pr_info("Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
nid, start_pfn << PAGE_SHIFT, (end_pfn << PAGE_SHIFT) - 1);
nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
nd = __va(nd_pa);
/* report and initialize */
pr_info("NUMA: NODE_DATA [mem %#010Lx-%#010Lx]\n",
pr_info("NODE_DATA [mem %#010Lx-%#010Lx]\n",
nd_pa, nd_pa + nd_size - 1);
tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT);
if (tnid != nid)
pr_info("NUMA: NODE_DATA(%d) on node %d\n", nid, tnid);
pr_info("NODE_DATA(%d) on node %d\n", nid, tnid);
node_data[nid] = nd;
memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
......@@ -235,8 +291,7 @@ static int __init numa_alloc_distance(void)
numa_distance[i * numa_distance_cnt + j] = i == j ?
LOCAL_DISTANCE : REMOTE_DISTANCE;
pr_debug("NUMA: Initialized distance table, cnt=%d\n",
numa_distance_cnt);
pr_debug("Initialized distance table, cnt=%d\n", numa_distance_cnt);
return 0;
}
......@@ -257,20 +312,20 @@ static int __init numa_alloc_distance(void)
void __init numa_set_distance(int from, int to, int distance)
{
if (!numa_distance) {
pr_warn_once("NUMA: Warning: distance table not allocated yet\n");
pr_warn_once("Warning: distance table not allocated yet\n");
return;
}
if (from >= numa_distance_cnt || to >= numa_distance_cnt ||
from < 0 || to < 0) {
pr_warn_once("NUMA: Warning: node ids are out of bound, from=%d to=%d distance=%d\n",
pr_warn_once("Warning: node ids are out of bound, from=%d to=%d distance=%d\n",
from, to, distance);
return;
}
if ((u8)distance != distance ||
(from == to && distance != LOCAL_DISTANCE)) {
pr_warn_once("NUMA: Warning: invalid distance parameter, from=%d to=%d distance=%d\n",
pr_warn_once("Warning: invalid distance parameter, from=%d to=%d distance=%d\n",
from, to, distance);
return;
}
......@@ -297,7 +352,7 @@ static int __init numa_register_nodes(void)
/* Check that valid nid is set to memblks */
for_each_memblock(memory, mblk)
if (mblk->nid == NUMA_NO_NODE || mblk->nid >= MAX_NUMNODES) {
pr_warn("NUMA: Warning: invalid memblk node %d [mem %#010Lx-%#010Lx]\n",
pr_warn("Warning: invalid memblk node %d [mem %#010Lx-%#010Lx]\n",
mblk->nid, mblk->base,
mblk->base + mblk->size - 1);
return -EINVAL;
......@@ -335,8 +390,10 @@ static int __init numa_init(int (*init_func)(void))
if (ret < 0)
return ret;
if (nodes_empty(numa_nodes_parsed))
if (nodes_empty(numa_nodes_parsed)) {
pr_info("No NUMA configuration found\n");
return -EINVAL;
}
ret = numa_register_nodes();
if (ret < 0)
......@@ -344,10 +401,6 @@ static int __init numa_init(int (*init_func)(void))
setup_node_to_cpumask_map();
/* init boot processor */
cpu_to_node_map[0] = 0;
map_cpu_to_node(0, 0);
return 0;
}
......@@ -367,10 +420,8 @@ static int __init dummy_numa_init(void)
if (numa_off)
pr_info("NUMA disabled\n"); /* Forced off on command line. */
else
pr_info("No NUMA configuration found\n");
pr_info("NUMA: Faking a node at [mem %#018Lx-%#018Lx]\n",
0LLU, PFN_PHYS(max_pfn) - 1);
pr_info("Faking a node at [mem %#018Lx-%#018Lx]\n",
0LLU, PFN_PHYS(max_pfn) - 1);
for_each_memblock(memory, mblk) {
ret = numa_add_memblk(0, mblk->base, mblk->base + mblk->size);
......
arch/arm64/mm/pageattr.c
View file @ 7af8a0f8
......@@ -139,4 +139,43 @@ void __kernel_map_pages(struct page *page, int numpages, int enable)
__pgprot(0),
__pgprot(PTE_VALID));
}
#endif
#ifdef CONFIG_HIBERNATION
/*
* When built with CONFIG_DEBUG_PAGEALLOC and CONFIG_HIBERNATION, this function
* is used to determine if a linear map page has been marked as not-valid by
* CONFIG_DEBUG_PAGEALLOC. Walk the page table and check the PTE_VALID bit.
* This is based on kern_addr_valid(), which almost does what we need.
*
* Because this is only called on the kernel linear map, p?d_sect() implies
* p?d_present(). When debug_pagealloc is enabled, sections mappings are
* disabled.
*/
bool kernel_page_present(struct page *page)
{
pgd_t *pgd;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
unsigned long addr = (unsigned long)page_address(page);
pgd = pgd_offset_k(addr);
if (pgd_none(*pgd))
return false;
pud = pud_offset(pgd, addr);
if (pud_none(*pud))
return false;
if (pud_sect(*pud))
return true;
pmd = pmd_offset(pud, addr);
if (pmd_none(*pmd))
return false;
if (pmd_sect(*pmd))
return true;
pte = pte_offset_kernel(pmd, addr);
return pte_valid(*pte);
}
#endif /* CONFIG_HIBERNATION */
#endif /* CONFIG_DEBUG_PAGEALLOC */
arch/arm64/mm/pgd.c
View file @ 7af8a0f8
......@@ -26,8 +26,6 @@
#include <asm/page.h>
#include <asm/tlbflush.h>
#include "mm.h"
static struct kmem_cache *pgd_cache;
pgd_t *pgd_alloc(struct mm_struct *mm)
......
arch/arm64/mm/proc.S
View file @ 7af8a0f8
......@@ -83,6 +83,7 @@ ENDPROC(cpu_do_suspend)
*
* x0: Address of context pointer
*/
.pushsection ".idmap.text", "ax"
ENTRY(cpu_do_resume)
ldp x2, x3, [x0]
ldp x4, x5, [x0, #16]
......@@ -120,6 +121,7 @@ ENTRY(cpu_do_resume)
isb
ret
ENDPROC(cpu_do_resume)
.popsection
#endif
/*
......@@ -134,17 +136,12 @@ ENTRY(cpu_do_switch_mm)
bfi x0, x1, #48, #16 // set the ASID
msr ttbr0_el1, x0 // set TTBR0
isb
alternative_if_not ARM64_WORKAROUND_CAVIUM_27456
ret
nop
nop
nop
alternative_else
alternative_if ARM64_WORKAROUND_CAVIUM_27456
ic iallu
dsb nsh
isb
alternative_else_nop_endif
ret
alternative_endif
ENDPROC(cpu_do_switch_mm)
.pushsection ".idmap.text", "ax"
......@@ -181,6 +178,7 @@ ENDPROC(idmap_cpu_replace_ttbr1)
* Initialise the processor for turning the MMU on. Return in x0 the
* value of the SCTLR_EL1 register.
*/
.pushsection ".idmap.text", "ax"
ENTRY(__cpu_setup)
tlbi vmalle1 // Invalidate local TLB
dsb nsh
......@@ -266,3 +264,4 @@ ENDPROC(__cpu_setup)
crval:
.word 0xfcffffff // clear
.word 0x34d5d91d // set
.popsection
drivers/clocksource/Kconfig
View file @ 7af8a0f8
......@@ -305,6 +305,16 @@ config ARM_ARCH_TIMER_EVTSTREAM
This must be disabled for hardware validation purposes to detect any
hardware anomalies of missing events.
config FSL_ERRATUM_A008585
bool "Workaround for Freescale/NXP Erratum A-008585"
default y
depends on ARM_ARCH_TIMER && ARM64
help
This option enables a workaround for Freescale/NXP Erratum
A-008585 ("ARM generic timer may contain an erroneous
value"). The workaround will only be active if the
fsl,erratum-a008585 property is found in the timer node.
config ARM_GLOBAL_TIMER
bool "Support for the ARM global timer" if COMPILE_TEST
select CLKSRC_OF if OF
......
drivers/clocksource/arm_arch_timer.c
View file @ 7af8a0f8
......@@ -94,6 +94,43 @@ early_param("clocksource.arm_arch_timer.evtstrm", early_evtstrm_cfg);
* Architected system timer support.
*/
#ifdef CONFIG_FSL_ERRATUM_A008585
DEFINE_STATIC_KEY_FALSE(arch_timer_read_ool_enabled);
EXPORT_SYMBOL_GPL(arch_timer_read_ool_enabled);
static int fsl_a008585_enable = -1;
static int __init early_fsl_a008585_cfg(char *buf)
{
int ret;
bool val;
ret = strtobool(buf, &val);
if (ret)
return ret;
fsl_a008585_enable = val;
return 0;
}
early_param("clocksource.arm_arch_timer.fsl-a008585", early_fsl_a008585_cfg);
u32 __fsl_a008585_read_cntp_tval_el0(void)
{
return __fsl_a008585_read_reg(cntp_tval_el0);
}
u32 __fsl_a008585_read_cntv_tval_el0(void)
{
return __fsl_a008585_read_reg(cntv_tval_el0);
}
u64 __fsl_a008585_read_cntvct_el0(void)
{
return __fsl_a008585_read_reg(cntvct_el0);
}
EXPORT_SYMBOL(__fsl_a008585_read_cntvct_el0);
#endif /* CONFIG_FSL_ERRATUM_A008585 */
static __always_inline
void arch_timer_reg_write(int access, enum arch_timer_reg reg, u32 val,
struct clock_event_device *clk)
......@@ -243,6 +280,40 @@ static __always_inline void set_next_event(const int access, unsigned long evt,
arch_timer_reg_write(access, ARCH_TIMER_REG_CTRL, ctrl, clk);
}
#ifdef CONFIG_FSL_ERRATUM_A008585
static __always_inline void fsl_a008585_set_next_event(const int access,
unsigned long evt, struct clock_event_device *clk)
{
unsigned long ctrl;
u64 cval = evt + arch_counter_get_cntvct();
ctrl = arch_timer_reg_read(access, ARCH_TIMER_REG_CTRL, clk);
ctrl |= ARCH_TIMER_CTRL_ENABLE;
ctrl &= ~ARCH_TIMER_CTRL_IT_MASK;
if (access == ARCH_TIMER_PHYS_ACCESS)
write_sysreg(cval, cntp_cval_el0);
else if (access == ARCH_TIMER_VIRT_ACCESS)
write_sysreg(cval, cntv_cval_el0);
arch_timer_reg_write(access, ARCH_TIMER_REG_CTRL, ctrl, clk);
}
static int fsl_a008585_set_next_event_virt(unsigned long evt,
struct clock_event_device *clk)
{
fsl_a008585_set_next_event(ARCH_TIMER_VIRT_ACCESS, evt, clk);
return 0;
}
static int fsl_a008585_set_next_event_phys(unsigned long evt,
struct clock_event_device *clk)
{
fsl_a008585_set_next_event(ARCH_TIMER_PHYS_ACCESS, evt, clk);
return 0;
}
#endif /* CONFIG_FSL_ERRATUM_A008585 */
static int arch_timer_set_next_event_virt(unsigned long evt,
struct clock_event_device *clk)
{
......@@ -271,6 +342,19 @@ static int arch_timer_set_next_event_phys_mem(unsigned long evt,
return 0;
}
static void fsl_a008585_set_sne(struct clock_event_device *clk)
{
#ifdef CONFIG_FSL_ERRATUM_A008585
if (!static_branch_unlikely(&arch_timer_read_ool_enabled))
return;
if (arch_timer_uses_ppi == VIRT_PPI)
clk->set_next_event = fsl_a008585_set_next_event_virt;
else
clk->set_next_event = fsl_a008585_set_next_event_phys;
#endif
}
static void __arch_timer_setup(unsigned type,
struct clock_event_device *clk)
{
......@@ -299,6 +383,8 @@ static void __arch_timer_setup(unsigned type,
default:
BUG();
}
fsl_a008585_set_sne(clk);
} else {
clk->features |= CLOCK_EVT_FEAT_DYNIRQ;
clk->name = "arch_mem_timer";
......@@ -515,15 +601,19 @@ static void __init arch_counter_register(unsigned type)
arch_timer_read_counter = arch_counter_get_cntvct;
else
arch_timer_read_counter = arch_counter_get_cntpct;
} else {
arch_timer_read_counter = arch_counter_get_cntvct_mem;
/* If the clocksource name is "arch_sys_counter" the
* VDSO will attempt to read the CP15-based counter.
* Ensure this does not happen when CP15-based
* counter is not available.
clocksource_counter.archdata.vdso_direct = true;
#ifdef CONFIG_FSL_ERRATUM_A008585
/*
* Don't use the vdso fastpath if errata require using
* the out-of-line counter accessor.
*/
clocksource_counter.name = "arch_mem_counter";
if (static_branch_unlikely(&arch_timer_read_ool_enabled))
clocksource_counter.archdata.vdso_direct = false;
#endif
} else {
arch_timer_read_counter = arch_counter_get_cntvct_mem;
}
start_count = arch_timer_read_counter();
......@@ -800,6 +890,15 @@ static int __init arch_timer_of_init(struct device_node *np)
arch_timer_c3stop = !of_property_read_bool(np, "always-on");
#ifdef CONFIG_FSL_ERRATUM_A008585
if (fsl_a008585_enable < 0)
fsl_a008585_enable = of_property_read_bool(np, "fsl,erratum-a008585");
if (fsl_a008585_enable) {
static_branch_enable(&arch_timer_read_ool_enabled);
pr_info("Enabling workaround for FSL erratum A-008585\n");
}
#endif
/*
* If we cannot rely on firmware initializing the timer registers then
* we should use the physical timers instead.
......
drivers/of/of_numa.c
View file @ 7af8a0f8
This diff is collapsed.
drivers/perf/arm_pmu.c
View file @ 7af8a0f8
This diff is collapsed.
include/linux/cpu.h
View file @ 7af8a0f8
......@@ -228,7 +228,11 @@ static inline void cpu_hotplug_done(void) {}
#endif /* CONFIG_HOTPLUG_CPU */
#ifdef CONFIG_PM_SLEEP_SMP
extern int disable_nonboot_cpus(void);
extern int freeze_secondary_cpus(int primary);
static inline int disable_nonboot_cpus(void)
{
return freeze_secondary_cpus(0);
}
extern void enable_nonboot_cpus(void);
#else /* !CONFIG_PM_SLEEP_SMP */
static inline int disable_nonboot_cpus(void) { return 0; }
......
include/linux/cpuhotplug.h
View file @ 7af8a0f8
This diff is collapsed.
include/linux/jump_label.h
View file @ 7af8a0f8
This diff is collapsed.
include/linux/perf/arm_pmu.h
View file @ 7af8a0f8
This diff is collapsed.
kernel/cpu.c
View file @ 7af8a0f8
This diff is collapsed.
mm/mmap.c
View file @ 7af8a0f8
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7