1. 30 Sep, 2022 1 commit
    • Catalin Marinas's avatar
      Merge branches 'for-next/doc', 'for-next/sve', 'for-next/sysreg',... · b23ec74c
      Catalin Marinas authored
      Merge branches 'for-next/doc', 'for-next/sve', 'for-next/sysreg', 'for-next/gettimeofday', 'for-next/stacktrace', 'for-next/atomics', 'for-next/el1-exceptions', 'for-next/a510-erratum-2658417', 'for-next/defconfig', 'for-next/tpidr2_el0' and 'for-next/ftrace', remote-tracking branch 'arm64/for-next/perf' into for-next/core
      
      * arm64/for-next/perf:
        arm64: asm/perf_regs.h: Avoid C++-style comment in UAPI header
        arm64/sve: Add Perf extensions documentation
        perf: arm64: Add SVE vector granule register to user regs
        MAINTAINERS: add maintainers for Alibaba' T-Head PMU driver
        drivers/perf: add DDR Sub-System Driveway PMU driver for Yitian 710 SoC
        docs: perf: Add description for Alibaba's T-Head PMU driver
      
      * for-next/doc:
        : Documentation/arm64 updates
        arm64/sve: Document our actual ABI for clearing registers on syscall
      
      * for-next/sve:
        : SVE updates
        arm64/sysreg: Add hwcap for SVE EBF16
      
      * for-next/sysreg: (35 commits)
        : arm64 system registers generation (more conversions)
        arm64/sysreg: Fix a few missed conversions
        arm64/sysreg: Convert ID_AA64AFRn_EL1 to automatic generation
        arm64/sysreg: Convert ID_AA64DFR1_EL1 to automatic generation
        arm64/sysreg: Convert ID_AA64FDR0_EL1 to automatic generation
        arm64/sysreg: Use feature numbering for PMU and SPE revisions
        arm64/sysreg: Add _EL1 into ID_AA64DFR0_EL1 definition names
        arm64/sysreg: Align field names in ID_AA64DFR0_EL1 with architecture
        arm64/sysreg: Add defintion for ALLINT
        arm64/sysreg: Convert SCXTNUM_EL1 to automatic generation
        arm64/sysreg: Convert TIPDR_EL1 to automatic generation
        arm64/sysreg: Convert ID_AA64PFR1_EL1 to automatic generation
        arm64/sysreg: Convert ID_AA64PFR0_EL1 to automatic generation
        arm64/sysreg: Convert ID_AA64MMFR2_EL1 to automatic generation
        arm64/sysreg: Convert ID_AA64MMFR1_EL1 to automatic generation
        arm64/sysreg: Convert ID_AA64MMFR0_EL1 to automatic generation
        arm64/sysreg: Convert HCRX_EL2 to automatic generation
        arm64/sysreg: Standardise naming of ID_AA64PFR1_EL1 SME enumeration
        arm64/sysreg: Standardise naming of ID_AA64PFR1_EL1 BTI enumeration
        arm64/sysreg: Standardise naming of ID_AA64PFR1_EL1 fractional version fields
        arm64/sysreg: Standardise naming for MTE feature enumeration
        ...
      
      * for-next/gettimeofday:
        : Use self-synchronising counter access in gettimeofday() (if FEAT_ECV)
        arm64: vdso: use SYS_CNTVCTSS_EL0 for gettimeofday
        arm64: alternative: patch alternatives in the vDSO
        arm64: module: move find_section to header
      
      * for-next/stacktrace:
        : arm64 stacktrace cleanups and improvements
        arm64: stacktrace: track hyp stacks in unwinder's address space
        arm64: stacktrace: track all stack boundaries explicitly
        arm64: stacktrace: remove stack type from fp translator
        arm64: stacktrace: rework stack boundary discovery
        arm64: stacktrace: add stackinfo_on_stack() helper
        arm64: stacktrace: move SDEI stack helpers to stacktrace code
        arm64: stacktrace: rename unwind_next_common() -> unwind_next_frame_record()
        arm64: stacktrace: simplify unwind_next_common()
        arm64: stacktrace: fix kerneldoc comments
      
      * for-next/atomics:
        : arm64 atomics improvements
        arm64: atomic: always inline the assembly
        arm64: atomics: remove LL/SC trampolines
      
      * for-next/el1-exceptions:
        : Improve the reporting of EL1 exceptions
        arm64: rework BTI exception handling
        arm64: rework FPAC exception handling
        arm64: consistently pass ESR_ELx to die()
        arm64: die(): pass 'err' as long
        arm64: report EL1 UNDEFs better
      
      * for-next/a510-erratum-2658417:
        : Cortex-A510: 2658417: remove BF16 support due to incorrect result
        arm64: errata: remove BF16 HWCAP due to incorrect result on Cortex-A510
        arm64: cpufeature: Expose get_arm64_ftr_reg() outside cpufeature.c
        arm64: cpufeature: Force HWCAP to be based on the sysreg visible to user-space
      
      * for-next/defconfig:
        : arm64 defconfig updates
        arm64: defconfig: Add Coresight as module
        arm64: Enable docker support in defconfig
        arm64: defconfig: Enable memory hotplug and hotremove config
        arm64: configs: Enable all PMUs provided by Arm
      
      * for-next/tpidr2_el0:
        : arm64 ptrace() support for TPIDR2_EL0
        kselftest/arm64: Add coverage of TPIDR2_EL0 ptrace interface
        arm64/ptrace: Support access to TPIDR2_EL0
        arm64/ptrace: Document extension of NT_ARM_TLS to cover TPIDR2_EL0
        kselftest/arm64: Add test coverage for NT_ARM_TLS
      
      * for-next/ftrace:
        : arm64 ftraces updates/fixes
        arm64: ftrace: fix module PLTs with mcount
        arm64: module: Remove unused plt_entry_is_initialized()
        arm64: module: Make plt_equals_entry() static
      b23ec74c
  2. 29 Sep, 2022 4 commits
  3. 22 Sep, 2022 6 commits
  4. 21 Sep, 2022 8 commits
  5. 16 Sep, 2022 14 commits
    • James Morse's avatar
      arm64: errata: remove BF16 HWCAP due to incorrect result on Cortex-A510 · 1bdb0fbb
      James Morse authored
      Cortex-A510's erratum #2658417 causes two BF16 instructions to return the
      wrong result in rare circumstances when a pair of A510 CPUs are using
      shared neon hardware.
      
      The two instructions affected are BFMMLA and VMMLA, support for these is
      indicated by the BF16 HWCAP. Remove it on affected platforms.
      Signed-off-by: default avatarJames Morse <james.morse@arm.com>
      Link: https://lore.kernel.org/r/20220909165938.3931307-4-james.morse@arm.com
      [catalin.marinas@arm.com: add revision to the Kconfig help; remove .type]
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      1bdb0fbb
    • James Morse's avatar
      arm64: cpufeature: Expose get_arm64_ftr_reg() outside cpufeature.c · 445c953e
      James Morse authored
      get_arm64_ftr_reg() returns the properties of a system register based
      on its instruction encoding.
      
      This is needed by erratum workaround in cpu_errata.c to modify the
      user-space visible view of id registers.
      Signed-off-by: default avatarJames Morse <james.morse@arm.com>
      Reviewed-by: default avatarSuzuki K Poulose <suzuki.poulose@arm.com>
      Link: https://lore.kernel.org/r/20220909165938.3931307-3-james.morse@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      445c953e
    • James Morse's avatar
      arm64: cpufeature: Force HWCAP to be based on the sysreg visible to user-space · 237405eb
      James Morse authored
      arm64 advertises hardware features to user-space via HWCAPs, and by
      emulating access to the CPUs id registers. The cpufeature code has a
      sanitised system-wide view of an id register, and a sanitised user-space
      view of an id register, where some features use their 'safe' value
      instead of the hardware value.
      
      It is currently possible for a HWCAP to be advertised where the user-space
      view of the id register does not show the feature as supported.
      Erratum workaround need to remove both the HWCAP, and the feature from
      the user-space view of the id register. This involves duplicating the
      code, and spreading it over cpufeature.c and cpu_errata.c.
      
      Make the HWCAP code use the user-space view of id registers. This ensures
      the values never diverge, and allows erratum workaround to remove HWCAP
      by modifying the user-space view of the id register.
      Signed-off-by: default avatarJames Morse <james.morse@arm.com>
      Reviewed-by: default avatarSuzuki K Poulose <suzuki.poulose@arm.com>
      Link: https://lore.kernel.org/r/20220909165938.3931307-2-james.morse@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      237405eb
    • Mark Brown's avatar
      arm64/sysreg: Convert ID_AA64AFRn_EL1 to automatic generation · 10453bf1
      Mark Brown authored
      Convert ID_AA64AFRn_EL1 to automatic generation as per DDI0487I.a, no
      functional changes.
      Signed-off-by: default avatarMark Brown <broonie@kernel.org>
      Link: https://lore.kernel.org/r/20220910163354.860255-7-broonie@kernel.orgSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      10453bf1
    • Mark Brown's avatar
      arm64/sysreg: Convert ID_AA64DFR1_EL1 to automatic generation · c65c6178
      Mark Brown authored
      Convert ID_AA64FDR1_EL1 to automatic generation as per DDI0487I.a, no
      functional changes.
      Signed-off-by: default avatarMark Brown <broonie@kernel.org>
      Link: https://lore.kernel.org/r/20220910163354.860255-6-broonie@kernel.orgSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      c65c6178
    • Mark Brown's avatar
      arm64/sysreg: Convert ID_AA64FDR0_EL1 to automatic generation · e62a2d26
      Mark Brown authored
      Convert ID_AA64DFR0_EL1 to automatic generation as per DDI0487I.a, no
      functional changes.
      Signed-off-by: default avatarMark Brown <broonie@kernel.org>
      Link: https://lore.kernel.org/r/20220910163354.860255-5-broonie@kernel.orgSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      e62a2d26
    • Mark Brown's avatar
      arm64/sysreg: Use feature numbering for PMU and SPE revisions · 121a8fc0
      Mark Brown authored
      Currently the kernel refers to the versions of the PMU and SPE features by
      the version of the architecture where those features were updated but the
      ARM refers to them using the FEAT_ names for the features. To improve
      consistency and help with updating for newer features and since v9 will
      make our current naming scheme a bit more confusing update the macros
      identfying features to use the FEAT_ based scheme.
      Signed-off-by: default avatarMark Brown <broonie@kernel.org>
      Link: https://lore.kernel.org/r/20220910163354.860255-4-broonie@kernel.orgSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      121a8fc0
    • Mark Brown's avatar
      arm64/sysreg: Add _EL1 into ID_AA64DFR0_EL1 definition names · fcf37b38
      Mark Brown authored
      Normally we include the full register name in the defines for fields within
      registers but this has not been followed for ID registers. In preparation
      for automatic generation of defines add the _EL1s into the defines for
      ID_AA64DFR0_EL1 to follow the convention. No functional changes.
      Signed-off-by: default avatarMark Brown <broonie@kernel.org>
      Link: https://lore.kernel.org/r/20220910163354.860255-3-broonie@kernel.orgSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      fcf37b38
    • Mark Brown's avatar
      arm64/sysreg: Align field names in ID_AA64DFR0_EL1 with architecture · c0357a73
      Mark Brown authored
      The naming scheme the architecture uses for the fields in ID_AA64DFR0_EL1
      does not align well with kernel conventions, using as it does a lot of
      MixedCase in various arrangements. In preparation for automatically
      generating the defines for this register rename the defines used to match
      what is in the architecture.
      Signed-off-by: default avatarMark Brown <broonie@kernel.org>
      Link: https://lore.kernel.org/r/20220910163354.860255-2-broonie@kernel.orgSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      c0357a73
    • Mark Rutland's avatar
      arm64: rework BTI exception handling · 830a2a4d
      Mark Rutland authored
      If a BTI exception is taken from EL1, the entry code will treat this as
      an unhandled exception and will panic() the kernel. This is inconsistent
      with the way we handle FPAC exceptions, which have a dedicated handler
      and only necessarily kill the thread from which the exception was taken
      from, and we don't log all the information that could be relevant to
      debug the issue.
      
      The code in do_bti() has:
      
      	BUG_ON(!user_mode(regs));
      
      ... and it seems like the intent was to call this for EL1 BTI
      exceptions, as with FPAC, but this was omitted due to an oversight.
      
      This patch adds separate EL0 and EL1 BTI exception handlers, with the
      latter calling die() directly to report the original context the BTI
      exception was taken from. This matches our handling of FPAC exceptions.
      
      Prior to this patch, a BTI failure is reported as:
      
      | Unhandled 64-bit el1h sync exception on CPU0, ESR 0x0000000034000002 -- BTI
      | CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.19.0-rc3-00131-g7d937ff0221d-dirty #9
      | Hardware name: linux,dummy-virt (DT)
      | pstate: 20400809 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=-c)
      | pc : test_bti_callee+0x4/0x10
      | lr : test_bti_caller+0x1c/0x28
      | sp : ffff80000800bdf0
      | x29: ffff80000800bdf0 x28: 0000000000000000 x27: 0000000000000000
      | x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
      | x23: ffff80000a2b8000 x22: 0000000000000000 x21: 0000000000000000
      | x20: ffff8000099fa5b0 x19: ffff800009ff7000 x18: fffffbfffda37000
      | x17: 3120676e696d7573 x16: 7361202c6e6f6974 x15: 0000000041a90000
      | x14: 0040000000000041 x13: 0040000000000001 x12: ffff000001a90000
      | x11: fffffbfffda37480 x10: 0068000000000703 x9 : 0001000040000000
      | x8 : 0000000000090000 x7 : 0068000000000f03 x6 : 0060000000000f83
      | x5 : ffff80000a2b6000 x4 : ffff0000028d0000 x3 : ffff800009f78378
      | x2 : 0000000000000000 x1 : 0000000040210000 x0 : ffff8000080257e4
      | Kernel panic - not syncing: Unhandled exception
      | CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.19.0-rc3-00131-g7d937ff0221d-dirty #9
      | Hardware name: linux,dummy-virt (DT)
      | Call trace:
      |  dump_backtrace.part.0+0xcc/0xe0
      |  show_stack+0x18/0x5c
      |  dump_stack_lvl+0x64/0x80
      |  dump_stack+0x18/0x34
      |  panic+0x170/0x360
      |  arm64_exit_nmi.isra.0+0x0/0x80
      |  el1h_64_sync_handler+0x64/0xd0
      |  el1h_64_sync+0x64/0x68
      |  test_bti_callee+0x4/0x10
      |  smp_cpus_done+0xb0/0xbc
      |  smp_init+0x7c/0x8c
      |  kernel_init_freeable+0x128/0x28c
      |  kernel_init+0x28/0x13c
      |  ret_from_fork+0x10/0x20
      
      With this patch applied, a BTI failure is reported as:
      
      | Internal error: Oops - BTI: 0000000034000002 [#1] PREEMPT SMP
      | Modules linked in:
      | CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.19.0-rc3-00132-g0ad98265d582-dirty #8
      | Hardware name: linux,dummy-virt (DT)
      | pstate: 20400809 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=-c)
      | pc : test_bti_callee+0x4/0x10
      | lr : test_bti_caller+0x1c/0x28
      | sp : ffff80000800bdf0
      | x29: ffff80000800bdf0 x28: 0000000000000000 x27: 0000000000000000
      | x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
      | x23: ffff80000a2b8000 x22: 0000000000000000 x21: 0000000000000000
      | x20: ffff8000099fa5b0 x19: ffff800009ff7000 x18: fffffbfffda37000
      | x17: 3120676e696d7573 x16: 7361202c6e6f6974 x15: 0000000041a90000
      | x14: 0040000000000041 x13: 0040000000000001 x12: ffff000001a90000
      | x11: fffffbfffda37480 x10: 0068000000000703 x9 : 0001000040000000
      | x8 : 0000000000090000 x7 : 0068000000000f03 x6 : 0060000000000f83
      | x5 : ffff80000a2b6000 x4 : ffff0000028d0000 x3 : ffff800009f78378
      | x2 : 0000000000000000 x1 : 0000000040210000 x0 : ffff800008025804
      | Call trace:
      |  test_bti_callee+0x4/0x10
      |  smp_cpus_done+0xb0/0xbc
      |  smp_init+0x7c/0x8c
      |  kernel_init_freeable+0x128/0x28c
      |  kernel_init+0x28/0x13c
      |  ret_from_fork+0x10/0x20
      | Code: d50323bf d53cd040 d65f03c0 d503233f (d50323bf)
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Reviewed-by: default avatarAnshuman Khandual <anshuman.khandual@arm.com>
      Cc: Alexandru Elisei <alexandru.elisei@arm.com>
      Cc: Amit Daniel Kachhap <amit.kachhap@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220913101732.3925290-6-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      830a2a4d
    • Mark Rutland's avatar
      arm64: rework FPAC exception handling · a1fafa3b
      Mark Rutland authored
      If an FPAC exception is taken from EL1, the entry code will call
      do_ptrauth_fault(), where due to:
      
      	BUG_ON(!user_mode(regs))
      
      ... the kernel will report a problem within do_ptrauth_fault() rather
      than reporting the original context the FPAC exception was taken from.
      The pt_regs and ESR value reported will be from within
      do_ptrauth_fault() and the code dump will be for the BRK in BUG_ON(),
      which isn't sufficient to debug the cause of the original exception.
      
      This patch makes the reporting better by having separate EL0 and EL1
      FPAC exception handlers, with the latter calling die() directly to
      report the original context the FPAC exception was taken from.
      
      Note that we only need to prevent kprobes of the EL1 FPAC handler, since
      the EL0 FPAC handler cannot be called recursively.
      
      For consistency with do_el0_svc*(), I've named the split functions
      do_el{0,1}_fpac() rather than do_el{0,1}_ptrauth_fault(). I've also
      clarified the comment to not imply there are casues other than FPAC
      exceptions.
      
      Prior to this patch FPAC exceptions are reported as:
      
      | kernel BUG at arch/arm64/kernel/traps.c:517!
      | Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT SMP
      | Modules linked in:
      | CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.19.0-rc3-00130-g9c8a180a1cdf-dirty #12
      | Hardware name: FVP Base RevC (DT)
      | pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
      | pc : do_ptrauth_fault+0x3c/0x40
      | lr : el1_fpac+0x34/0x54
      | sp : ffff80000a3bbc80
      | x29: ffff80000a3bbc80 x28: ffff0008001d8000 x27: 0000000000000000
      | x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
      | x23: 0000000020400009 x22: ffff800008f70fa4 x21: ffff80000a3bbe00
      | x20: 0000000072000000 x19: ffff80000a3bbcb0 x18: fffffbfffda37000
      | x17: 3120676e696d7573 x16: 7361202c6e6f6974 x15: 0000000081a90000
      | x14: 0040000000000041 x13: 0040000000000001 x12: ffff000001a90000
      | x11: fffffbfffda37480 x10: 0068000000000703 x9 : 0001000080000000
      | x8 : 0000000000090000 x7 : 0068000000000f03 x6 : 0060000000000783
      | x5 : ffff80000a3bbcb0 x4 : ffff0008001d8000 x3 : 0000000072000000
      | x2 : 0000000000000000 x1 : 0000000020400009 x0 : ffff80000a3bbcb0
      | Call trace:
      |  do_ptrauth_fault+0x3c/0x40
      |  el1h_64_sync_handler+0xc4/0xd0
      |  el1h_64_sync+0x64/0x68
      |  test_pac+0x8/0x10
      |  smp_init+0x7c/0x8c
      |  kernel_init_freeable+0x128/0x28c
      |  kernel_init+0x28/0x13c
      |  ret_from_fork+0x10/0x20
      | Code: 97fffe5e a8c17bfd d50323bf d65f03c0 (d4210000)
      
      With this patch applied FPAC exceptions are reported as:
      
      | Internal error: Oops - FPAC: 0000000072000000 [#1] PREEMPT SMP
      | Modules linked in:
      | CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.19.0-rc3-00132-g78846e1c4757-dirty #11
      | Hardware name: FVP Base RevC (DT)
      | pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
      | pc : test_pac+0x8/0x10
      | lr : 0x0
      | sp : ffff80000a3bbe00
      | x29: ffff80000a3bbe00 x28: 0000000000000000 x27: 0000000000000000
      | x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
      | x23: ffff80000a2c8000 x22: 0000000000000000 x21: 0000000000000000
      | x20: ffff8000099fa5b0 x19: ffff80000a007000 x18: fffffbfffda37000
      | x17: 3120676e696d7573 x16: 7361202c6e6f6974 x15: 0000000081a90000
      | x14: 0040000000000041 x13: 0040000000000001 x12: ffff000001a90000
      | x11: fffffbfffda37480 x10: 0068000000000703 x9 : 0001000080000000
      | x8 : 0000000000090000 x7 : 0068000000000f03 x6 : 0060000000000783
      | x5 : ffff80000a2c6000 x4 : ffff0008001d8000 x3 : ffff800009f88378
      | x2 : 0000000000000000 x1 : 0000000080210000 x0 : ffff000001a90000
      | Call trace:
      |  test_pac+0x8/0x10
      |  smp_init+0x7c/0x8c
      |  kernel_init_freeable+0x128/0x28c
      |  kernel_init+0x28/0x13c
      |  ret_from_fork+0x10/0x20
      | Code: d50323bf d65f03c0 d503233f aa1f03fe (d50323bf)
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Reviewed-by: default avatarAnshuman Khandual <anshuman.khandual@arm.com>
      Cc: Alexandru Elisei <alexandru.elisei@arm.com>
      Cc: Amit Daniel Kachhap <amit.kachhap@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220913101732.3925290-5-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      a1fafa3b
    • Mark Rutland's avatar
      arm64: consistently pass ESR_ELx to die() · 0f2cb928
      Mark Rutland authored
      Currently, bug_handler() and kasan_handler() call die() with '0' as the
      'err' value, whereas die_kernel_fault() passes the ESR_ELx value.
      
      For consistency, this patch ensures we always pass the ESR_ELx value to
      die(). As this is only called for exceptions taken from kernel mode,
      there should be no user-visible change as a result of this patch.
      
      For UNDEFINED exceptions, I've had to modify do_undefinstr() and its
      callers to pass the ESR_ELx value. In all cases the ESR_ELx value had
      already been read and was available.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Mark Brown <broonie@kernel.org>
      Cc: Alexandru Elisei <alexandru.elisei@arm.com>
      Cc: Amit Daniel Kachhap <amit.kachhap@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will@kernel.org>
      Reviewed-by: default avatarAnshuman Khandual <anshuman.khandual@arm.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Link: https://lore.kernel.org/r/20220913101732.3925290-4-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      0f2cb928
    • Mark Rutland's avatar
      arm64: die(): pass 'err' as long · 18906ff9
      Mark Rutland authored
      Recently, we reworked a lot of code to consistentlt pass ESR_ELx as a
      64-bit quantity. However, we missed that this can be passed into die()
      and __die() as the 'err' parameter where it is truncated to a 32-bit
      int.
      
      As notify_die() already takes 'err' as a long, this patch changes die()
      and __die() to also take 'err' as a long, ensuring that the full value
      of ESR_ELx is retained.
      
      At the same time, die() is updated to consistently log 'err' as a
      zero-padded 64-bit quantity.
      
      Subsequent patches will pass the ESR_ELx value to die() for a number of
      exceptions.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Reviewed-by: default avatarAnshuman Khandual <anshuman.khandual@arm.com>
      Cc: Alexandru Elisei <alexandru.elisei@arm.com>
      Cc: Amit Daniel Kachhap <amit.kachhap@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220913101732.3925290-3-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      18906ff9
    • Mark Rutland's avatar
      arm64: report EL1 UNDEFs better · b502c87d
      Mark Rutland authored
      If an UNDEFINED exception is taken from EL1, and do_undefinstr() doesn't
      find any suitable undef_hook, it will call:
      
      	BUG_ON(!user_mode(regs))
      
      ... and the kernel will report a failure witin do_undefinstr() rather
      than reporting the original context that the UNDEFINED exception was
      taken from. The pt_regs and ESR value reported within the BUG() handler
      will be from within do_undefinstr() and the code dump will be for the
      BRK in BUG_ON(), which isn't sufficient to debug the cause of the
      original exception.
      
      This patch makes the reporting better by having do_undefinstr() call
      die() directly in this case to report the original context from which
      the UNDEFINED exception was taken.
      
      Prior to this patch, an undefined instruction is reported as:
      
      | kernel BUG at arch/arm64/kernel/traps.c:497!
      | Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
      | Modules linked in:
      | CPU: 0 PID: 0 Comm: swapper Not tainted 5.19.0-rc3-00127-geff044f1b04e-dirty #3
      | Hardware name: linux,dummy-virt (DT)
      | pstate: 000000c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
      | pc : do_undefinstr+0x28c/0x2ac
      | lr : do_undefinstr+0x298/0x2ac
      | sp : ffff800009f63bc0
      | x29: ffff800009f63bc0 x28: ffff800009f73c00 x27: ffff800009644a70
      | x26: ffff8000096778a8 x25: 0000000000000040 x24: 0000000000000000
      | x23: 00000000800000c5 x22: ffff800009894060 x21: ffff800009f63d90
      | x20: 0000000000000000 x19: ffff800009f63c40 x18: 0000000000000006
      | x17: 0000000000403000 x16: 00000000bfbfd000 x15: ffff800009f63830
      | x14: ffffffffffffffff x13: 0000000000000000 x12: 0000000000000019
      | x11: 0101010101010101 x10: 0000000000161b98 x9 : 0000000000000000
      | x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
      | x5 : ffff800009f761d0 x4 : 0000000000000000 x3 : ffff80000a2b80f8
      | x2 : 0000000000000000 x1 : ffff800009f73c00 x0 : 00000000800000c5
      | Call trace:
      |  do_undefinstr+0x28c/0x2ac
      |  el1_undef+0x2c/0x4c
      |  el1h_64_sync_handler+0x84/0xd0
      |  el1h_64_sync+0x64/0x68
      |  setup_arch+0x550/0x598
      |  start_kernel+0x88/0x6ac
      |  __primary_switched+0xb8/0xc0
      | Code: 17ffff95 a9425bf5 17ffffb8 a9025bf5 (d4210000)
      
      With this patch applied, an undefined instruction is reported as:
      
      | Internal error: Oops - Undefined instruction: 0 [#1] PREEMPT SMP
      | Modules linked in:
      | CPU: 0 PID: 0 Comm: swapper Not tainted 5.19.0-rc3-00128-gf27cfcc80e52-dirty #5
      | Hardware name: linux,dummy-virt (DT)
      | pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
      | pc : setup_arch+0x550/0x598
      | lr : setup_arch+0x50c/0x598
      | sp : ffff800009f63d90
      | x29: ffff800009f63d90 x28: 0000000081000200 x27: ffff800009644a70
      | x26: ffff8000096778c8 x25: 0000000000000040 x24: 0000000000000000
      | x23: 0000000000000100 x22: ffff800009f69a58 x21: ffff80000a2b80b8
      | x20: 0000000000000000 x19: 0000000000000000 x18: 0000000000000006
      | x17: 0000000000403000 x16: 00000000bfbfd000 x15: ffff800009f63830
      | x14: ffffffffffffffff x13: 0000000000000000 x12: 0000000000000019
      | x11: 0101010101010101 x10: 0000000000161b98 x9 : 0000000000000000
      | x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
      | x5 : 0000000000000008 x4 : 0000000000000010 x3 : 0000000000000000
      | x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
      | Call trace:
      |  setup_arch+0x550/0x598
      |  start_kernel+0x88/0x6ac
      |  __primary_switched+0xb8/0xc0
      | Code: b4000080 90ffed80 912ac000 97db745f (00000000)
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Cc: Alexandru Elisei <alexandru.elisei@arm.com>
      Cc: Amit Daniel Kachhap <amit.kachhap@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will@kernel.org>
      Reviewed-by: default avatarAnshuman Khandual <anshuman.khandual@arm.com>
      Link: https://lore.kernel.org/r/20220913101732.3925290-2-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      b502c87d
  6. 09 Sep, 2022 7 commits
    • Mark Rutland's avatar
      arm64: atomic: always inline the assembly · 78f6f5c9
      Mark Rutland authored
      The __lse_*() and __ll_sc_*() atomic implementations are marked as
      inline rather than __always_inline, permitting a compiler to generate
      out-of-line versions, which may be instrumented.
      
      We marked the atomic wrappers as __always_inline in commit:
      
        c35a824c ("arm64: make atomic helpers __always_inline")
      
      ... but did not think to do the same for the underlying implementations.
      
      If the compiler were to out-of-line an LSE or LL/SC atomic, this could
      break noinstr code. Ensure this doesn't happen by marking the underlying
      implementations as __always_inline.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220817155914.3975112-3-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      78f6f5c9
    • Mark Rutland's avatar
      arm64: atomics: remove LL/SC trampolines · b2c3ccbd
      Mark Rutland authored
      When CONFIG_ARM64_LSE_ATOMICS=y, each use of an LL/SC atomic results in
      a fragment of code being generated in a subsection without a clear
      association with its caller. A trampoline in the caller branches to the
      LL/SC atomic with with a direct branch, and the atomic directly branches
      back into its trampoline.
      
      This breaks backtracing, as any PC within the out-of-line fragment will
      be symbolized as an offset from the nearest prior symbol (which may not
      be the function using the atomic), and since the atomic returns with a
      direct branch, the caller's PC may be missing from the backtrace.
      
      For example, with secondary_start_kernel() hacked to contain
      atomic_inc(NULL), the resulting exception can be reported as being taken
      from cpus_are_stuck_in_kernel():
      
      | Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
      | Mem abort info:
      |   ESR = 0x0000000096000004
      |   EC = 0x25: DABT (current EL), IL = 32 bits
      |   SET = 0, FnV = 0
      |   EA = 0, S1PTW = 0
      |   FSC = 0x04: level 0 translation fault
      | Data abort info:
      |   ISV = 0, ISS = 0x00000004
      |   CM = 0, WnR = 0
      | [0000000000000000] user address but active_mm is swapper
      | Internal error: Oops: 96000004 [#1] PREEMPT SMP
      | Modules linked in:
      | CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.19.0-11219-geb555cb5-dirty #3
      | Hardware name: linux,dummy-virt (DT)
      | pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
      | pc : cpus_are_stuck_in_kernel+0xa4/0x120
      | lr : secondary_start_kernel+0x164/0x170
      | sp : ffff80000a4cbe90
      | x29: ffff80000a4cbe90 x28: 0000000000000000 x27: 0000000000000000
      | x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
      | x23: 0000000000000000 x22: 0000000000000000 x21: 0000000000000000
      | x20: 0000000000000001 x19: 0000000000000001 x18: 0000000000000008
      | x17: 3030383832343030 x16: 3030303030307830 x15: ffff80000a4cbab0
      | x14: 0000000000000001 x13: 5d31666130663133 x12: 3478305b20313030
      | x11: 3030303030303078 x10: 3020726f73736563 x9 : 726f737365636f72
      | x8 : ffff800009ff2ef0 x7 : 0000000000000003 x6 : 0000000000000000
      | x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000100
      | x2 : 0000000000000000 x1 : ffff0000029bd880 x0 : 0000000000000000
      | Call trace:
      |  cpus_are_stuck_in_kernel+0xa4/0x120
      |  __secondary_switched+0xb0/0xb4
      | Code: 35ffffa3 17fffc6c d53cd040 f9800011 (885f7c01)
      | ---[ end trace 0000000000000000 ]---
      
      This is confusing and hinders debugging, and will be problematic for
      CONFIG_LIVEPATCH as these cases cannot be unwound reliably.
      
      This is very similar to recent issues with out-of-line exception fixups,
      which were removed in commits:
      
        35d67794 ("arm64: lib: __arch_clear_user(): fold fixups into body")
        4012e0e2 ("arm64: lib: __arch_copy_from_user(): fold fixups into body")
        139f9ab7 ("arm64: lib: __arch_copy_to_user(): fold fixups into body")
      
      When the trampolines were introduced in commit:
      
        addfc386 ("arm64: atomics: avoid out-of-line ll/sc atomics")
      
      The rationale was to improve icache performance by grouping the LL/SC
      atomics together. This has never been measured, and this theoretical
      benefit is outweighed by other factors:
      
      * As the subsections are collapsed into sections at object file
        granularity, these are spread out throughout the kernel and can share
        cachelines with unrelated code regardless.
      
      * GCC 12.1.0 has been observed to place the trampoline out-of-line in
        specialised __ll_sc_*() functions, introducing more branching than was
        intended.
      
      * Removing the trampolines has been observed to shrink a defconfig
        kernel Image by 64KiB when building with GCC 12.1.0.
      
      This patch removes the LL/SC trampolines, meaning that the LL/SC atomics
      will be inlined into their callers (or placed in out-of line functions
      using regular BL/RET pairs). When CONFIG_ARM64_LSE_ATOMICS=y, the LL/SC
      atomics are always called in an unlikely branch, and will be placed in a
      cold portion of the function, so this should have minimal impact to the
      hot paths.
      
      Other than the improved backtracing, there should be no functional
      change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220817155914.3975112-2-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      b2c3ccbd
    • Mark Rutland's avatar
      arm64: stacktrace: track hyp stacks in unwinder's address space · 4b5e694e
      Mark Rutland authored
      Currently unwind_next_frame_record() has an optional callback to convert
      the address space of the FP. This is necessary for the NVHE unwinder,
      which tracks the stacks in the hyp VA space, but accesses the frame
      records in the kernel VA space.
      
      This is a bit unfortunate since it clutters unwind_next_frame_record(),
      which will get in the way of future rework.
      
      Instead, this patch changes the NVHE unwinder to track the stacks in the
      kernel's VA space and translate to FP prior to calling
      unwind_next_frame_record(). This removes the need for the translate_fp()
      callback, as all unwinders consistently track stacks in the native
      address space of the unwinder.
      
      At the same time, this patch consolidates the generation of the stack
      addresses behind the stackinfo_get_*() helpers.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Reviewed-by: default avatarKalesh Singh <kaleshsingh@google.com>
      Reviewed-by: default avatarMadhavan T. Venkataraman <madvenka@linux.microsoft.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Cc: Fuad Tabba <tabba@google.com>
      Cc: Marc Zyngier <maz@kernel.org>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220901130646.1316937-10-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      4b5e694e
    • Mark Rutland's avatar
      arm64: stacktrace: track all stack boundaries explicitly · 8df13730
      Mark Rutland authored
      Currently we call an on_accessible_stack() callback for each step of the
      unwinder, requiring redundant work to be performed in the core of the
      unwind loop (e.g. disabling preemption around accesses to per-cpu
      variables containing stack boundaries). To prevent unwind loops which go
      through a stack multiple times, we have to track the set of unwound
      stacks, requiring a stack_type enum which needs to cater for all the
      stacks of all possible callees. To prevent loops within a stack, we must
      track the prior FP values.
      
      This patch reworks the unwinder to minimize the work in the core of the
      unwinder, and to remove the need for the stack_type enum. The set of
      accessible stacks (and their boundaries) are determined at the start of
      the unwind, and the current stack is tracked during the unwind, with
      completed stacks removed from the set of accessible stacks. This makes
      the boundary checks more accurate (e.g. detecting overlapped frame
      records), and removes the need for separate tracking of the prior FP and
      visited stacks.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Reviewed-by: default avatarKalesh Singh <kaleshsingh@google.com>
      Reviewed-by: default avatarMadhavan T. Venkataraman <madvenka@linux.microsoft.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Cc: Fuad Tabba <tabba@google.com>
      Cc: Marc Zyngier <maz@kernel.org>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220901130646.1316937-9-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      8df13730
    • Mark Rutland's avatar
      arm64: stacktrace: remove stack type from fp translator · bd8abd68
      Mark Rutland authored
      In subsequent patches we'll remove the stack_type enum, and move the FP
      translation logic out of the raw FP unwind code.
      
      In preparation for doing so, this patch removes the type parameter from
      the FP translation callback, and modifies kvm_nvhe_stack_kern_va() to
      determine the relevant stack directly.
      
      So that kvm_nvhe_stack_kern_va() can use the stackinfo_*() helpers,
      these are moved earlier in the file, but are not modified in any way.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Reviewed-by: default avatarKalesh Singh <kaleshsingh@google.com>
      Reviewed-by: default avatarMadhavan T. Venkataraman <madvenka@linux.microsoft.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Cc: Fuad Tabba <tabba@google.com>
      Cc: Marc Zyngier <maz@kernel.org>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220901130646.1316937-8-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      bd8abd68
    • Mark Rutland's avatar
      arm64: stacktrace: rework stack boundary discovery · d1f684e4
      Mark Rutland authored
      In subsequent patches we'll want to acquire the stack boundaries
      ahead-of-time, and we'll need to be able to acquire the relevant
      stack_info regardless of whether we have an object the happens to be on
      the stack.
      
      This patch replaces the on_XXX_stack() helpers with stackinfo_get_XXX()
      helpers, with the caller being responsible for the checking whether an
      object is on a relevant stack. For the moment this is moved into the
      on_accessible_stack() functions, making these slightly larger;
      subsequent patches will remove the on_accessible_stack() functions and
      simplify the logic.
      
      The on_irq_stack() and on_task_stack() helpers are kept as these are
      used by IRQ entry sequences and stackleak respectively. As they're only
      used as predicates, the stack_info pointer parameter is removed in both
      cases.
      
      As the on_accessible_stack() functions are always passed a non-NULL info
      pointer, these now update info unconditionally. When updating the type
      to STACK_TYPE_UNKNOWN, the low/high bounds are also modified, but as
      these will not be consumed this should have no adverse affect.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Reviewed-by: default avatarKalesh Singh <kaleshsingh@google.com>
      Reviewed-by: default avatarMadhavan T. Venkataraman <madvenka@linux.microsoft.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Cc: Fuad Tabba <tabba@google.com>
      Cc: Marc Zyngier <maz@kernel.org>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220901130646.1316937-7-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      d1f684e4
    • Mark Rutland's avatar
      arm64: stacktrace: add stackinfo_on_stack() helper · 36f9a879
      Mark Rutland authored
      Factor the core predicate out of on_stack() into a helper which can be
      used on a pre-populated stack_info.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Reviewed-by: default avatarKalesh Singh <kaleshsingh@google.com>
      Reviewed-by: default avatarMadhavan T. Venkataraman <madvenka@linux.microsoft.com>
      Reviewed-by: default avatarMark Brown <broonie@kernel.org>
      Cc: Fuad Tabba <tabba@google.com>
      Cc: Marc Zyngier <maz@kernel.org>
      Cc: Will Deacon <will@kernel.org>
      Link: https://lore.kernel.org/r/20220901130646.1316937-6-mark.rutland@arm.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      36f9a879