Commit d5e45e81 authored by Palmer Dabbelt's avatar Palmer Dabbelt

Merge patch series "riscv: Add vector ISA support"

Andy Chiu <andy.chiu@sifive.com> says:

This is the v21 patch series for adding Vector extension support in
Linux. Please refer to [1] for the introduction of the patchset. The
v21 patch series was aimed to solve build issues from v19, provide usage
guideline for the prctl interface, and address review comments on v20.

Thank every one who has been reviewing, suggesting on the topic. Hope
this get a step closer to the final merge.

* b4-shazam-merge: (27 commits)
  selftests: add .gitignore file for RISC-V hwprobe
  selftests: Test RISC-V Vector prctl interface
  riscv: Add documentation for Vector
  riscv: Enable Vector code to be built
  riscv: detect assembler support for .option arch
  riscv: Add sysctl to set the default vector rule for new processes
  riscv: Add prctl controls for userspace vector management
  riscv: hwcap: change ELF_HWCAP to a function
  riscv: KVM: Add vector lazy save/restore support
  riscv: kvm: Add V extension to KVM ISA
  riscv: prevent stack corruption by reserving task_pt_regs(p) early
  riscv: signal: validate altstack to reflect Vector
  riscv: signal: Report signal frame size to userspace via auxv
  riscv: signal: Add sigcontext save/restore for vector
  riscv: signal: check fp-reserved words unconditionally
  riscv: Add ptrace vector support
  riscv: Allocate user's vector context in the first-use trap
  riscv: Add task switch support for vector
  riscv: Introduce struct/helpers to save/restore per-task Vector state
  riscv: Introduce riscv_v_vsize to record size of Vector context
  ...

Link: https://lore.kernel.org/r/20230605110724.21391-1-andy.chiu@sifive.comSigned-off-by: default avatarPalmer Dabbelt <palmer@rivosinc.com>
parents 748462b5 1e726951
......@@ -64,6 +64,9 @@ The following keys are defined:
* :c:macro:`RISCV_HWPROBE_IMA_C`: The C extension is supported, as defined
by version 2.2 of the RISC-V ISA manual.
* :c:macro:`RISCV_HWPROBE_IMA_V`: The V extension is supported, as defined by
version 1.0 of the RISC-V Vector extension manual.
* :c:macro:`RISCV_HWPROBE_KEY_CPUPERF_0`: A bitmask that contains performance
information about the selected set of processors.
......
......@@ -10,6 +10,7 @@ RISC-V architecture
hwprobe
patch-acceptance
uabi
vector
features
......
.. SPDX-License-Identifier: GPL-2.0
=========================================
Vector Extension Support for RISC-V Linux
=========================================
This document briefly outlines the interface provided to userspace by Linux in
order to support the use of the RISC-V Vector Extension.
1. prctl() Interface
---------------------
Two new prctl() calls are added to allow programs to manage the enablement
status for the use of Vector in userspace. The intended usage guideline for
these interfaces is to give init systems a way to modify the availability of V
for processes running under its domain. Calling thess interfaces is not
recommended in libraries routines because libraries should not override policies
configured from the parant process. Also, users must noted that these interfaces
are not portable to non-Linux, nor non-RISC-V environments, so it is discourage
to use in a portable code. To get the availability of V in an ELF program,
please read :c:macro:`COMPAT_HWCAP_ISA_V` bit of :c:macro:`ELF_HWCAP` in the
auxiliary vector.
* prctl(PR_RISCV_V_SET_CONTROL, unsigned long arg)
Sets the Vector enablement status of the calling thread, where the control
argument consists of two 2-bit enablement statuses and a bit for inheritance
mode. Other threads of the calling process are unaffected.
Enablement status is a tri-state value each occupying 2-bit of space in
the control argument:
* :c:macro:`PR_RISCV_V_VSTATE_CTRL_DEFAULT`: Use the system-wide default
enablement status on execve(). The system-wide default setting can be
controlled via sysctl interface (see sysctl section below).
* :c:macro:`PR_RISCV_V_VSTATE_CTRL_ON`: Allow Vector to be run for the
thread.
* :c:macro:`PR_RISCV_V_VSTATE_CTRL_OFF`: Disallow Vector. Executing Vector
instructions under such condition will trap and casuse the termination of the thread.
arg: The control argument is a 5-bit value consisting of 3 parts, and
accessed by 3 masks respectively.
The 3 masks, PR_RISCV_V_VSTATE_CTRL_CUR_MASK,
PR_RISCV_V_VSTATE_CTRL_NEXT_MASK, and PR_RISCV_V_VSTATE_CTRL_INHERIT
represents bit[1:0], bit[3:2], and bit[4]. bit[1:0] accounts for the
enablement status of current thread, and the setting at bit[3:2] takes place
at next execve(). bit[4] defines the inheritance mode of the setting in
bit[3:2].
* :c:macro:`PR_RISCV_V_VSTATE_CTRL_CUR_MASK`: bit[1:0]: Account for the
Vector enablement status for the calling thread. The calling thread is
not able to turn off Vector once it has been enabled. The prctl() call
fails with EPERM if the value in this mask is PR_RISCV_V_VSTATE_CTRL_OFF
but the current enablement status is not off. Setting
PR_RISCV_V_VSTATE_CTRL_DEFAULT here takes no effect but to set back
the original enablement status.
* :c:macro:`PR_RISCV_V_VSTATE_CTRL_NEXT_MASK`: bit[3:2]: Account for the
Vector enablement setting for the calling thread at the next execve()
system call. If PR_RISCV_V_VSTATE_CTRL_DEFAULT is used in this mask,
then the enablement status will be decided by the system-wide
enablement status when execve() happen.
* :c:macro:`PR_RISCV_V_VSTATE_CTRL_INHERIT`: bit[4]: the inheritance
mode for the setting at PR_RISCV_V_VSTATE_CTRL_NEXT_MASK. If the bit
is set then the following execve() will not clear the setting in both
PR_RISCV_V_VSTATE_CTRL_NEXT_MASK and PR_RISCV_V_VSTATE_CTRL_INHERIT.
This setting persists across changes in the system-wide default value.
Return value:
* 0 on success;
* EINVAL: Vector not supported, invalid enablement status for current or
next mask;
* EPERM: Turning off Vector in PR_RISCV_V_VSTATE_CTRL_CUR_MASK if Vector
was enabled for the calling thread.
On success:
* A valid setting for PR_RISCV_V_VSTATE_CTRL_CUR_MASK takes place
immediately. The enablement status specified in
PR_RISCV_V_VSTATE_CTRL_NEXT_MASK happens at the next execve() call, or
all following execve() calls if PR_RISCV_V_VSTATE_CTRL_INHERIT bit is
set.
* Every successful call overwrites a previous setting for the calling
thread.
* prctl(PR_RISCV_V_GET_CONTROL)
Gets the same Vector enablement status for the calling thread. Setting for
next execve() call and the inheritance bit are all OR-ed together.
Note that ELF programs are able to get the availability of V for itself by
reading :c:macro:`COMPAT_HWCAP_ISA_V` bit of :c:macro:`ELF_HWCAP` in the
auxiliary vector.
Return value:
* a nonnegative value on success;
* EINVAL: Vector not supported.
2. System runtime configuration (sysctl)
-----------------------------------------
To mitigate the ABI impact of expansion of the signal stack, a
policy mechanism is provided to the administrators, distro maintainers, and
developers to control the default Vector enablement status for userspace
processes in form of sysctl knob:
* /proc/sys/abi/riscv_v_default_allow
Writing the text representation of 0 or 1 to this file sets the default
system enablement status for new starting userspace programs. Valid values
are:
* 0: Do not allow Vector code to be executed as the default for new processes.
* 1: Allow Vector code to be executed as the default for new processes.
Reading this file returns the current system default enablement status.
At every execve() call, a new enablement status of the new process is set to
the system default, unless:
* PR_RISCV_V_VSTATE_CTRL_INHERIT is set for the calling process, and the
setting in PR_RISCV_V_VSTATE_CTRL_NEXT_MASK is not
PR_RISCV_V_VSTATE_CTRL_DEFAULT. Or,
* The setting in PR_RISCV_V_VSTATE_CTRL_NEXT_MASK is not
PR_RISCV_V_VSTATE_CTRL_DEFAULT.
Modifying the system default enablement status does not affect the enablement
status of any existing process of thread that do not make an execve() call.
......@@ -264,6 +264,12 @@ config RISCV_DMA_NONCOHERENT
config AS_HAS_INSN
def_bool $(as-instr,.insn r 51$(comma) 0$(comma) 0$(comma) t0$(comma) t0$(comma) zero)
config AS_HAS_OPTION_ARCH
# https://reviews.llvm.org/D123515
def_bool y
depends on $(as-instr, .option arch$(comma) +m)
depends on !$(as-instr, .option arch$(comma) -i)
source "arch/riscv/Kconfig.socs"
source "arch/riscv/Kconfig.errata"
......@@ -462,13 +468,44 @@ config RISCV_ISA_SVPBMT
If you don't know what to do here, say Y.
config TOOLCHAIN_HAS_V
bool
default y
depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64iv)
depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32iv)
depends on LLD_VERSION >= 140000 || LD_VERSION >= 23800
depends on AS_HAS_OPTION_ARCH
config RISCV_ISA_V
bool "VECTOR extension support"
depends on TOOLCHAIN_HAS_V
depends on FPU
select DYNAMIC_SIGFRAME
default y
help
Say N here if you want to disable all vector related procedure
in the kernel.
If you don't know what to do here, say Y.
config RISCV_ISA_V_DEFAULT_ENABLE
bool "Enable userspace Vector by default"
depends on RISCV_ISA_V
default y
help
Say Y here if you want to enable Vector in userspace by default.
Otherwise, userspace has to make explicit prctl() call to enable
Vector, or enable it via the sysctl interface.
If you don't know what to do here, say Y.
config TOOLCHAIN_HAS_ZBB
bool
default y
depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zbb)
depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zbb)
depends on LLD_VERSION >= 150000 || LD_VERSION >= 23900
depends on AS_IS_GNU
depends on AS_HAS_OPTION_ARCH
config RISCV_ISA_ZBB
bool "Zbb extension support for bit manipulation instructions"
......
......@@ -60,6 +60,7 @@ riscv-march-$(CONFIG_ARCH_RV32I) := rv32ima
riscv-march-$(CONFIG_ARCH_RV64I) := rv64ima
riscv-march-$(CONFIG_FPU) := $(riscv-march-y)fd
riscv-march-$(CONFIG_RISCV_ISA_C) := $(riscv-march-y)c
riscv-march-$(CONFIG_RISCV_ISA_V) := $(riscv-march-y)v
ifdef CONFIG_TOOLCHAIN_NEEDS_OLD_ISA_SPEC
KBUILD_CFLAGS += -Wa,-misa-spec=2.2
......@@ -71,7 +72,10 @@ endif
# Check if the toolchain supports Zihintpause extension
riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) := $(riscv-march-y)_zihintpause
KBUILD_CFLAGS += -march=$(subst fd,,$(riscv-march-y))
# Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by
# matching non-v and non-multi-letter extensions out with the filter ([^v_]*)
KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/')
KBUILD_AFLAGS += -march=$(riscv-march-y)
KBUILD_CFLAGS += -mno-save-restore
......
......@@ -24,16 +24,24 @@
#define SR_FS_CLEAN _AC(0x00004000, UL)
#define SR_FS_DIRTY _AC(0x00006000, UL)
#define SR_VS _AC(0x00000600, UL) /* Vector Status */
#define SR_VS_OFF _AC(0x00000000, UL)
#define SR_VS_INITIAL _AC(0x00000200, UL)
#define SR_VS_CLEAN _AC(0x00000400, UL)
#define SR_VS_DIRTY _AC(0x00000600, UL)
#define SR_XS _AC(0x00018000, UL) /* Extension Status */
#define SR_XS_OFF _AC(0x00000000, UL)
#define SR_XS_INITIAL _AC(0x00008000, UL)
#define SR_XS_CLEAN _AC(0x00010000, UL)
#define SR_XS_DIRTY _AC(0x00018000, UL)
#define SR_FS_VS (SR_FS | SR_VS) /* Vector and Floating-Point Unit */
#ifndef CONFIG_64BIT
#define SR_SD _AC(0x80000000, UL) /* FS/XS dirty */
#define SR_SD _AC(0x80000000, UL) /* FS/VS/XS dirty */
#else
#define SR_SD _AC(0x8000000000000000, UL) /* FS/XS dirty */
#define SR_SD _AC(0x8000000000000000, UL) /* FS/VS/XS dirty */
#endif
#ifdef CONFIG_64BIT
......@@ -375,6 +383,12 @@
#define CSR_MVIPH 0x319
#define CSR_MIPH 0x354
#define CSR_VSTART 0x8
#define CSR_VCSR 0xf
#define CSR_VL 0xc20
#define CSR_VTYPE 0xc21
#define CSR_VLENB 0xc22
#ifdef CONFIG_RISCV_M_MODE
# define CSR_STATUS CSR_MSTATUS
# define CSR_IE CSR_MIE
......
......@@ -66,7 +66,7 @@ extern bool compat_elf_check_arch(Elf32_Ehdr *hdr);
* via a bitmap that coorespends to each single-letter ISA extension. This is
* essentially defunct, but will remain for compatibility with userspace.
*/
#define ELF_HWCAP (elf_hwcap & ((1UL << RISCV_ISA_EXT_BASE) - 1))
#define ELF_HWCAP riscv_get_elf_hwcap()
extern unsigned long elf_hwcap;
/*
......@@ -105,6 +105,15 @@ do { \
get_cache_size(3, CACHE_TYPE_UNIFIED)); \
NEW_AUX_ENT(AT_L3_CACHEGEOMETRY, \
get_cache_geometry(3, CACHE_TYPE_UNIFIED)); \
/* \
* Should always be nonzero unless there's a kernel bug. \
* If we haven't determined a sensible value to give to \
* userspace, omit the entry: \
*/ \
if (likely(signal_minsigstksz)) \
NEW_AUX_ENT(AT_MINSIGSTKSZ, signal_minsigstksz); \
else \
NEW_AUX_ENT(AT_IGNORE, 0); \
} while (0)
#define ARCH_HAS_SETUP_ADDITIONAL_PAGES
struct linux_binprm;
......
......@@ -22,6 +22,7 @@
#define RISCV_ISA_EXT_m ('m' - 'a')
#define RISCV_ISA_EXT_s ('s' - 'a')
#define RISCV_ISA_EXT_u ('u' - 'a')
#define RISCV_ISA_EXT_v ('v' - 'a')
/*
* These macros represent the logical IDs of each multi-letter RISC-V ISA
......@@ -60,6 +61,8 @@
#include <linux/jump_label.h>
unsigned long riscv_get_elf_hwcap(void);
struct riscv_isa_ext_data {
/* Name of the extension displayed to userspace via /proc/cpuinfo */
char uprop[RISCV_ISA_EXT_NAME_LEN_MAX];
......
......@@ -137,6 +137,26 @@
#define RVG_OPCODE_JALR 0x67
#define RVG_OPCODE_JAL 0x6f
#define RVG_OPCODE_SYSTEM 0x73
#define RVG_SYSTEM_CSR_OFF 20
#define RVG_SYSTEM_CSR_MASK GENMASK(12, 0)
/* parts of opcode for RVF, RVD and RVQ */
#define RVFDQ_FL_FS_WIDTH_OFF 12
#define RVFDQ_FL_FS_WIDTH_MASK GENMASK(3, 0)
#define RVFDQ_FL_FS_WIDTH_W 2
#define RVFDQ_FL_FS_WIDTH_D 3
#define RVFDQ_LS_FS_WIDTH_Q 4
#define RVFDQ_OPCODE_FL 0x07
#define RVFDQ_OPCODE_FS 0x27
/* parts of opcode for RVV */
#define RVV_OPCODE_VECTOR 0x57
#define RVV_VL_VS_WIDTH_8 0
#define RVV_VL_VS_WIDTH_16 5
#define RVV_VL_VS_WIDTH_32 6
#define RVV_VL_VS_WIDTH_64 7
#define RVV_OPCODE_VL RVFDQ_OPCODE_FL
#define RVV_OPCODE_VS RVFDQ_OPCODE_FS
/* parts of opcode for RVC*/
#define RVC_OPCODE_C0 0x0
......@@ -304,6 +324,15 @@ static __always_inline bool riscv_insn_is_branch(u32 code)
(RVC_X(x_, RVC_B_IMM_7_6_OPOFF, RVC_B_IMM_7_6_MASK) << RVC_B_IMM_7_6_OFF) | \
(RVC_IMM_SIGN(x_) << RVC_B_IMM_SIGN_OFF); })
#define RVG_EXTRACT_SYSTEM_CSR(x) \
({typeof(x) x_ = (x); RV_X(x_, RVG_SYSTEM_CSR_OFF, RVG_SYSTEM_CSR_MASK); })
#define RVFDQ_EXTRACT_FL_FS_WIDTH(x) \
({typeof(x) x_ = (x); RV_X(x_, RVFDQ_FL_FS_WIDTH_OFF, \
RVFDQ_FL_FS_WIDTH_MASK); })
#define RVV_EXRACT_VL_VS_WIDTH(x) RVFDQ_EXTRACT_FL_FS_WIDTH(x)
/*
* Get the immediate from a J-type instruction.
*
......
......@@ -15,6 +15,7 @@
#include <linux/spinlock.h>
#include <asm/hwcap.h>
#include <asm/kvm_aia.h>
#include <asm/ptrace.h>
#include <asm/kvm_vcpu_fp.h>
#include <asm/kvm_vcpu_insn.h>
#include <asm/kvm_vcpu_sbi.h>
......@@ -145,6 +146,7 @@ struct kvm_cpu_context {
unsigned long sstatus;
unsigned long hstatus;
union __riscv_fp_state fp;
struct __riscv_v_ext_state vector;
};
struct kvm_vcpu_csr {
......
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright (C) 2022 SiFive
*
* Authors:
* Vincent Chen <vincent.chen@sifive.com>
* Greentime Hu <greentime.hu@sifive.com>
*/
#ifndef __KVM_VCPU_RISCV_VECTOR_H
#define __KVM_VCPU_RISCV_VECTOR_H
#include <linux/types.h>
#ifdef CONFIG_RISCV_ISA_V
#include <asm/vector.h>
#include <asm/kvm_host.h>
static __always_inline void __kvm_riscv_vector_save(struct kvm_cpu_context *context)
{
__riscv_v_vstate_save(&context->vector, context->vector.datap);
}
static __always_inline void __kvm_riscv_vector_restore(struct kvm_cpu_context *context)
{
__riscv_v_vstate_restore(&context->vector, context->vector.datap);
}
void kvm_riscv_vcpu_vector_reset(struct kvm_vcpu *vcpu);
void kvm_riscv_vcpu_guest_vector_save(struct kvm_cpu_context *cntx,
unsigned long *isa);
void kvm_riscv_vcpu_guest_vector_restore(struct kvm_cpu_context *cntx,
unsigned long *isa);
void kvm_riscv_vcpu_host_vector_save(struct kvm_cpu_context *cntx);
void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx);
int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
struct kvm_cpu_context *cntx);
void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu);
#else
struct kvm_cpu_context;
static inline void kvm_riscv_vcpu_vector_reset(struct kvm_vcpu *vcpu)
{
}
static inline void kvm_riscv_vcpu_guest_vector_save(struct kvm_cpu_context *cntx,
unsigned long *isa)
{
}
static inline void kvm_riscv_vcpu_guest_vector_restore(struct kvm_cpu_context *cntx,
unsigned long *isa)
{
}
static inline void kvm_riscv_vcpu_host_vector_save(struct kvm_cpu_context *cntx)
{
}
static inline void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx)
{
}
static inline int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
struct kvm_cpu_context *cntx)
{
return 0;
}
static inline void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu)
{
}
#endif
int kvm_riscv_vcpu_get_reg_vector(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg,
unsigned long rtype);
int kvm_riscv_vcpu_set_reg_vector(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg,
unsigned long rtype);
#endif
......@@ -7,6 +7,7 @@
#define _ASM_RISCV_PROCESSOR_H
#include <linux/const.h>
#include <linux/cache.h>
#include <vdso/processor.h>
......@@ -39,6 +40,8 @@ struct thread_struct {
unsigned long s[12]; /* s[0]: frame pointer */
struct __riscv_d_ext_state fstate;
unsigned long bad_cause;
unsigned long vstate_ctrl;
struct __riscv_v_ext_state vstate;
};
/* Whitelist the fstate from the task_struct for hardened usercopy */
......@@ -80,6 +83,16 @@ int riscv_of_parent_hartid(struct device_node *node, unsigned long *hartid);
extern void riscv_fill_hwcap(void);
extern int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
extern unsigned long signal_minsigstksz __ro_after_init;
#ifdef CONFIG_RISCV_ISA_V
/* Userspace interface for PR_RISCV_V_{SET,GET}_VS prctl()s: */
#define RISCV_V_SET_CONTROL(arg) riscv_v_vstate_ctrl_set_current(arg)
#define RISCV_V_GET_CONTROL() riscv_v_vstate_ctrl_get_current()
extern long riscv_v_vstate_ctrl_set_current(unsigned long arg);
extern long riscv_v_vstate_ctrl_get_current(void);
#endif /* CONFIG_RISCV_ISA_V */
#endif /* __ASSEMBLY__ */
#endif /* _ASM_RISCV_PROCESSOR_H */
......@@ -8,6 +8,7 @@
#include <linux/jump_label.h>
#include <linux/sched/task_stack.h>
#include <asm/vector.h>
#include <asm/hwcap.h>
#include <asm/processor.h>
#include <asm/ptrace.h>
......@@ -46,7 +47,7 @@ static inline void fstate_restore(struct task_struct *task,
}
}
static inline void __switch_to_aux(struct task_struct *prev,
static inline void __switch_to_fpu(struct task_struct *prev,
struct task_struct *next)
{
struct pt_regs *regs;
......@@ -66,7 +67,7 @@ static __always_inline bool has_fpu(void)
static __always_inline bool has_fpu(void) { return false; }
#define fstate_save(task, regs) do { } while (0)
#define fstate_restore(task, regs) do { } while (0)
#define __switch_to_aux(__prev, __next) do { } while (0)
#define __switch_to_fpu(__prev, __next) do { } while (0)
#endif
extern struct task_struct *__switch_to(struct task_struct *,
......@@ -77,7 +78,9 @@ do { \
struct task_struct *__prev = (prev); \
struct task_struct *__next = (next); \
if (has_fpu()) \
__switch_to_aux(__prev, __next); \
__switch_to_fpu(__prev, __next); \
if (has_vector()) \
__switch_to_vector(__prev, __next); \
((last) = __switch_to(__prev, __next)); \
} while (0)
......
......@@ -81,6 +81,9 @@ struct thread_info {
.preempt_count = INIT_PREEMPT_COUNT, \
}
void arch_release_task_struct(struct task_struct *tsk);
int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
#endif /* !__ASSEMBLY__ */
/*
......
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* Copyright (C) 2020 SiFive
*/
#ifndef __ASM_RISCV_VECTOR_H
#define __ASM_RISCV_VECTOR_H
#include <linux/types.h>
#include <uapi/asm-generic/errno.h>
#ifdef CONFIG_RISCV_ISA_V
#include <linux/stringify.h>
#include <linux/sched.h>
#include <linux/sched/task_stack.h>
#include <asm/ptrace.h>
#include <asm/hwcap.h>
#include <asm/csr.h>
#include <asm/asm.h>
extern unsigned long riscv_v_vsize;
int riscv_v_setup_vsize(void);
bool riscv_v_first_use_handler(struct pt_regs *regs);
static __always_inline bool has_vector(void)
{
return riscv_has_extension_unlikely(RISCV_ISA_EXT_v);
}
static inline void __riscv_v_vstate_clean(struct pt_regs *regs)
{
regs->status = (regs->status & ~SR_VS) | SR_VS_CLEAN;
}
static inline void riscv_v_vstate_off(struct pt_regs *regs)
{
regs->status = (regs->status & ~SR_VS) | SR_VS_OFF;
}
static inline void riscv_v_vstate_on(struct pt_regs *regs)
{
regs->status = (regs->status & ~SR_VS) | SR_VS_INITIAL;
}
static inline bool riscv_v_vstate_query(struct pt_regs *regs)
{
return (regs->status & SR_VS) != 0;
}
static __always_inline void riscv_v_enable(void)
{
csr_set(CSR_SSTATUS, SR_VS);
}
static __always_inline void riscv_v_disable(void)
{
csr_clear(CSR_SSTATUS, SR_VS);
}
static __always_inline void __vstate_csr_save(struct __riscv_v_ext_state *dest)
{
asm volatile (
"csrr %0, " __stringify(CSR_VSTART) "\n\t"
"csrr %1, " __stringify(CSR_VTYPE) "\n\t"
"csrr %2, " __stringify(CSR_VL) "\n\t"
"csrr %3, " __stringify(CSR_VCSR) "\n\t"
: "=r" (dest->vstart), "=r" (dest->vtype), "=r" (dest->vl),
"=r" (dest->vcsr) : :);
}
static __always_inline void __vstate_csr_restore(struct __riscv_v_ext_state *src)
{
asm volatile (
".option push\n\t"
".option arch, +v\n\t"
"vsetvl x0, %2, %1\n\t"
".option pop\n\t"
"csrw " __stringify(CSR_VSTART) ", %0\n\t"
"csrw " __stringify(CSR_VCSR) ", %3\n\t"
: : "r" (src->vstart), "r" (src->vtype), "r" (src->vl),
"r" (src->vcsr) :);
}
static inline void __riscv_v_vstate_save(struct __riscv_v_ext_state *save_to,
void *datap)
{
unsigned long vl;
riscv_v_enable();
__vstate_csr_save(save_to);
asm volatile (
".option push\n\t"
".option arch, +v\n\t"
"vsetvli %0, x0, e8, m8, ta, ma\n\t"
"vse8.v v0, (%1)\n\t"
"add %1, %1, %0\n\t"
"vse8.v v8, (%1)\n\t"
"add %1, %1, %0\n\t"
"vse8.v v16, (%1)\n\t"
"add %1, %1, %0\n\t"
"vse8.v v24, (%1)\n\t"
".option pop\n\t"
: "=&r" (vl) : "r" (datap) : "memory");
riscv_v_disable();
}
static inline void __riscv_v_vstate_restore(struct __riscv_v_ext_state *restore_from,
void *datap)
{
unsigned long vl;
riscv_v_enable();
asm volatile (
".option push\n\t"
".option arch, +v\n\t"
"vsetvli %0, x0, e8, m8, ta, ma\n\t"
"vle8.v v0, (%1)\n\t"
"add %1, %1, %0\n\t"
"vle8.v v8, (%1)\n\t"
"add %1, %1, %0\n\t"
"vle8.v v16, (%1)\n\t"
"add %1, %1, %0\n\t"
"vle8.v v24, (%1)\n\t"
".option pop\n\t"
: "=&r" (vl) : "r" (datap) : "memory");
__vstate_csr_restore(restore_from);
riscv_v_disable();
}
static inline void riscv_v_vstate_save(struct task_struct *task,
struct pt_regs *regs)
{
if ((regs->status & SR_VS) == SR_VS_DIRTY) {
struct __riscv_v_ext_state *vstate = &task->thread.vstate;
__riscv_v_vstate_save(vstate, vstate->datap);
__riscv_v_vstate_clean(regs);
}
}
static inline void riscv_v_vstate_restore(struct task_struct *task,
struct pt_regs *regs)
{
if ((regs->status & SR_VS) != SR_VS_OFF) {
struct __riscv_v_ext_state *vstate = &task->thread.vstate;
__riscv_v_vstate_restore(vstate, vstate->datap);
__riscv_v_vstate_clean(regs);
}
}
static inline void __switch_to_vector(struct task_struct *prev,
struct task_struct *next)
{
struct pt_regs *regs;
regs = task_pt_regs(prev);
riscv_v_vstate_save(prev, regs);
riscv_v_vstate_restore(next, task_pt_regs(next));
}
void riscv_v_vstate_ctrl_init(struct task_struct *tsk);
bool riscv_v_vstate_ctrl_user_allowed(void);
#else /* ! CONFIG_RISCV_ISA_V */
struct pt_regs;
static inline int riscv_v_setup_vsize(void) { return -EOPNOTSUPP; }
static __always_inline bool has_vector(void) { return false; }
static inline bool riscv_v_first_use_handler(struct pt_regs *regs) { return false; }
static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; }
static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; }
#define riscv_v_vsize (0)
#define riscv_v_vstate_save(task, regs) do {} while (0)
#define riscv_v_vstate_restore(task, regs) do {} while (0)
#define __switch_to_vector(__prev, __next) do {} while (0)
#define riscv_v_vstate_off(regs) do {} while (0)
#define riscv_v_vstate_on(regs) do {} while (0)
#endif /* CONFIG_RISCV_ISA_V */
#endif /* ! __ASM_RISCV_VECTOR_H */
......@@ -35,5 +35,6 @@
/* entries in ARCH_DLINFO */
#define AT_VECTOR_SIZE_ARCH 9
#define AT_MINSIGSTKSZ 51
#endif /* _UAPI_ASM_RISCV_AUXVEC_H */
......@@ -21,5 +21,6 @@
#define COMPAT_HWCAP_ISA_F (1 << ('F' - 'A'))
#define COMPAT_HWCAP_ISA_D (1 << ('D' - 'A'))
#define COMPAT_HWCAP_ISA_C (1 << ('C' - 'A'))
#define COMPAT_HWCAP_ISA_V (1 << ('V' - 'A'))
#endif /* _UAPI_ASM_RISCV_HWCAP_H */
......@@ -25,6 +25,7 @@ struct riscv_hwprobe {
#define RISCV_HWPROBE_KEY_IMA_EXT_0 4
#define RISCV_HWPROBE_IMA_FD (1 << 0)
#define RISCV_HWPROBE_IMA_C (1 << 1)
#define RISCV_HWPROBE_IMA_V (1 << 2)
#define RISCV_HWPROBE_KEY_CPUPERF_0 5
#define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0)
#define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0)
......
......@@ -121,6 +121,7 @@ enum KVM_RISCV_ISA_EXT_ID {
KVM_RISCV_ISA_EXT_ZICBOZ,
KVM_RISCV_ISA_EXT_ZBB,
KVM_RISCV_ISA_EXT_SSAIA,
KVM_RISCV_ISA_EXT_V,
KVM_RISCV_ISA_EXT_MAX,
};
......@@ -203,6 +204,13 @@ enum KVM_RISCV_SBI_EXT_ID {
#define KVM_REG_RISCV_SBI_MULTI_REG_LAST \
KVM_REG_RISCV_SBI_MULTI_REG(KVM_RISCV_SBI_EXT_MAX - 1)
/* V extension registers are mapped as type 9 */
#define KVM_REG_RISCV_VECTOR (0x09 << KVM_REG_RISCV_TYPE_SHIFT)
#define KVM_REG_RISCV_VECTOR_CSR_REG(name) \
(offsetof(struct __riscv_v_ext_state, name) / sizeof(unsigned long))
#define KVM_REG_RISCV_VECTOR_REG(n) \
((n) + sizeof(struct __riscv_v_ext_state) / sizeof(unsigned long))
#endif
#endif /* __LINUX_KVM_RISCV_H */
......@@ -71,12 +71,51 @@ struct __riscv_q_ext_state {
__u32 reserved[3];
};
struct __riscv_ctx_hdr {
__u32 magic;
__u32 size;
};
struct __riscv_extra_ext_header {
__u32 __padding[129] __attribute__((aligned(16)));
/*
* Reserved for expansion of sigcontext structure. Currently zeroed
* upon signal, and must be zero upon sigreturn.
*/
__u32 reserved;
struct __riscv_ctx_hdr hdr;
};
union __riscv_fp_state {
struct __riscv_f_ext_state f;
struct __riscv_d_ext_state d;
struct __riscv_q_ext_state q;
};
struct __riscv_v_ext_state {
unsigned long vstart;
unsigned long vl;
unsigned long vtype;
unsigned long vcsr;
void *datap;
/*
* In signal handler, datap will be set a correct user stack offset
* and vector registers will be copied to the address of datap
* pointer.
*
* In ptrace syscall, datap will be set to zero and the vector
* registers will be copied to the address right after this
* structure.
*/
};
/*
* According to spec: The number of bits in a single vector register,
* VLEN >= ELEN, which must be a power of 2, and must be no greater than
* 2^16 = 65536bits = 8192bytes
*/
#define RISCV_MAX_VLENB (8192)
#endif /* __ASSEMBLY__ */
#endif /* _UAPI_ASM_RISCV_PTRACE_H */
......@@ -8,6 +8,17 @@
#include <asm/ptrace.h>
/* The Magic number for signal context frame header. */
#define RISCV_V_MAGIC 0x53465457
#define END_MAGIC 0x0
/* The size of END signal context header. */
#define END_HDR_SIZE 0x0
struct __sc_riscv_v_state {
struct __riscv_v_ext_state v_state;
} __attribute__((aligned(16)));
/*
* Signal context structure
*
......@@ -16,7 +27,10 @@
*/
struct sigcontext {
struct user_regs_struct sc_regs;
union __riscv_fp_state sc_fpregs;
union {
union __riscv_fp_state sc_fpregs;
struct __riscv_extra_ext_header sc_extdesc;
};
};
#endif /* _UAPI_ASM_RISCV_SIGCONTEXT_H */
......@@ -56,6 +56,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/
obj-$(CONFIG_RISCV_M_MODE) += traps_misaligned.o
obj-$(CONFIG_FPU) += fpu.o
obj-$(CONFIG_RISCV_ISA_V) += vector.o
obj-$(CONFIG_SMP) += smpboot.o
obj-$(CONFIG_SMP) += smp.o
obj-$(CONFIG_SMP) += cpu_ops.o
......
......@@ -21,6 +21,7 @@
#include <asm/hwcap.h>
#include <asm/patch.h>
#include <asm/processor.h>
#include <asm/vector.h>
#define NUM_ALPHA_EXTS ('z' - 'a' + 1)
......@@ -112,6 +113,7 @@ void __init riscv_fill_hwcap(void)
isa2hwcap['f' - 'a'] = COMPAT_HWCAP_ISA_F;
isa2hwcap['d' - 'a'] = COMPAT_HWCAP_ISA_D;
isa2hwcap['c' - 'a'] = COMPAT_HWCAP_ISA_C;
isa2hwcap['v' - 'a'] = COMPAT_HWCAP_ISA_V;
elf_hwcap = 0;
......@@ -292,6 +294,17 @@ void __init riscv_fill_hwcap(void)
elf_hwcap &= ~COMPAT_HWCAP_ISA_F;
}
if (elf_hwcap & COMPAT_HWCAP_ISA_V) {
riscv_v_setup_vsize();
/*
* ISA string in device tree might have 'v' flag, but
* CONFIG_RISCV_ISA_V is disabled in kernel.
* Clear V flag in elf_hwcap if CONFIG_RISCV_ISA_V is disabled.
*/
if (!IS_ENABLED(CONFIG_RISCV_ISA_V))
elf_hwcap &= ~COMPAT_HWCAP_ISA_V;
}
memset(print_str, 0, sizeof(print_str));
for (i = 0, j = 0; i < NUM_ALPHA_EXTS; i++)
if (riscv_isa[0] & BIT_MASK(i))
......@@ -305,6 +318,18 @@ void __init riscv_fill_hwcap(void)
pr_info("riscv: ELF capabilities %s\n", print_str);
}
unsigned long riscv_get_elf_hwcap(void)
{
unsigned long hwcap;
hwcap = (elf_hwcap & ((1UL << RISCV_ISA_EXT_BASE) - 1));
if (!riscv_v_vstate_ctrl_user_allowed())
hwcap &= ~COMPAT_HWCAP_ISA_V;
return hwcap;
}
#ifdef CONFIG_RISCV_ALTERNATIVE
/*
* Alternative patch sites consider 48 bits when determining when to patch
......
......@@ -48,10 +48,10 @@ _save_context:
* Disable user-mode memory access as it should only be set in the
* actual user copy routines.
*
* Disable the FPU to detect illegal usage of floating point in kernel
* space.
* Disable the FPU/Vector to detect illegal usage of floating point
* or vector in kernel space.
*/
li t0, SR_SUM | SR_FS
li t0, SR_SUM | SR_FS_VS
REG_L s0, TASK_TI_USER_SP(tp)
csrrc s1, CSR_STATUS, t0
......
......@@ -140,10 +140,10 @@ secondary_start_sbi:
.option pop
/*
* Disable FPU to detect illegal usage of
* floating point in kernel space
* Disable FPU & VECTOR to detect illegal usage of
* floating point or vector in kernel space
*/
li t0, SR_FS
li t0, SR_FS_VS
csrc CSR_STATUS, t0
/* Set trap vector to spin forever to help debug */
......@@ -234,10 +234,10 @@ pmp_done:
.option pop
/*
* Disable FPU to detect illegal usage of
* floating point in kernel space
* Disable FPU & VECTOR to detect illegal usage of
* floating point or vector in kernel space
*/
li t0, SR_FS
li t0, SR_FS_VS
csrc CSR_STATUS, t0
#ifdef CONFIG_RISCV_BOOT_SPINWAIT
......@@ -301,6 +301,7 @@ clear_bss_done:
la tp, init_task
la sp, init_thread_union + THREAD_SIZE
XIP_FIXUP_OFFSET sp
addi sp, sp, -PT_SIZE_ON_STACK
#ifdef CONFIG_BUILTIN_DTB
la a0, __dtb_start
XIP_FIXUP_OFFSET a0
......@@ -318,6 +319,7 @@ clear_bss_done:
/* Restore C environment */
la tp, init_task
la sp, init_thread_union + THREAD_SIZE
addi sp, sp, -PT_SIZE_ON_STACK
#ifdef CONFIG_KASAN
call kasan_early_init
......@@ -392,7 +394,7 @@ ENTRY(reset_regs)
#ifdef CONFIG_FPU
csrr t0, CSR_MISA
andi t0, t0, (COMPAT_HWCAP_ISA_F | COMPAT_HWCAP_ISA_D)
beqz t0, .Lreset_regs_done
beqz t0, .Lreset_regs_done_fpu
li t1, SR_FS
csrs CSR_STATUS, t1
......@@ -430,8 +432,31 @@ ENTRY(reset_regs)
fmv.s.x f31, zero
csrw fcsr, 0
/* note that the caller must clear SR_FS */
.Lreset_regs_done_fpu:
#endif /* CONFIG_FPU */
.Lreset_regs_done:
#ifdef CONFIG_RISCV_ISA_V
csrr t0, CSR_MISA
li t1, COMPAT_HWCAP_ISA_V
and t0, t0, t1
beqz t0, .Lreset_regs_done_vector
/*
* Clear vector registers and reset vcsr
* VLMAX has a defined value, VLEN is a constant,
* and this form of vsetvli is defined to set vl to VLMAX.
*/
li t1, SR_VS
csrs CSR_STATUS, t1
csrs CSR_VCSR, x0
vsetvli t1, x0, e8, m8, ta, ma
vmv.v.i v0, 0
vmv.v.i v8, 0
vmv.v.i v16, 0
vmv.v.i v24, 0
/* note that the caller must clear SR_VS */
.Lreset_regs_done_vector:
#endif /* CONFIG_RISCV_ISA_V */
ret
END(reset_regs)
#endif /* CONFIG_RISCV_M_MODE */
......@@ -24,6 +24,7 @@
#include <asm/switch_to.h>
#include <asm/thread_info.h>
#include <asm/cpuidle.h>
#include <asm/vector.h>
register unsigned long gp_in_global __asm__("gp");
......@@ -146,12 +147,29 @@ void flush_thread(void)
fstate_off(current, task_pt_regs(current));
memset(&current->thread.fstate, 0, sizeof(current->thread.fstate));
#endif
#ifdef CONFIG_RISCV_ISA_V
/* Reset vector state */
riscv_v_vstate_ctrl_init(current);
riscv_v_vstate_off(task_pt_regs(current));
kfree(current->thread.vstate.datap);
memset(&current->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
#endif
}
void arch_release_task_struct(struct task_struct *tsk)
{
/* Free the vector context of datap. */
if (has_vector())
kfree(tsk->thread.vstate.datap);
}
int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
{
fstate_save(src, task_pt_regs(src));
*dst = *src;
/* clear entire V context, including datap for a new task */
memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
return 0;
}
......@@ -176,6 +194,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
p->thread.s[1] = (unsigned long)args->fn_arg;
} else {
*childregs = *(current_pt_regs());
/* Turn off status.VS */
riscv_v_vstate_off(childregs);
if (usp) /* User fork */
childregs->sp = usp;
if (clone_flags & CLONE_SETTLS)
......
......@@ -7,6 +7,7 @@
* Copied from arch/tile/kernel/ptrace.c
*/
#include <asm/vector.h>
#include <asm/ptrace.h>
#include <asm/syscall.h>
#include <asm/thread_info.h>
......@@ -24,6 +25,9 @@ enum riscv_regset {
#ifdef CONFIG_FPU
REGSET_F,
#endif
#ifdef CONFIG_RISCV_ISA_V
REGSET_V,
#endif
};
static int riscv_gpr_get(struct task_struct *target,
......@@ -80,6 +84,61 @@ static int riscv_fpr_set(struct task_struct *target,
}
#endif
#ifdef CONFIG_RISCV_ISA_V
static int riscv_vr_get(struct task_struct *target,
const struct user_regset *regset,
struct membuf to)
{
struct __riscv_v_ext_state *vstate = &target->thread.vstate;
if (!riscv_v_vstate_query(task_pt_regs(target)))
return -EINVAL;
/*
* Ensure the vector registers have been saved to the memory before
* copying them to membuf.
*/
if (target == current)
riscv_v_vstate_save(current, task_pt_regs(current));
/* Copy vector header from vstate. */
membuf_write(&to, vstate, offsetof(struct __riscv_v_ext_state, datap));
membuf_zero(&to, sizeof(vstate->datap));
/* Copy all the vector registers from vstate. */
return membuf_write(&to, vstate->datap, riscv_v_vsize);
}
static int riscv_vr_set(struct task_struct *target,
const struct user_regset *regset,
unsigned int pos, unsigned int count,
const void *kbuf, const void __user *ubuf)
{
int ret, size;
struct __riscv_v_ext_state *vstate = &target->thread.vstate;
if (!riscv_v_vstate_query(task_pt_regs(target)))
return -EINVAL;
/* Copy rest of the vstate except datap */
ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, vstate, 0,
offsetof(struct __riscv_v_ext_state, datap));
if (unlikely(ret))
return ret;
/* Skip copy datap. */
size = sizeof(vstate->datap);
count -= size;
ubuf += size;
/* Copy all the vector registers. */
pos = 0;
ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, vstate->datap,
0, riscv_v_vsize);
return ret;
}
#endif
static const struct user_regset riscv_user_regset[] = {
[REGSET_X] = {
.core_note_type = NT_PRSTATUS,
......@@ -99,6 +158,17 @@ static const struct user_regset riscv_user_regset[] = {
.set = riscv_fpr_set,
},
#endif
#ifdef CONFIG_RISCV_ISA_V
[REGSET_V] = {
.core_note_type = NT_RISCV_VECTOR,
.align = 16,
.n = ((32 * RISCV_MAX_VLENB) +
sizeof(struct __riscv_v_ext_state)) / sizeof(__u32),
.size = sizeof(__u32),
.regset_get = riscv_vr_get,
.set = riscv_vr_set,
},
#endif
};
static const struct user_regset_view riscv_user_native_view = {
......
......@@ -264,6 +264,8 @@ static void __init parse_dtb(void)
#endif
}
extern void __init init_rt_signal_env(void);
void __init setup_arch(char **cmdline_p)
{
parse_dtb();
......@@ -304,6 +306,7 @@ void __init setup_arch(char **cmdline_p)
riscv_init_cbo_blocksizes();
riscv_fill_hwcap();
init_rt_signal_env();
apply_boot_alternatives();
if (IS_ENABLED(CONFIG_RISCV_ISA_ZICBOM) &&
riscv_isa_extension_available(NULL, ZICBOM))
......
......@@ -19,10 +19,14 @@
#include <asm/signal.h>
#include <asm/signal32.h>
#include <asm/switch_to.h>
#include <asm/vector.h>
#include <asm/csr.h>
#include <asm/cacheflush.h>
unsigned long signal_minsigstksz __ro_after_init;
extern u32 __user_rt_sigreturn[2];
static size_t riscv_v_sc_size __ro_after_init;
#define DEBUG_SIG 0
......@@ -40,26 +44,13 @@ static long restore_fp_state(struct pt_regs *regs,
{
long err;
struct __riscv_d_ext_state __user *state = &sc_fpregs->d;
size_t i;
err = __copy_from_user(&current->thread.fstate, state, sizeof(*state));
if (unlikely(err))
return err;
fstate_restore(current, regs);
/* We support no other extension state at this time. */
for (i = 0; i < ARRAY_SIZE(sc_fpregs->q.reserved); i++) {
u32 value;
err = __get_user(value, &sc_fpregs->q.reserved[i]);
if (unlikely(err))
break;
if (value != 0)
return -EINVAL;
}
return err;
return 0;
}
static long save_fp_state(struct pt_regs *regs,
......@@ -67,52 +58,186 @@ static long save_fp_state(struct pt_regs *regs,
{
long err;
struct __riscv_d_ext_state __user *state = &sc_fpregs->d;
size_t i;
fstate_save(current, regs);
err = __copy_to_user(state, &current->thread.fstate, sizeof(*state));
return err;
}
#else
#define save_fp_state(task, regs) (0)
#define restore_fp_state(task, regs) (0)
#endif
#ifdef CONFIG_RISCV_ISA_V
static long save_v_state(struct pt_regs *regs, void __user **sc_vec)
{
struct __riscv_ctx_hdr __user *hdr;
struct __sc_riscv_v_state __user *state;
void __user *datap;
long err;
hdr = *sc_vec;
/* Place state to the user's signal context space after the hdr */
state = (struct __sc_riscv_v_state __user *)(hdr + 1);
/* Point datap right after the end of __sc_riscv_v_state */
datap = state + 1;
/* datap is designed to be 16 byte aligned for better performance */
WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16)));
riscv_v_vstate_save(current, regs);
/* Copy everything of vstate but datap. */
err = __copy_to_user(&state->v_state, &current->thread.vstate,
offsetof(struct __riscv_v_ext_state, datap));
/* Copy the pointer datap itself. */
err |= __put_user(datap, &state->v_state.datap);
/* Copy the whole vector content to user space datap. */
err |= __copy_to_user(datap, current->thread.vstate.datap, riscv_v_vsize);
/* Copy magic to the user space after saving all vector conetext */
err |= __put_user(RISCV_V_MAGIC, &hdr->magic);
err |= __put_user(riscv_v_sc_size, &hdr->size);
if (unlikely(err))
return err;
/* We support no other extension state at this time. */
for (i = 0; i < ARRAY_SIZE(sc_fpregs->q.reserved); i++) {
err = __put_user(0, &sc_fpregs->q.reserved[i]);
if (unlikely(err))
break;
}
/* Only progress the sv_vec if everything has done successfully */
*sc_vec += riscv_v_sc_size;
return 0;
}
/*
* Restore Vector extension context from the user's signal frame. This function
* assumes a valid extension header. So magic and size checking must be done by
* the caller.
*/
static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec)
{
long err;
struct __sc_riscv_v_state __user *state = sc_vec;
void __user *datap;
/* Copy everything of __sc_riscv_v_state except datap. */
err = __copy_from_user(&current->thread.vstate, &state->v_state,
offsetof(struct __riscv_v_ext_state, datap));
if (unlikely(err))
return err;
/* Copy the pointer datap itself. */
err = __get_user(datap, &state->v_state.datap);
if (unlikely(err))
return err;
/*
* Copy the whole vector content from user space datap. Use
* copy_from_user to prevent information leak.
*/
err = copy_from_user(current->thread.vstate.datap, datap, riscv_v_vsize);
if (unlikely(err))
return err;
riscv_v_vstate_restore(current, regs);
return err;
}
#else
#define save_fp_state(task, regs) (0)
#define restore_fp_state(task, regs) (0)
#define save_v_state(task, regs) (0)
#define __restore_v_state(task, regs) (0)
#endif
static long restore_sigcontext(struct pt_regs *regs,
struct sigcontext __user *sc)
{
void __user *sc_ext_ptr = &sc->sc_extdesc.hdr;
__u32 rsvd;
long err;
/* sc_regs is structured the same as the start of pt_regs */
err = __copy_from_user(regs, &sc->sc_regs, sizeof(sc->sc_regs));
if (unlikely(err))
return err;
/* Restore the floating-point state. */
if (has_fpu())
err |= restore_fp_state(regs, &sc->sc_fpregs);
if (has_fpu()) {
err = restore_fp_state(regs, &sc->sc_fpregs);
if (unlikely(err))
return err;
}
/* Check the reserved word before extensions parsing */
err = __get_user(rsvd, &sc->sc_extdesc.reserved);
if (unlikely(err))
return err;
if (unlikely(rsvd))
return -EINVAL;
while (!err) {
__u32 magic, size;
struct __riscv_ctx_hdr __user *head = sc_ext_ptr;
err |= __get_user(magic, &head->magic);
err |= __get_user(size, &head->size);
if (unlikely(err))
return err;
sc_ext_ptr += sizeof(*head);
switch (magic) {
case END_MAGIC:
if (size != END_HDR_SIZE)
return -EINVAL;
return 0;
case RISCV_V_MAGIC:
if (!has_vector() || !riscv_v_vstate_query(regs) ||
size != riscv_v_sc_size)
return -EINVAL;
err = __restore_v_state(regs, sc_ext_ptr);
break;
default:
return -EINVAL;
}
sc_ext_ptr = (void __user *)head + size;
}
return err;
}
static size_t get_rt_frame_size(bool cal_all)
{
struct rt_sigframe __user *frame;
size_t frame_size;
size_t total_context_size = 0;
frame_size = sizeof(*frame);
if (has_vector()) {
if (cal_all || riscv_v_vstate_query(task_pt_regs(current)))
total_context_size += riscv_v_sc_size;
}
/*
* Preserved a __riscv_ctx_hdr for END signal context header if an
* extension uses __riscv_extra_ext_header
*/
if (total_context_size)
total_context_size += sizeof(struct __riscv_ctx_hdr);
frame_size += total_context_size;
frame_size = round_up(frame_size, 16);
return frame_size;
}
SYSCALL_DEFINE0(rt_sigreturn)
{
struct pt_regs *regs = current_pt_regs();
struct rt_sigframe __user *frame;
struct task_struct *task;
sigset_t set;
size_t frame_size = get_rt_frame_size(false);
/* Always make any pending restarted system calls return -EINTR */
current->restart_block.fn = do_no_restart_syscall;
frame = (struct rt_sigframe __user *)regs->sp;
if (!access_ok(frame, sizeof(*frame)))
if (!access_ok(frame, frame_size))
goto badframe;
if (__copy_from_user(&set, &frame->uc.uc_sigmask, sizeof(set)))
......@@ -146,12 +271,23 @@ static long setup_sigcontext(struct rt_sigframe __user *frame,
struct pt_regs *regs)
{
struct sigcontext __user *sc = &frame->uc.uc_mcontext;
struct __riscv_ctx_hdr __user *sc_ext_ptr = &sc->sc_extdesc.hdr;
long err;
/* sc_regs is structured the same as the start of pt_regs */
err = __copy_to_user(&sc->sc_regs, regs, sizeof(sc->sc_regs));
/* Save the floating-point state. */
if (has_fpu())
err |= save_fp_state(regs, &sc->sc_fpregs);
/* Save the vector state. */
if (has_vector() && riscv_v_vstate_query(regs))
err |= save_v_state(regs, (void __user **)&sc_ext_ptr);
/* Write zero to fp-reserved space and check it on restore_sigcontext */
err |= __put_user(0, &sc->sc_extdesc.reserved);
/* And put END __riscv_ctx_hdr at the end. */
err |= __put_user(END_MAGIC, &sc_ext_ptr->magic);
err |= __put_user(END_HDR_SIZE, &sc_ext_ptr->size);
return err;
}
......@@ -175,6 +311,13 @@ static inline void __user *get_sigframe(struct ksignal *ksig,
/* Align the stack frame. */
sp &= ~0xfUL;
/*
* Fail if the size of the altstack is not large enough for the
* sigframe construction.
*/
if (current->sas_ss_size && sp < current->sas_ss_sp)
return (void __user __force *)-1UL;
return (void __user *)sp;
}
......@@ -184,9 +327,10 @@ static int setup_rt_frame(struct ksignal *ksig, sigset_t *set,
struct rt_sigframe __user *frame;
long err = 0;
unsigned long __maybe_unused addr;
size_t frame_size = get_rt_frame_size(false);
frame = get_sigframe(ksig, regs, sizeof(*frame));
if (!access_ok(frame, sizeof(*frame)))
frame = get_sigframe(ksig, regs, frame_size);
if (!access_ok(frame, frame_size))
return -EFAULT;
err |= copy_siginfo_to_user(&frame->info, &ksig->info);
......@@ -319,3 +463,23 @@ void arch_do_signal_or_restart(struct pt_regs *regs)
*/
restore_saved_sigmask();
}
void init_rt_signal_env(void);
void __init init_rt_signal_env(void)
{
riscv_v_sc_size = sizeof(struct __riscv_ctx_hdr) +
sizeof(struct __sc_riscv_v_state) + riscv_v_vsize;
/*
* Determine the stack space required for guaranteed signal delivery.
* The signal_minsigstksz will be populated into the AT_MINSIGSTKSZ entry
* in the auxiliary array at process startup.
*/
signal_minsigstksz = get_rt_frame_size(true);
}
#ifdef CONFIG_DYNAMIC_SIGFRAME
bool sigaltstack_size_valid(size_t ss_size)
{
return ss_size > get_rt_frame_size(false);
}
#endif /* CONFIG_DYNAMIC_SIGFRAME */
......@@ -32,6 +32,8 @@
#include <asm/tlbflush.h>
#include <asm/sections.h>
#include <asm/smp.h>
#include <uapi/asm/hwcap.h>
#include <asm/vector.h>
#include "head.h"
......@@ -244,6 +246,11 @@ asmlinkage __visible void smp_callin(void)
set_cpu_online(curr_cpuid, 1);
probe_vendor_features(curr_cpuid);
if (has_vector()) {
if (riscv_v_setup_vsize())
elf_hwcap &= ~COMPAT_HWCAP_ISA_V;
}
/*
* Remote TLB flushes are ignored while the CPU is offline, so emit
* a local TLB flush right now just in case.
......
......@@ -10,6 +10,7 @@
#include <asm/cpufeature.h>
#include <asm/hwprobe.h>
#include <asm/sbi.h>
#include <asm/vector.h>
#include <asm/switch_to.h>
#include <asm/uaccess.h>
#include <asm/unistd.h>
......@@ -171,6 +172,9 @@ static void hwprobe_one_pair(struct riscv_hwprobe *pair,
if (riscv_isa_extension_available(NULL, c))
pair->value |= RISCV_HWPROBE_IMA_C;
if (has_vector())
pair->value |= RISCV_HWPROBE_IMA_V;
break;
case RISCV_HWPROBE_KEY_CPUPERF_0:
......
......@@ -26,6 +26,7 @@
#include <asm/ptrace.h>
#include <asm/syscall.h>
#include <asm/thread_info.h>
#include <asm/vector.h>
int show_unhandled_signals = 1;
......@@ -145,8 +146,29 @@ DO_ERROR_INFO(do_trap_insn_misaligned,
SIGBUS, BUS_ADRALN, "instruction address misaligned");
DO_ERROR_INFO(do_trap_insn_fault,
SIGSEGV, SEGV_ACCERR, "instruction access fault");
DO_ERROR_INFO(do_trap_insn_illegal,
SIGILL, ILL_ILLOPC, "illegal instruction");
asmlinkage __visible __trap_section void do_trap_insn_illegal(struct pt_regs *regs)
{
if (user_mode(regs)) {
irqentry_enter_from_user_mode(regs);
local_irq_enable();
if (!riscv_v_first_use_handler(regs))
do_trap_error(regs, SIGILL, ILL_ILLOPC, regs->epc,
"Oops - illegal instruction");
irqentry_exit_to_user_mode(regs);
} else {
irqentry_state_t state = irqentry_nmi_enter(regs);
do_trap_error(regs, SIGILL, ILL_ILLOPC, regs->epc,
"Oops - illegal instruction");
irqentry_nmi_exit(regs, state);
}
}
DO_ERROR_INFO(do_trap_load_fault,
SIGSEGV, SEGV_ACCERR, "load access fault");
#ifndef CONFIG_RISCV_M_MODE
......
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* Copyright (C) 2023 SiFive
* Author: Andy Chiu <andy.chiu@sifive.com>
*/
#include <linux/export.h>
#include <linux/sched/signal.h>
#include <linux/types.h>
#include <linux/slab.h>
#include <linux/sched.h>
#include <linux/uaccess.h>
#include <linux/prctl.h>
#include <asm/thread_info.h>
#include <asm/processor.h>
#include <asm/insn.h>
#include <asm/vector.h>
#include <asm/csr.h>
#include <asm/elf.h>
#include <asm/ptrace.h>
#include <asm/bug.h>
static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE);
unsigned long riscv_v_vsize __read_mostly;
EXPORT_SYMBOL_GPL(riscv_v_vsize);
int riscv_v_setup_vsize(void)
{
unsigned long this_vsize;
/* There are 32 vector registers with vlenb length. */
riscv_v_enable();
this_vsize = csr_read(CSR_VLENB) * 32;
riscv_v_disable();
if (!riscv_v_vsize) {
riscv_v_vsize = this_vsize;
return 0;
}
if (riscv_v_vsize != this_vsize) {
WARN(1, "RISCV_ISA_V only supports one vlenb on SMP systems");
return -EOPNOTSUPP;
}
return 0;
}
static bool insn_is_vector(u32 insn_buf)
{
u32 opcode = insn_buf & __INSN_OPCODE_MASK;
u32 width, csr;
/*
* All V-related instructions, including CSR operations are 4-Byte. So,
* do not handle if the instruction length is not 4-Byte.
*/
if (unlikely(GET_INSN_LENGTH(insn_buf) != 4))
return false;
switch (opcode) {
case RVV_OPCODE_VECTOR:
return true;
case RVV_OPCODE_VL:
case RVV_OPCODE_VS:
width = RVV_EXRACT_VL_VS_WIDTH(insn_buf);
if (width == RVV_VL_VS_WIDTH_8 || width == RVV_VL_VS_WIDTH_16 ||
width == RVV_VL_VS_WIDTH_32 || width == RVV_VL_VS_WIDTH_64)
return true;
break;
case RVG_OPCODE_SYSTEM:
csr = RVG_EXTRACT_SYSTEM_CSR(insn_buf);
if ((csr >= CSR_VSTART && csr <= CSR_VCSR) ||
(csr >= CSR_VL && csr <= CSR_VLENB))
return true;
}
return false;
}
static int riscv_v_thread_zalloc(void)
{
void *datap;
datap = kzalloc(riscv_v_vsize, GFP_KERNEL);
if (!datap)
return -ENOMEM;
current->thread.vstate.datap = datap;
memset(&current->thread.vstate, 0, offsetof(struct __riscv_v_ext_state,
datap));
return 0;
}
#define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK)
#define VSTATE_CTRL_GET_NEXT(x) (((x) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) >> 2)
#define VSTATE_CTRL_MAKE_NEXT(x) (((x) << 2) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK)
#define VSTATE_CTRL_GET_INHERIT(x) (!!((x) & PR_RISCV_V_VSTATE_CTRL_INHERIT))
static inline int riscv_v_ctrl_get_cur(struct task_struct *tsk)
{
return VSTATE_CTRL_GET_CUR(tsk->thread.vstate_ctrl);
}
static inline int riscv_v_ctrl_get_next(struct task_struct *tsk)
{
return VSTATE_CTRL_GET_NEXT(tsk->thread.vstate_ctrl);
}
static inline bool riscv_v_ctrl_test_inherit(struct task_struct *tsk)
{
return VSTATE_CTRL_GET_INHERIT(tsk->thread.vstate_ctrl);
}
static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt,
bool inherit)
{
unsigned long ctrl;
ctrl = cur & PR_RISCV_V_VSTATE_CTRL_CUR_MASK;
ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt);
if (inherit)
ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT;
tsk->thread.vstate_ctrl = ctrl;
}
bool riscv_v_vstate_ctrl_user_allowed(void)
{
return riscv_v_ctrl_get_cur(current) == PR_RISCV_V_VSTATE_CTRL_ON;
}
EXPORT_SYMBOL_GPL(riscv_v_vstate_ctrl_user_allowed);
bool riscv_v_first_use_handler(struct pt_regs *regs)
{
u32 __user *epc = (u32 __user *)regs->epc;
u32 insn = (u32)regs->badaddr;
/* Do not handle if V is not supported, or disabled */
if (!(ELF_HWCAP & COMPAT_HWCAP_ISA_V))
return false;
/* If V has been enabled then it is not the first-use trap */
if (riscv_v_vstate_query(regs))
return false;
/* Get the instruction */
if (!insn) {
if (__get_user(insn, epc))
return false;
}
/* Filter out non-V instructions */
if (!insn_is_vector(insn))
return false;
/* Sanity check. datap should be null by the time of the first-use trap */
WARN_ON(current->thread.vstate.datap);
/*
* Now we sure that this is a V instruction. And it executes in the
* context where VS has been off. So, try to allocate the user's V
* context and resume execution.
*/
if (riscv_v_thread_zalloc()) {
force_sig(SIGBUS);
return true;
}
riscv_v_vstate_on(regs);
return true;
}
void riscv_v_vstate_ctrl_init(struct task_struct *tsk)
{
bool inherit;
int cur, next;
if (!has_vector())
return;
next = riscv_v_ctrl_get_next(tsk);
if (!next) {
if (READ_ONCE(riscv_v_implicit_uacc))
cur = PR_RISCV_V_VSTATE_CTRL_ON;
else
cur = PR_RISCV_V_VSTATE_CTRL_OFF;
} else {
cur = next;
}
/* Clear next mask if inherit-bit is not set */
inherit = riscv_v_ctrl_test_inherit(tsk);
if (!inherit)
next = PR_RISCV_V_VSTATE_CTRL_DEFAULT;
riscv_v_ctrl_set(tsk, cur, next, inherit);
}
long riscv_v_vstate_ctrl_get_current(void)
{
if (!has_vector())
return -EINVAL;
return current->thread.vstate_ctrl & PR_RISCV_V_VSTATE_CTRL_MASK;
}
long riscv_v_vstate_ctrl_set_current(unsigned long arg)
{
bool inherit;
int cur, next;
if (!has_vector())
return -EINVAL;
if (arg & ~PR_RISCV_V_VSTATE_CTRL_MASK)
return -EINVAL;
cur = VSTATE_CTRL_GET_CUR(arg);
switch (cur) {
case PR_RISCV_V_VSTATE_CTRL_OFF:
/* Do not allow user to turn off V if current is not off */
if (riscv_v_ctrl_get_cur(current) != PR_RISCV_V_VSTATE_CTRL_OFF)
return -EPERM;
break;
case PR_RISCV_V_VSTATE_CTRL_ON:
break;
case PR_RISCV_V_VSTATE_CTRL_DEFAULT:
cur = riscv_v_ctrl_get_cur(current);
break;
default:
return -EINVAL;
}
next = VSTATE_CTRL_GET_NEXT(arg);
inherit = VSTATE_CTRL_GET_INHERIT(arg);
switch (next) {
case PR_RISCV_V_VSTATE_CTRL_DEFAULT:
case PR_RISCV_V_VSTATE_CTRL_OFF:
case PR_RISCV_V_VSTATE_CTRL_ON:
riscv_v_ctrl_set(current, cur, next, inherit);
return 0;
}
return -EINVAL;
}
#ifdef CONFIG_SYSCTL
static struct ctl_table riscv_v_default_vstate_table[] = {
{
.procname = "riscv_v_default_allow",
.data = &riscv_v_implicit_uacc,
.maxlen = sizeof(riscv_v_implicit_uacc),
.mode = 0644,
.proc_handler = proc_dobool,
},
{ }
};
static int __init riscv_v_sysctl_init(void)
{
if (has_vector())
if (!register_sysctl("abi", riscv_v_default_vstate_table))
return -EINVAL;
return 0;
}
#else /* ! CONFIG_SYSCTL */
static int __init riscv_v_sysctl_init(void) { return 0; }
#endif /* ! CONFIG_SYSCTL */
static int riscv_v_init(void)
{
return riscv_v_sysctl_init();
}
core_initcall(riscv_v_init);
......@@ -17,6 +17,7 @@ kvm-y += mmu.o
kvm-y += vcpu.o
kvm-y += vcpu_exit.o
kvm-y += vcpu_fp.o
kvm-y += vcpu_vector.o
kvm-y += vcpu_insn.o
kvm-y += vcpu_switch.o
kvm-y += vcpu_sbi.o
......
......@@ -22,6 +22,8 @@
#include <asm/cacheflush.h>
#include <asm/hwcap.h>
#include <asm/sbi.h>
#include <asm/vector.h>
#include <asm/kvm_vcpu_vector.h>
const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = {
KVM_GENERIC_VCPU_STATS(),
......@@ -57,6 +59,7 @@ static const unsigned long kvm_isa_ext_arr[] = {
[KVM_RISCV_ISA_EXT_H] = RISCV_ISA_EXT_h,
[KVM_RISCV_ISA_EXT_I] = RISCV_ISA_EXT_i,
[KVM_RISCV_ISA_EXT_M] = RISCV_ISA_EXT_m,
[KVM_RISCV_ISA_EXT_V] = RISCV_ISA_EXT_v,
KVM_ISA_EXT_ARR(SSAIA),
KVM_ISA_EXT_ARR(SSTC),
......@@ -85,6 +88,8 @@ static bool kvm_riscv_vcpu_isa_enable_allowed(unsigned long ext)
switch (ext) {
case KVM_RISCV_ISA_EXT_H:
return false;
case KVM_RISCV_ISA_EXT_V:
return riscv_v_vstate_ctrl_user_allowed();
default:
break;
}
......@@ -138,6 +143,8 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu)
kvm_riscv_vcpu_fp_reset(vcpu);
kvm_riscv_vcpu_vector_reset(vcpu);
kvm_riscv_vcpu_timer_reset(vcpu);
kvm_riscv_vcpu_aia_reset(vcpu);
......@@ -198,6 +205,9 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
cntx->hstatus |= HSTATUS_SPVP;
cntx->hstatus |= HSTATUS_SPV;
if (kvm_riscv_vcpu_alloc_vector_context(vcpu, cntx))
return -ENOMEM;
/* By default, make CY, TM, and IR counters accessible in VU mode */
reset_csr->scounteren = 0x7;
......@@ -241,6 +251,9 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
/* Free unused pages pre-allocated for G-stage page table mappings */
kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
/* Free vector context space for host and guest kernel */
kvm_riscv_vcpu_free_vector_context(vcpu);
}
int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
......@@ -679,6 +692,9 @@ static int kvm_riscv_vcpu_set_reg(struct kvm_vcpu *vcpu,
return kvm_riscv_vcpu_set_reg_isa_ext(vcpu, reg);
case KVM_REG_RISCV_SBI_EXT:
return kvm_riscv_vcpu_set_reg_sbi_ext(vcpu, reg);
case KVM_REG_RISCV_VECTOR:
return kvm_riscv_vcpu_set_reg_vector(vcpu, reg,
KVM_REG_RISCV_VECTOR);
default:
break;
}
......@@ -708,6 +724,9 @@ static int kvm_riscv_vcpu_get_reg(struct kvm_vcpu *vcpu,
return kvm_riscv_vcpu_get_reg_isa_ext(vcpu, reg);
case KVM_REG_RISCV_SBI_EXT:
return kvm_riscv_vcpu_get_reg_sbi_ext(vcpu, reg);
case KVM_REG_RISCV_VECTOR:
return kvm_riscv_vcpu_get_reg_vector(vcpu, reg,
KVM_REG_RISCV_VECTOR);
default:
break;
}
......@@ -1002,6 +1021,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
kvm_riscv_vcpu_host_fp_save(&vcpu->arch.host_context);
kvm_riscv_vcpu_guest_fp_restore(&vcpu->arch.guest_context,
vcpu->arch.isa);
kvm_riscv_vcpu_host_vector_save(&vcpu->arch.host_context);
kvm_riscv_vcpu_guest_vector_restore(&vcpu->arch.guest_context,
vcpu->arch.isa);
kvm_riscv_vcpu_aia_load(vcpu, cpu);
......@@ -1021,6 +1043,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
kvm_riscv_vcpu_host_fp_restore(&vcpu->arch.host_context);
kvm_riscv_vcpu_timer_save(vcpu);
kvm_riscv_vcpu_guest_vector_save(&vcpu->arch.guest_context,
vcpu->arch.isa);
kvm_riscv_vcpu_host_vector_restore(&vcpu->arch.host_context);
csr->vsstatus = csr_read(CSR_VSSTATUS);
csr->vsie = csr_read(CSR_VSIE);
......
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2022 SiFive
*
* Authors:
* Vincent Chen <vincent.chen@sifive.com>
* Greentime Hu <greentime.hu@sifive.com>
*/
#include <linux/errno.h>
#include <linux/err.h>
#include <linux/kvm_host.h>
#include <linux/uaccess.h>
#include <asm/hwcap.h>
#include <asm/kvm_vcpu_vector.h>
#include <asm/vector.h>
#ifdef CONFIG_RISCV_ISA_V
void kvm_riscv_vcpu_vector_reset(struct kvm_vcpu *vcpu)
{
unsigned long *isa = vcpu->arch.isa;
struct kvm_cpu_context *cntx = &vcpu->arch.guest_context;
cntx->sstatus &= ~SR_VS;
if (riscv_isa_extension_available(isa, v)) {
cntx->sstatus |= SR_VS_INITIAL;
WARN_ON(!cntx->vector.datap);
memset(cntx->vector.datap, 0, riscv_v_vsize);
} else {
cntx->sstatus |= SR_VS_OFF;
}
}
static void kvm_riscv_vcpu_vector_clean(struct kvm_cpu_context *cntx)
{
cntx->sstatus &= ~SR_VS;
cntx->sstatus |= SR_VS_CLEAN;
}
void kvm_riscv_vcpu_guest_vector_save(struct kvm_cpu_context *cntx,
unsigned long *isa)
{
if ((cntx->sstatus & SR_VS) == SR_VS_DIRTY) {
if (riscv_isa_extension_available(isa, v))
__kvm_riscv_vector_save(cntx);
kvm_riscv_vcpu_vector_clean(cntx);
}
}
void kvm_riscv_vcpu_guest_vector_restore(struct kvm_cpu_context *cntx,
unsigned long *isa)
{
if ((cntx->sstatus & SR_VS) != SR_VS_OFF) {
if (riscv_isa_extension_available(isa, v))
__kvm_riscv_vector_restore(cntx);
kvm_riscv_vcpu_vector_clean(cntx);
}
}
void kvm_riscv_vcpu_host_vector_save(struct kvm_cpu_context *cntx)
{
/* No need to check host sstatus as it can be modified outside */
if (riscv_isa_extension_available(NULL, v))
__kvm_riscv_vector_save(cntx);
}
void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx)
{
if (riscv_isa_extension_available(NULL, v))
__kvm_riscv_vector_restore(cntx);
}
int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
struct kvm_cpu_context *cntx)
{
cntx->vector.datap = kmalloc(riscv_v_vsize, GFP_KERNEL);
if (!cntx->vector.datap)
return -ENOMEM;
vcpu->arch.host_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL);
if (!vcpu->arch.host_context.vector.datap)
return -ENOMEM;
return 0;
}
void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu)
{
kfree(vcpu->arch.guest_reset_context.vector.datap);
kfree(vcpu->arch.host_context.vector.datap);
}
#endif
static void *kvm_riscv_vcpu_vreg_addr(struct kvm_vcpu *vcpu,
unsigned long reg_num,
size_t reg_size)
{
struct kvm_cpu_context *cntx = &vcpu->arch.guest_context;
void *reg_val;
size_t vlenb = riscv_v_vsize / 32;
if (reg_num < KVM_REG_RISCV_VECTOR_REG(0)) {
if (reg_size != sizeof(unsigned long))
return NULL;
switch (reg_num) {
case KVM_REG_RISCV_VECTOR_CSR_REG(vstart):
reg_val = &cntx->vector.vstart;
break;
case KVM_REG_RISCV_VECTOR_CSR_REG(vl):
reg_val = &cntx->vector.vl;
break;
case KVM_REG_RISCV_VECTOR_CSR_REG(vtype):
reg_val = &cntx->vector.vtype;
break;
case KVM_REG_RISCV_VECTOR_CSR_REG(vcsr):
reg_val = &cntx->vector.vcsr;
break;
case KVM_REG_RISCV_VECTOR_CSR_REG(datap):
default:
return NULL;
}
} else if (reg_num <= KVM_REG_RISCV_VECTOR_REG(31)) {
if (reg_size != vlenb)
return NULL;
reg_val = cntx->vector.datap
+ (reg_num - KVM_REG_RISCV_VECTOR_REG(0)) * vlenb;
} else {
return NULL;
}
return reg_val;
}
int kvm_riscv_vcpu_get_reg_vector(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg,
unsigned long rtype)
{
unsigned long *isa = vcpu->arch.isa;
unsigned long __user *uaddr =
(unsigned long __user *)(unsigned long)reg->addr;
unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK |
KVM_REG_SIZE_MASK |
rtype);
void *reg_val = NULL;
size_t reg_size = KVM_REG_SIZE(reg->id);
if (rtype == KVM_REG_RISCV_VECTOR &&
riscv_isa_extension_available(isa, v)) {
reg_val = kvm_riscv_vcpu_vreg_addr(vcpu, reg_num, reg_size);
}
if (!reg_val)
return -EINVAL;
if (copy_to_user(uaddr, reg_val, reg_size))
return -EFAULT;
return 0;
}
int kvm_riscv_vcpu_set_reg_vector(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg,
unsigned long rtype)
{
unsigned long *isa = vcpu->arch.isa;
unsigned long __user *uaddr =
(unsigned long __user *)(unsigned long)reg->addr;
unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK |
KVM_REG_SIZE_MASK |
rtype);
void *reg_val = NULL;
size_t reg_size = KVM_REG_SIZE(reg->id);
if (rtype == KVM_REG_RISCV_VECTOR &&
riscv_isa_extension_available(isa, v)) {
reg_val = kvm_riscv_vcpu_vreg_addr(vcpu, reg_num, reg_size);
}
if (!reg_val)
return -EINVAL;
if (copy_from_user(reg_val, uaddr, reg_size))
return -EFAULT;
return 0;
}
......@@ -440,6 +440,7 @@ typedef struct elf64_shdr {
#define NT_MIPS_DSP 0x800 /* MIPS DSP ASE registers */
#define NT_MIPS_FP_MODE 0x801 /* MIPS floating-point mode */
#define NT_MIPS_MSA 0x802 /* MIPS SIMD registers */
#define NT_RISCV_VECTOR 0x900 /* RISC-V vector registers */
#define NT_LOONGARCH_CPUCFG 0xa00 /* LoongArch CPU config registers */
#define NT_LOONGARCH_CSR 0xa01 /* LoongArch control and status registers */
#define NT_LOONGARCH_LSX 0xa02 /* LoongArch Loongson SIMD Extension registers */
......
......@@ -294,4 +294,15 @@ struct prctl_mm_map {
#define PR_SET_MEMORY_MERGE 67
#define PR_GET_MEMORY_MERGE 68
#define PR_RISCV_V_SET_CONTROL 69
#define PR_RISCV_V_GET_CONTROL 70
# define PR_RISCV_V_VSTATE_CTRL_DEFAULT 0
# define PR_RISCV_V_VSTATE_CTRL_OFF 1
# define PR_RISCV_V_VSTATE_CTRL_ON 2
# define PR_RISCV_V_VSTATE_CTRL_INHERIT (1 << 4)
# define PR_RISCV_V_VSTATE_CTRL_CUR_MASK 0x3
# define PR_RISCV_V_VSTATE_CTRL_NEXT_MASK 0xc
# define PR_RISCV_V_VSTATE_CTRL_MASK 0x1f
#endif /* _LINUX_PRCTL_H */
......@@ -140,6 +140,12 @@
#ifndef GET_TAGGED_ADDR_CTRL
# define GET_TAGGED_ADDR_CTRL() (-EINVAL)
#endif
#ifndef RISCV_V_SET_CONTROL
# define RISCV_V_SET_CONTROL(a) (-EINVAL)
#endif
#ifndef RISCV_V_GET_CONTROL
# define RISCV_V_GET_CONTROL() (-EINVAL)
#endif
/*
* this is where the system-wide overflow UID and GID are defined, for
......@@ -2708,6 +2714,12 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
error = !!test_bit(MMF_VM_MERGE_ANY, &me->mm->flags);
break;
#endif
case PR_RISCV_V_SET_CONTROL:
error = RISCV_V_SET_CONTROL(arg2);
break;
case PR_RISCV_V_GET_CONTROL:
error = RISCV_V_GET_CONTROL();
break;
default:
error = -EINVAL;
break;
......
......@@ -5,7 +5,7 @@
ARCH ?= $(shell uname -m 2>/dev/null || echo not)
ifneq (,$(filter $(ARCH),riscv))
RISCV_SUBTARGETS ?= hwprobe
RISCV_SUBTARGETS ?= hwprobe vector
else
RISCV_SUBTARGETS :=
endif
......
# SPDX-License-Identifier: GPL-2.0
# Copyright (C) 2021 ARM Limited
# Originally tools/testing/arm64/abi/Makefile
TEST_GEN_PROGS := vstate_prctl
TEST_GEN_PROGS_EXTENDED := vstate_exec_nolibc
include ../../lib.mk
$(OUTPUT)/vstate_prctl: vstate_prctl.c ../hwprobe/sys_hwprobe.S
$(CC) -static -o$@ $(CFLAGS) $(LDFLAGS) $^
$(OUTPUT)/vstate_exec_nolibc: vstate_exec_nolibc.c
$(CC) -nostdlib -static -include ../../../../include/nolibc/nolibc.h \
-Wall $(CFLAGS) $(LDFLAGS) $^ -o $@ -lgcc
// SPDX-License-Identifier: GPL-2.0-only
#include <sys/prctl.h>
#define THIS_PROGRAM "./vstate_exec_nolibc"
int main(int argc, char **argv)
{
int rc, pid, status, test_inherit = 0;
long ctrl, ctrl_c;
char *exec_argv[2], *exec_envp[2];
if (argc > 1)
test_inherit = 1;
ctrl = my_syscall1(__NR_prctl, PR_RISCV_V_GET_CONTROL);
if (ctrl < 0) {
puts("PR_RISCV_V_GET_CONTROL is not supported\n");
return ctrl;
}
if (test_inherit) {
pid = fork();
if (pid == -1) {
puts("fork failed\n");
exit(-1);
}
/* child */
if (!pid) {
exec_argv[0] = THIS_PROGRAM;
exec_argv[1] = NULL;
exec_envp[0] = NULL;
exec_envp[1] = NULL;
/* launch the program again to check inherit */
rc = execve(THIS_PROGRAM, exec_argv, exec_envp);
if (rc) {
puts("child execve failed\n");
exit(-1);
}
}
} else {
pid = fork();
if (pid == -1) {
puts("fork failed\n");
exit(-1);
}
if (!pid) {
rc = my_syscall1(__NR_prctl, PR_RISCV_V_GET_CONTROL);
if (rc != ctrl) {
puts("child's vstate_ctrl not equal to parent's\n");
exit(-1);
}
asm volatile (".option push\n\t"
".option arch, +v\n\t"
"vsetvli x0, x0, e32, m8, ta, ma\n\t"
".option pop\n\t"
);
exit(ctrl);
}
}
rc = waitpid(-1, &status, 0);
if (WIFEXITED(status) && WEXITSTATUS(status) == -1) {
puts("child exited abnormally\n");
exit(-1);
}
if (WIFSIGNALED(status)) {
if (WTERMSIG(status) != SIGILL) {
puts("child was terminated by unexpected signal\n");
exit(-1);
}
if ((ctrl & PR_RISCV_V_VSTATE_CTRL_CUR_MASK) != PR_RISCV_V_VSTATE_CTRL_OFF) {
puts("child signaled by illegal V access but vstate_ctrl is not off\n");
exit(-1);
}
/* child terminated, and its vstate_ctrl is off */
exit(ctrl);
}
ctrl_c = WEXITSTATUS(status);
if (test_inherit) {
if (ctrl & PR_RISCV_V_VSTATE_CTRL_INHERIT) {
if (!(ctrl_c & PR_RISCV_V_VSTATE_CTRL_INHERIT)) {
puts("parent has inherit bit, but child has not\n");
exit(-1);
}
}
rc = (ctrl & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) >> 2;
if (rc != PR_RISCV_V_VSTATE_CTRL_DEFAULT) {
if (rc != (ctrl_c & PR_RISCV_V_VSTATE_CTRL_CUR_MASK)) {
puts("parent's next setting does not equal to child's\n");
exit(-1);
}
if (!(ctrl & PR_RISCV_V_VSTATE_CTRL_INHERIT)) {
if ((ctrl_c & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) !=
PR_RISCV_V_VSTATE_CTRL_DEFAULT) {
puts("must clear child's next vstate_ctrl if !inherit\n");
exit(-1);
}
}
}
}
return ctrl;
}
// SPDX-License-Identifier: GPL-2.0-only
#include <sys/prctl.h>
#include <unistd.h>
#include <asm/hwprobe.h>
#include <errno.h>
#include <sys/wait.h>
#include "../../kselftest.h"
/*
* Rather than relying on having a new enough libc to define this, just do it
* ourselves. This way we don't need to be coupled to a new-enough libc to
* contain the call.
*/
long riscv_hwprobe(struct riscv_hwprobe *pairs, size_t pair_count,
size_t cpu_count, unsigned long *cpus, unsigned int flags);
#define NEXT_PROGRAM "./vstate_exec_nolibc"
static int launch_test(int test_inherit)
{
char *exec_argv[3], *exec_envp[1];
int rc, pid, status;
pid = fork();
if (pid < 0) {
ksft_test_result_fail("fork failed %d", pid);
return -1;
}
if (!pid) {
exec_argv[0] = NEXT_PROGRAM;
exec_argv[1] = test_inherit != 0 ? "x" : NULL;
exec_argv[2] = NULL;
exec_envp[0] = NULL;
/* launch the program again to check inherit */
rc = execve(NEXT_PROGRAM, exec_argv, exec_envp);
if (rc) {
perror("execve");
ksft_test_result_fail("child execve failed %d\n", rc);
exit(-1);
}
}
rc = waitpid(-1, &status, 0);
if (rc < 0) {
ksft_test_result_fail("waitpid failed\n");
return -3;
}
if ((WIFEXITED(status) && WEXITSTATUS(status) == -1) ||
WIFSIGNALED(status)) {
ksft_test_result_fail("child exited abnormally\n");
return -4;
}
return WEXITSTATUS(status);
}
int test_and_compare_child(long provided, long expected, int inherit)
{
int rc;
rc = prctl(PR_RISCV_V_SET_CONTROL, provided);
if (rc != 0) {
ksft_test_result_fail("prctl with provided arg %lx failed with code %d\n",
provided, rc);
return -1;
}
rc = launch_test(inherit);
if (rc != expected) {
ksft_test_result_fail("Test failed, check %d != %d\n", rc,
expected);
return -2;
}
return 0;
}
#define PR_RISCV_V_VSTATE_CTRL_CUR_SHIFT 0
#define PR_RISCV_V_VSTATE_CTRL_NEXT_SHIFT 2
int main(void)
{
struct riscv_hwprobe pair;
long flag, expected;
long rc;
pair.key = RISCV_HWPROBE_KEY_IMA_EXT_0;
rc = riscv_hwprobe(&pair, 1, 0, NULL, 0);
if (rc < 0) {
ksft_test_result_fail("hwprobe() failed with %d\n", rc);
return -1;
}
if (pair.key != RISCV_HWPROBE_KEY_IMA_EXT_0) {
ksft_test_result_fail("hwprobe cannot probe RISCV_HWPROBE_KEY_IMA_EXT_0\n");
return -2;
}
if (!(pair.value & RISCV_HWPROBE_IMA_V)) {
rc = prctl(PR_RISCV_V_GET_CONTROL);
if (rc != -1 || errno != EINVAL) {
ksft_test_result_fail("GET_CONTROL should fail on kernel/hw without V\n");
return -3;
}
rc = prctl(PR_RISCV_V_SET_CONTROL, PR_RISCV_V_VSTATE_CTRL_ON);
if (rc != -1 || errno != EINVAL) {
ksft_test_result_fail("GET_CONTROL should fail on kernel/hw without V\n");
return -4;
}
ksft_test_result_skip("Vector not supported\n");
return 0;
}
flag = PR_RISCV_V_VSTATE_CTRL_ON;
rc = prctl(PR_RISCV_V_SET_CONTROL, flag);
if (rc != 0) {
ksft_test_result_fail("Enabling V for current should always success\n");
return -5;
}
flag = PR_RISCV_V_VSTATE_CTRL_OFF;
rc = prctl(PR_RISCV_V_SET_CONTROL, flag);
if (rc != -1 || errno != EPERM) {
ksft_test_result_fail("Disabling current's V alive must fail with EPERM(%d)\n",
errno);
return -5;
}
/* Turn on next's vector explicitly and test */
flag = PR_RISCV_V_VSTATE_CTRL_ON << PR_RISCV_V_VSTATE_CTRL_NEXT_SHIFT;
if (test_and_compare_child(flag, PR_RISCV_V_VSTATE_CTRL_ON, 0))
return -6;
/* Turn off next's vector explicitly and test */
flag = PR_RISCV_V_VSTATE_CTRL_OFF << PR_RISCV_V_VSTATE_CTRL_NEXT_SHIFT;
if (test_and_compare_child(flag, PR_RISCV_V_VSTATE_CTRL_OFF, 0))
return -7;
/* Turn on next's vector explicitly and test inherit */
flag = PR_RISCV_V_VSTATE_CTRL_ON << PR_RISCV_V_VSTATE_CTRL_NEXT_SHIFT;
flag |= PR_RISCV_V_VSTATE_CTRL_INHERIT;
expected = flag | PR_RISCV_V_VSTATE_CTRL_ON;
if (test_and_compare_child(flag, expected, 0))
return -8;
if (test_and_compare_child(flag, expected, 1))
return -9;
/* Turn off next's vector explicitly and test inherit */
flag = PR_RISCV_V_VSTATE_CTRL_OFF << PR_RISCV_V_VSTATE_CTRL_NEXT_SHIFT;
flag |= PR_RISCV_V_VSTATE_CTRL_INHERIT;
expected = flag | PR_RISCV_V_VSTATE_CTRL_OFF;
if (test_and_compare_child(flag, expected, 0))
return -10;
if (test_and_compare_child(flag, expected, 1))
return -11;
/* arguments should fail with EINVAL */
rc = prctl(PR_RISCV_V_SET_CONTROL, 0xff0);
if (rc != -1 || errno != EINVAL) {
ksft_test_result_fail("Undefined control argument should return EINVAL\n");
return -12;
}
rc = prctl(PR_RISCV_V_SET_CONTROL, 0x3);
if (rc != -1 || errno != EINVAL) {
ksft_test_result_fail("Undefined control argument should return EINVAL\n");
return -12;
}
rc = prctl(PR_RISCV_V_SET_CONTROL, 0xc);
if (rc != -1 || errno != EINVAL) {
ksft_test_result_fail("Undefined control argument should return EINVAL\n");
return -12;
}
rc = prctl(PR_RISCV_V_SET_CONTROL, 0xc);
if (rc != -1 || errno != EINVAL) {
ksft_test_result_fail("Undefined control argument should return EINVAL\n");
return -12;
}
ksft_test_result_pass("tests for riscv_v_vstate_ctrl pass\n");
ksft_exit_pass();
return 0;
}
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment