Commit 88fe3529 authored by David S. Miller's avatar David S. Miller

Merge branch 'sparc64-ADI'

Khalid Aziz says:

====================
Application Data Integrity feature introduced by SPARC M7

V12 changes:
This series is same as v10 and v11 and was simply rebased on 4.16-rc2
kernel and patch 11 was added to update signal delivery code to use the
new helper functions added by Eric Biederman. Can mm maintainers please
review patches 2, 7, 8 and 9 which are arch independent, and
include/linux/mm.h and mm/ksm.c changes in patch 10 and ack these if
everything looks good?

SPARC M7 processor adds additional metadata for memory address space
that can be used to secure access to regions of memory. This additional
metadata is implemented as a 4-bit tag attached to each cacheline size
block of memory. A task can set a tag on any number of such blocks.
Access to such block is granted only if the virtual address used to
access that block of memory has the tag encoded in the uppermost 4 bits
of VA. Since sparc processor does not implement all 64 bits of VA, top 4
bits are available for ADI tags. Any mismatch between tag encoded in VA
and tag set on the memory block results in a trap. Tags are verified in
the VA presented to the MMU and tags are associated with the physical
page VA maps on to. If a memory page is swapped out and page frame gets
reused for another task, the tags are lost and hence must be saved when
swapping or migrating the page.

A userspace task enables ADI through mprotect(). This patch series adds
a page protection bit PROT_ADI and a corresponding VMA flag
VM_SPARC_ADI. VM_SPARC_ADI is used to trigger setting TTE.mcd bit in the
sparc pte that enables ADI checking on the corresponding page. MMU
validates the tag embedded in VA for every page that has TTE.mcd bit set
in its pte. After enabling ADI on a memory range, the userspace task can
set ADI version tags using stxa instruction with ASI_MCD_PRIMARY or
ASI_MCD_ST_BLKINIT_PRIMARY ASI.

Once userspace task calls mprotect() with PROT_ADI, kernel takes
following overall steps:

1. Find the VMAs covering the address range passed in to mprotect and
set VM_SPARC_ADI flag. If address range covers a subset of a VMA, the
VMA will be split.

2. When a page is allocated for a VA and the VMA covering this VA has
VM_SPARC_ADI flag set, set the TTE.mcd bit so MMU will check the
vwersion tag.

3. Userspace can now set version tags on the memory it has enabled ADI
on. Userspace accesses ADI enabled memory using a virtual address that
has the version tag embedded in the high bits. MMU validates this
version tag against the actual tag set on the memory. If tag matches,
MMU performs the VA->PA translation and access is granted. If there is a
mismatch, hypervisor sends a data access exception or precise memory
corruption detected exception depending upon whether precise exceptions
are enabled or not (controlled by MCDPERR register). Kernel sends
SIGSEGV to the task with appropriate si_code.

4. If a page is being swapped out or migrated, kernel must save any ADI
tags set on the page. Kernel maintains a page worth of tag storage
descriptors. Each descriptors pointsto a tag storage space and the
address range it covers. If the page being swapped out or migrated has
ADI enabled on it, kernel finds a tag storage descriptor that covers the
address range for the page or allocates a new descriptor if none of the
existing descriptors cover the address range. Kernel saves tags from the
page into the tag storage space descriptor points to.

5. When the page is swapped back in or reinstantiated after migration,
kernel restores the version tags on the new physical page by retrieving
the original tag from tag storage pointed to by a tag storage descriptor
for the virtual address range for new page.

User task can disable ADI by calling mprotect() again on the memory
range with PROT_ADI bit unset. Kernel clears the VM_SPARC_ADI flag in
VMAs, merges adjacent VMAs if necessary, and clears TTE.mcd bit in the
corresponding ptes.

IOMMU does not support ADI checking. Any version tags embedded in the
top bits of VA meant for IOMMU, are cleared and replaced with sign
extension of the first non-version tag bit (bit 59 for SPARC M7) for
IOMMU addresses.

This patch series adds support for this feature in 11 patches:

Patch 1/11
  Tag mismatch on access by a task results in a trap from hypervisor as
  data access exception or a precide memory corruption detected
  exception. As part of handling these exceptions, kernel sends a
  SIGSEGV to user process with special si_code to indicate which fault
  occurred. This patch adds three new si_codes to differentiate between
  various mismatch errors.

Patch 2/11
  When a page is swapped or migrated, metadata associated with the page
  must be saved so it can be restored later. This patch adds a new
  function that saves/restores this metadata when updating pte upon a
  swap/migration.

Patch 3/11
  SPARC M7 processor adds new fields to control registers to support ADI
  feature. It also adds a new exception for precise traps on tag
  mismatch. This patch adds definitions for the new control register
  fields, new ASIs for ADI and an exception handler for the precise trap
  on tag mismatch.

Patch 4/11
  New hypervisor fault types were added by sparc M7 processor to support
  ADI feature. This patch adds code to handle these fault types for data
  access exception handler.

Patch 5/11
  When ADI is in use for a page and a tag mismatch occurs, processor
  raises "Memory corruption Detected" trap. This patch adds a handler
  for this trap.

Patch 6/11
  ADI usage is governed by ADI properties on a platform. These
  properties are provided to kernel by firmware. Thsi patch adds new
  auxiliary vectors that provide these values to userpsace.

Patch 7/11
  arch_validate_prot() is used to validate the new protection bits asked
  for by the userspace app. Validating protection bits may need the
  context of address space the bits are being applied to. One such
  example is PROT_ADI bit on sparc processor that enables ADI protection
  on an address range. ADI protection applies only to addresses covered
  by physical RAM and not other PFN mapped addresses or device
  addresses. This patch adds "address" to the parameters being passed to
  arch_validate_prot() to provide that context.

Patch 8/11
  When protection bits are changed on a page, kernel carries forward all
  protection bits except for read/write/exec. Additional code was added
  to allow kernel to clear PKEY bits on x86 but this requirement to
  clear other bits is not unique to x86. This patch extends the existing
  code to allow other architectures to clear any other protection bits
  as well on protection bit change.

Patch 9/11
  When a processor supports additional metadata on memory pages, that
  additional metadata needs to be copied to new memory pages when those
  pages are moved. This patch allows architecture specific code to
  replace the default copy_highpage() routine with arch specific
  version that copies the metadata as well besides the data on the page.

Patch 10/11
  This patch adds support for a user space task to enable ADI and enable
  tag checking for subsets of its address space. As part of enabling
  this feature, this patch adds to support manipulation of precise
  exception for memory corruption detection, adds code to save and
  restore tags on page swap and migration, and adds code to handle ADI
  tagged addresses for DMA.

Patch 11/11
  Update signal delivery code in arch/sparc/kernel/traps_64.c to use
  the new helper function force_sig_fault() added by commit
  f8ec6601 ("signal: Add send_sig_fault and force_sig_fault").

Changelog v12:

	  - Rebased to 4.16-rc2
	  - Added patch 11 to update signal delivery functions

Changelog v11:

	  - Rebased to 4.15

Changelog v10:

	  - Patch 1/10: Updated si_codes definitions for SEGV to match 4.14
	  - Patch 2/10: No changes
	  - Patch 3/10: Updated copyright
	  - Patch 4/10: No changes
	  - Patch 5/10: No changes
	  - Patch 6/10: Updated copyright
	  - Patch 7/10: No changes
	  - Patch 8/10: No changes
	  - Patch 9/10: No changes
	  - Patch 10/10: Added code to return from kernel path to set
	    PSTATE.mcde if kernel continues execution in another thread
	      (Suggested by Anthony)

Changelog v9:

	  - Patch 1/10: No changes
	  - Patch 2/10: No changes
	  - Patch 3/10: No changes
	  - Patch 4/10: No changes
	  - Patch 5/10: No changes
	  - Patch 6/10: No changes
	  - Patch 7/10: No changes
	  - Patch 8/10: No changes
	  - Patch 9/10: New patch
	  - Patch 10/10: Patch 9 from v8. Added code to copy ADI tags when
	    pages are migrated. Updated code to detect overflow and underflow
	      of addresses when allocating tag storage.

Changelog v8:

	  - Patch 1/9: No changes
	  - Patch 2/9: Fixed and erroneous "}"
	  - Patch 3/9: Minor print formatting change
	  - Patch 4/9: No changes
	  - Patch 5/9: No changes
	  - Patch 6/9: Added AT_ADI_UEONADI back
	  - Patch 7/9: Added addr parameter to powerpc arch_validate_prot()
	  - Patch 8/9: No changes
	  - Patch 9/9:
	    - Documentation updates
	      - Added an IPI on mprotect(...PROT_ADI...) call and
	      	  restore of TSTATE.MCDE on context switch
		  	  - Removed restriction on enabling ADI on read-only
			      memory
				- Changed kzalloc() for tag storage to use GFP_NOWAIT
				  - Added code to handle overflow and underflow when
				      allocating tag storage
				      		 - Replaced sun_m7_patch_1insn_range() with
						     sun4v_patch_1insn_range()
							- Added membar after restoring ADI tags in
							    copy_user_highpage()

Changelog v7:

	  - Patch 1/9: No changes
	  - Patch 2/9: Updated parameters to arch specific swap in/out
	    handlers
	    - Patch 3/9: No changes
	    - Patch 4/9: new patch split off from patch 4/4 in v6
	    - Patch 5/9: new patch split off from patch 4/4 in v6
	    - Patch 6/9: new patch split off from patch 4/4 in v6
	    - Patch 7/9: new patch
	    - Patch 8/9: new patch
	    - Patch 9/9:
	      - Enhanced arch_validate_prot() to enable ADI only on
	      	  writable addresses backed by physical RAM
		  	   - Added support for saving/restoring ADI tags for each
			       ADI block size address range on a page on swap in/out
			       	   - copy ADI tags on COW
				     - Updated values for auxiliary vectors to not conflict
				         with values on other architectures to avoid conflict
					        in glibc
						   - Disable same page merging on ADI enabled pages
						     - Enable ADI only on writable addresses backed by
						         physical RAM
							 	  - Split parts of patch off into separate patches

Changelog v6:
	  - Patch 1/4: No changes
	  - Patch 2/4: No changes
	  - Patch 3/4: Added missing nop in the delay slot in
	    sun4v_mcd_detect_precise
	    - Patch 4/4: Eliminated instructions to read and write PSTATE
	      as well as MCDPER and PMCDPER on every access to userspace
	        addresses by setting PSTATE and PMCDPER correctly upon entry
		  into kernel

Changelog v5:
	  - Patch 1/4: No changes
	  - Patch 2/4: Replaced set_swp_pte_at() with new architecture
	    functions arch_do_swap_page() and arch_unmap_one() that
	      suppoprt architecture specific actions to be taken on page
	        swap and migration
		- Patch 3/4: Fixed indentation issues in assembly code
		- Patch 4/4:
		  - Fixed indentation issues and instrcuctions in assembly
		      code
			- Removed CONFIG_SPARC64 from mdesc.c
			  - Changed to maintain state of MCDPER register in thread
			      info flags as opposed to in mm context. MCDPER is a
			      	     per-thread state and belongs in thread info flag as
				     		  opposed to mm context which is shared across threads.
						  	    Added comments to clarify this is a lazily maintained
							    	    state and must be updated on context switch and
								    	    copy_process()
									    		   - Updated code to use the new arch_do_swap_page() and
											       arch_unmap_one() functions

Testing:

- All functionality was tested with 8K normal pages as well as hugepages
  using malloc, mmap and shm.
- Multiple long duration stress tests were run using hugepages over 2+
  months. Normal pages were tested with shorter duration stress tests.
- Tested swapping with malloc and shm by reducing max memory and
  allocating three times the available system memory by active processes
  using ADI on allocated memory. Ran through multiple hours long runs of
  this test.
- Tested page migration with malloc and shm by migrating data pages of
  active ADI test process using migratepages, back and forth between two
  nodes every few seconds over an hour long run. Verified page migration
  through /proc/<pid>/numa_maps.
- Tested COW support using test that forks children that read from
  ADI enabled pages shared with parent and other children and write to
  them as well forcing COW.
====================
Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
parents 8f5fd927 b9fa0365
Application Data Integrity (ADI)
================================
SPARC M7 processor adds the Application Data Integrity (ADI) feature.
ADI allows a task to set version tags on any subset of its address
space. Once ADI is enabled and version tags are set for ranges of
address space of a task, the processor will compare the tag in pointers
to memory in these ranges to the version set by the application
previously. Access to memory is granted only if the tag in given pointer
matches the tag set by the application. In case of mismatch, processor
raises an exception.
Following steps must be taken by a task to enable ADI fully:
1. Set the user mode PSTATE.mcde bit. This acts as master switch for
the task's entire address space to enable/disable ADI for the task.
2. Set TTE.mcd bit on any TLB entries that correspond to the range of
addresses ADI is being enabled on. MMU checks the version tag only
on the pages that have TTE.mcd bit set.
3. Set the version tag for virtual addresses using stxa instruction
and one of the MCD specific ASIs. Each stxa instruction sets the
given tag for one ADI block size number of bytes. This step must
be repeated for entire page to set tags for entire page.
ADI block size for the platform is provided by the hypervisor to kernel
in machine description tables. Hypervisor also provides the number of
top bits in the virtual address that specify the version tag. Once
version tag has been set for a memory location, the tag is stored in the
physical memory and the same tag must be present in the ADI version tag
bits of the virtual address being presented to the MMU. For example on
SPARC M7 processor, MMU uses bits 63-60 for version tags and ADI block
size is same as cacheline size which is 64 bytes. A task that sets ADI
version to, say 10, on a range of memory, must access that memory using
virtual addresses that contain 0xa in bits 63-60.
ADI is enabled on a set of pages using mprotect() with PROT_ADI flag.
When ADI is enabled on a set of pages by a task for the first time,
kernel sets the PSTATE.mcde bit fot the task. Version tags for memory
addresses are set with an stxa instruction on the addresses using
ASI_MCD_PRIMARY or ASI_MCD_ST_BLKINIT_PRIMARY. ADI block size is
provided by the hypervisor to the kernel. Kernel returns the value of
ADI block size to userspace using auxiliary vector along with other ADI
info. Following auxiliary vectors are provided by the kernel:
AT_ADI_BLKSZ ADI block size. This is the granularity and
alignment, in bytes, of ADI versioning.
AT_ADI_NBITS Number of ADI version bits in the VA
IMPORTANT NOTES:
- Version tag values of 0x0 and 0xf are reserved. These values match any
tag in virtual address and never generate a mismatch exception.
- Version tags are set on virtual addresses from userspace even though
tags are stored in physical memory. Tags are set on a physical page
after it has been allocated to a task and a pte has been created for
it.
- When a task frees a memory page it had set version tags on, the page
goes back to free page pool. When this page is re-allocated to a task,
kernel clears the page using block initialization ASI which clears the
version tags as well for the page. If a page allocated to a task is
freed and allocated back to the same task, old version tags set by the
task on that page will no longer be present.
- ADI tag mismatches are not detected for non-faulting loads.
- Kernel does not set any tags for user pages and it is entirely a
task's responsibility to set any version tags. Kernel does ensure the
version tags are preserved if a page is swapped out to the disk and
swapped back in. It also preserves that version tags if a page is
migrated.
- ADI works for any size pages. A userspace task need not be aware of
page size when using ADI. It can simply select a virtual address
range, enable ADI on the range using mprotect() and set version tags
for the entire range. mprotect() ensures range is aligned to page size
and is a multiple of page size.
- ADI tags can only be set on writable memory. For example, ADI tags can
not be set on read-only mappings.
ADI related traps
-----------------
With ADI enabled, following new traps may occur:
Disrupting memory corruption
When a store accesses a memory localtion that has TTE.mcd=1,
the task is running with ADI enabled (PSTATE.mcde=1), and the ADI
tag in the address used (bits 63:60) does not match the tag set on
the corresponding cacheline, a memory corruption trap occurs. By
default, it is a disrupting trap and is sent to the hypervisor
first. Hypervisor creates a sun4v error report and sends a
resumable error (TT=0x7e) trap to the kernel. The kernel sends
a SIGSEGV to the task that resulted in this trap with the following
info:
siginfo.si_signo = SIGSEGV;
siginfo.errno = 0;
siginfo.si_code = SEGV_ADIDERR;
siginfo.si_addr = addr; /* PC where first mismatch occurred */
siginfo.si_trapno = 0;
Precise memory corruption
When a store accesses a memory location that has TTE.mcd=1,
the task is running with ADI enabled (PSTATE.mcde=1), and the ADI
tag in the address used (bits 63:60) does not match the tag set on
the corresponding cacheline, a memory corruption trap occurs. If
MCD precise exception is enabled (MCDPERR=1), a precise
exception is sent to the kernel with TT=0x1a. The kernel sends
a SIGSEGV to the task that resulted in this trap with the following
info:
siginfo.si_signo = SIGSEGV;
siginfo.errno = 0;
siginfo.si_code = SEGV_ADIPERR;
siginfo.si_addr = addr; /* address that caused trap */
siginfo.si_trapno = 0;
NOTE: ADI tag mismatch on a load always results in precise trap.
MCD disabled
When a task has not enabled ADI and attempts to set ADI version
on a memory address, processor sends an MCD disabled trap. This
trap is handled by hypervisor first and the hypervisor vectors this
trap through to the kernel as Data Access Exception trap with
fault type set to 0xa (invalid ASI). When this occurs, the kernel
sends the task SIGSEGV signal with following info:
siginfo.si_signo = SIGSEGV;
siginfo.errno = 0;
siginfo.si_code = SEGV_ACCADI;
siginfo.si_addr = addr; /* address that caused trap */
siginfo.si_trapno = 0;
Sample program to use ADI
-------------------------
Following sample program is meant to illustrate how to use the ADI
functionality.
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <elf.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <sys/mman.h>
#include <asm/asi.h>
#ifndef AT_ADI_BLKSZ
#define AT_ADI_BLKSZ 48
#endif
#ifndef AT_ADI_NBITS
#define AT_ADI_NBITS 49
#endif
#ifndef PROT_ADI
#define PROT_ADI 0x10
#endif
#define BUFFER_SIZE 32*1024*1024UL
main(int argc, char* argv[], char* envp[])
{
unsigned long i, mcde, adi_blksz, adi_nbits;
char *shmaddr, *tmp_addr, *end, *veraddr, *clraddr;
int shmid, version;
Elf64_auxv_t *auxv;
adi_blksz = 0;
while(*envp++ != NULL);
for (auxv = (Elf64_auxv_t *)envp; auxv->a_type != AT_NULL; auxv++) {
switch (auxv->a_type) {
case AT_ADI_BLKSZ:
adi_blksz = auxv->a_un.a_val;
break;
case AT_ADI_NBITS:
adi_nbits = auxv->a_un.a_val;
break;
}
}
if (adi_blksz == 0) {
fprintf(stderr, "Oops! ADI is not supported\n");
exit(1);
}
printf("ADI capabilities:\n");
printf("\tBlock size = %ld\n", adi_blksz);
printf("\tNumber of bits = %ld\n", adi_nbits);
if ((shmid = shmget(2, BUFFER_SIZE,
IPC_CREAT | SHM_R | SHM_W)) < 0) {
perror("shmget failed");
exit(1);
}
shmaddr = shmat(shmid, NULL, 0);
if (shmaddr == (char *)-1) {
perror("shm attach failed");
shmctl(shmid, IPC_RMID, NULL);
exit(1);
}
if (mprotect(shmaddr, BUFFER_SIZE, PROT_READ|PROT_WRITE|PROT_ADI)) {
perror("mprotect failed");
goto err_out;
}
/* Set the ADI version tag on the shm segment
*/
version = 10;
tmp_addr = shmaddr;
end = shmaddr + BUFFER_SIZE;
while (tmp_addr < end) {
asm volatile(
"stxa %1, [%0]0x90\n\t"
:
: "r" (tmp_addr), "r" (version));
tmp_addr += adi_blksz;
}
asm volatile("membar #Sync\n\t");
/* Create a versioned address from the normal address by placing
* version tag in the upper adi_nbits bits
*/
tmp_addr = (void *) ((unsigned long)shmaddr << adi_nbits);
tmp_addr = (void *) ((unsigned long)tmp_addr >> adi_nbits);
veraddr = (void *) (((unsigned long)version << (64-adi_nbits))
| (unsigned long)tmp_addr);
printf("Starting the writes:\n");
for (i = 0; i < BUFFER_SIZE; i++) {
veraddr[i] = (char)(i);
if (!(i % (1024 * 1024)))
printf(".");
}
printf("\n");
printf("Verifying data...");
fflush(stdout);
for (i = 0; i < BUFFER_SIZE; i++)
if (veraddr[i] != (char)i)
printf("\nIndex %lu mismatched\n", i);
printf("Done.\n");
/* Disable ADI and clean up
*/
if (mprotect(shmaddr, BUFFER_SIZE, PROT_READ|PROT_WRITE)) {
perror("mprotect failed");
goto err_out;
}
if (shmdt((const void *)shmaddr) != 0)
perror("Detach failure");
shmctl(shmid, IPC_RMID, NULL);
exit(0);
err_out:
if (shmdt((const void *)shmaddr) != 0)
perror("Detach failure");
shmctl(shmid, IPC_RMID, NULL);
exit(1);
}
......@@ -43,7 +43,7 @@ static inline pgprot_t arch_vm_get_page_prot(unsigned long vm_flags)
}
#define arch_vm_get_page_prot(vm_flags) arch_vm_get_page_prot(vm_flags)
static inline bool arch_validate_prot(unsigned long prot)
static inline bool arch_validate_prot(unsigned long prot, unsigned long addr)
{
if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM | PROT_SAO))
return false;
......@@ -51,7 +51,7 @@ static inline bool arch_validate_prot(unsigned long prot)
return false;
return true;
}
#define arch_validate_prot(prot) arch_validate_prot(prot)
#define arch_validate_prot arch_validate_prot
#endif /* CONFIG_PPC64 */
#endif /* _ASM_POWERPC_MMAN_H */
......@@ -48,7 +48,7 @@ static inline long do_mmap2(unsigned long addr, size_t len,
{
long ret = -EINVAL;
if (!arch_validate_prot(prot))
if (!arch_validate_prot(prot, addr))
goto out;
if (shift) {
......
#ifndef ___ASM_SPARC_ADI_H
#define ___ASM_SPARC_ADI_H
#if defined(__sparc__) && defined(__arch64__)
#include <asm/adi_64.h>
#endif
#endif
/* adi_64.h: ADI related data structures
*
* Copyright (c) 2016 Oracle and/or its affiliates. All rights reserved.
* Author: Khalid Aziz (khalid.aziz@oracle.com)
*
* This work is licensed under the terms of the GNU GPL, version 2.
*/
#ifndef __ASM_SPARC64_ADI_H
#define __ASM_SPARC64_ADI_H
#include <linux/types.h>
#ifndef __ASSEMBLY__
struct adi_caps {
__u64 blksz;
__u64 nbits;
__u64 ue_on_adi;
};
struct adi_config {
bool enabled;
struct adi_caps caps;
};
extern struct adi_config adi_state;
extern void mdesc_adi_init(void);
static inline bool adi_capable(void)
{
return adi_state.enabled;
}
static inline unsigned long adi_blksize(void)
{
return adi_state.caps.blksz;
}
static inline unsigned long adi_nbits(void)
{
return adi_state.caps.nbits;
}
#endif /* __ASSEMBLY__ */
#endif /* !(__ASM_SPARC64_ADI_H) */
......@@ -10,6 +10,7 @@
#include <asm/processor.h>
#include <asm/extable_64.h>
#include <asm/spitfire.h>
#include <asm/adi.h>
/*
* Sparc section types
......@@ -215,9 +216,13 @@ extern unsigned int vdso_enabled;
#define ARCH_DLINFO \
do { \
extern struct adi_config adi_state; \
if (vdso_enabled) \
NEW_AUX_ENT(AT_SYSINFO_EHDR, \
(unsigned long)current->mm->context.vdso); \
NEW_AUX_ENT(AT_ADI_BLKSZ, adi_state.caps.blksz); \
NEW_AUX_ENT(AT_ADI_NBITS, adi_state.caps.nbits); \
NEW_AUX_ENT(AT_ADI_UEONADI, adi_state.caps.ue_on_adi); \
} while (0)
struct linux_binprm;
......
......@@ -570,6 +570,8 @@ struct hv_fault_status {
#define HV_FAULT_TYPE_RESV1 13
#define HV_FAULT_TYPE_UNALIGNED 14
#define HV_FAULT_TYPE_INV_PGSZ 15
#define HV_FAULT_TYPE_MCD 17
#define HV_FAULT_TYPE_MCD_DIS 18
/* Values 16 --> -2 are reserved. */
#define HV_FAULT_TYPE_MULTIPLE -1
......
......@@ -7,5 +7,87 @@
#ifndef __ASSEMBLY__
#define arch_mmap_check(addr,len,flags) sparc_mmap_check(addr,len)
int sparc_mmap_check(unsigned long addr, unsigned long len);
#endif
#ifdef CONFIG_SPARC64
#include <asm/adi_64.h>
static inline void ipi_set_tstate_mcde(void *arg)
{
struct mm_struct *mm = arg;
/* Set TSTATE_MCDE for the task using address map that ADI has been
* enabled on if the task is running. If not, it will be set
* automatically at the next context switch
*/
if (current->mm == mm) {
struct pt_regs *regs;
regs = task_pt_regs(current);
regs->tstate |= TSTATE_MCDE;
}
}
#define arch_calc_vm_prot_bits(prot, pkey) sparc_calc_vm_prot_bits(prot)
static inline unsigned long sparc_calc_vm_prot_bits(unsigned long prot)
{
if (adi_capable() && (prot & PROT_ADI)) {
struct pt_regs *regs;
if (!current->mm->context.adi) {
regs = task_pt_regs(current);
regs->tstate |= TSTATE_MCDE;
current->mm->context.adi = true;
on_each_cpu_mask(mm_cpumask(current->mm),
ipi_set_tstate_mcde, current->mm, 0);
}
return VM_SPARC_ADI;
} else {
return 0;
}
}
#define arch_vm_get_page_prot(vm_flags) sparc_vm_get_page_prot(vm_flags)
static inline pgprot_t sparc_vm_get_page_prot(unsigned long vm_flags)
{
return (vm_flags & VM_SPARC_ADI) ? __pgprot(_PAGE_MCD_4V) : __pgprot(0);
}
#define arch_validate_prot(prot, addr) sparc_validate_prot(prot, addr)
static inline int sparc_validate_prot(unsigned long prot, unsigned long addr)
{
if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM | PROT_ADI))
return 0;
if (prot & PROT_ADI) {
if (!adi_capable())
return 0;
if (addr) {
struct vm_area_struct *vma;
vma = find_vma(current->mm, addr);
if (vma) {
/* ADI can not be enabled on PFN
* mapped pages
*/
if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))
return 0;
/* Mergeable pages can become unmergeable
* if ADI is enabled on them even if they
* have identical data on them. This can be
* because ADI enabled pages with identical
* data may still not have identical ADI
* tags on them. Disallow ADI on mergeable
* pages.
*/
if (vma->vm_flags & VM_MERGEABLE)
return 0;
}
}
}
return 1;
}
#endif /* CONFIG_SPARC64 */
#endif /* __ASSEMBLY__ */
#endif /* __SPARC_MMAN_H__ */
......@@ -90,6 +90,20 @@ struct tsb_config {
#define MM_NUM_TSBS 1
#endif
/* ADI tags are stored when a page is swapped out and the storage for
* tags is allocated dynamically. There is a tag storage descriptor
* associated with each set of tag storage pages. Tag storage descriptors
* are allocated dynamically. Since kernel will allocate a full page for
* each tag storage descriptor, we can store up to
* PAGE_SIZE/sizeof(tag storage descriptor) descriptors on that page.
*/
typedef struct {
unsigned long start; /* Start address for this tag storage */
unsigned long end; /* Last address for tag storage */
unsigned char *tags; /* Where the tags are */
unsigned long tag_users; /* number of references to descriptor */
} tag_storage_desc_t;
typedef struct {
spinlock_t lock;
unsigned long sparc64_ctx_val;
......@@ -98,6 +112,9 @@ typedef struct {
struct tsb_config tsb_block[MM_NUM_TSBS];
struct hv_tsb_descr tsb_descr[MM_NUM_TSBS];
void *vdso;
bool adi;
tag_storage_desc_t *tag_store;
spinlock_t tag_lock;
} mm_context_t;
#endif /* !__ASSEMBLY__ */
......
......@@ -9,8 +9,10 @@
#include <linux/spinlock.h>
#include <linux/mm_types.h>
#include <linux/smp.h>
#include <linux/sched.h>
#include <asm/spitfire.h>
#include <asm/adi_64.h>
#include <asm-generic/mm_hooks.h>
#include <asm/percpu.h>
......@@ -136,6 +138,55 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, str
#define deactivate_mm(tsk,mm) do { } while (0)
#define activate_mm(active_mm, mm) switch_mm(active_mm, mm, NULL)
#define __HAVE_ARCH_START_CONTEXT_SWITCH
static inline void arch_start_context_switch(struct task_struct *prev)
{
/* Save the current state of MCDPER register for the process
* we are switching from
*/
if (adi_capable()) {
register unsigned long tmp_mcdper;
__asm__ __volatile__(
".word 0x83438000\n\t" /* rd %mcdper, %g1 */
"mov %%g1, %0\n\t"
: "=r" (tmp_mcdper)
:
: "g1");
if (tmp_mcdper)
set_tsk_thread_flag(prev, TIF_MCDPER);
else
clear_tsk_thread_flag(prev, TIF_MCDPER);
}
}
#define finish_arch_post_lock_switch finish_arch_post_lock_switch
static inline void finish_arch_post_lock_switch(void)
{
/* Restore the state of MCDPER register for the new process
* just switched to.
*/
if (adi_capable()) {
register unsigned long tmp_mcdper;
tmp_mcdper = test_thread_flag(TIF_MCDPER);
__asm__ __volatile__(
"mov %0, %%g1\n\t"
".word 0x9d800001\n\t" /* wr %g0, %g1, %mcdper" */
".word 0xaf902001\n\t" /* wrpr %g0, 1, %pmcdper */
:
: "ir" (tmp_mcdper)
: "g1");
if (current && current->mm && current->mm->context.adi) {
struct pt_regs *regs;
regs = task_pt_regs(current);
regs->tstate |= TSTATE_MCDE;
}
}
}
#endif /* !(__ASSEMBLY__) */
#endif /* !(__SPARC64_MMU_CONTEXT_H) */
......@@ -48,6 +48,12 @@ struct page;
void clear_user_page(void *addr, unsigned long vaddr, struct page *page);
#define copy_page(X,Y) memcpy((void *)(X), (void *)(Y), PAGE_SIZE)
void copy_user_page(void *to, void *from, unsigned long vaddr, struct page *topage);
#define __HAVE_ARCH_COPY_USER_HIGHPAGE
struct vm_area_struct;
void copy_user_highpage(struct page *to, struct page *from,
unsigned long vaddr, struct vm_area_struct *vma);
#define __HAVE_ARCH_COPY_HIGHPAGE
void copy_highpage(struct page *to, struct page *from);
/* Unlike sparc32, sparc64's parameter passing API is more
* sane in that structures which as small enough are passed
......
......@@ -19,6 +19,7 @@
#include <asm/types.h>
#include <asm/spitfire.h>
#include <asm/asi.h>
#include <asm/adi.h>
#include <asm/page.h>
#include <asm/processor.h>
......@@ -164,6 +165,8 @@ bool kern_addr_valid(unsigned long addr);
#define _PAGE_E_4V _AC(0x0000000000000800,UL) /* side-Effect */
#define _PAGE_CP_4V _AC(0x0000000000000400,UL) /* Cacheable in P-Cache */
#define _PAGE_CV_4V _AC(0x0000000000000200,UL) /* Cacheable in V-Cache */
/* Bit 9 is used to enable MCD corruption detection instead on M7 */
#define _PAGE_MCD_4V _AC(0x0000000000000200,UL) /* Memory Corruption */
#define _PAGE_P_4V _AC(0x0000000000000100,UL) /* Privileged Page */
#define _PAGE_EXEC_4V _AC(0x0000000000000080,UL) /* Executable Page */
#define _PAGE_W_4V _AC(0x0000000000000040,UL) /* Writable */
......@@ -604,6 +607,18 @@ static inline pte_t pte_mkspecial(pte_t pte)
return pte;
}
static inline pte_t pte_mkmcd(pte_t pte)
{
pte_val(pte) |= _PAGE_MCD_4V;
return pte;
}
static inline pte_t pte_mknotmcd(pte_t pte)
{
pte_val(pte) &= ~_PAGE_MCD_4V;
return pte;
}
static inline unsigned long pte_young(pte_t pte)
{
unsigned long mask;
......@@ -1046,6 +1061,39 @@ int page_in_phys_avail(unsigned long paddr);
int remap_pfn_range(struct vm_area_struct *, unsigned long, unsigned long,
unsigned long, pgprot_t);
void adi_restore_tags(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long addr, pte_t pte);
int adi_save_tags(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long addr, pte_t oldpte);
#define __HAVE_ARCH_DO_SWAP_PAGE
static inline void arch_do_swap_page(struct mm_struct *mm,
struct vm_area_struct *vma,
unsigned long addr,
pte_t pte, pte_t oldpte)
{
/* If this is a new page being mapped in, there can be no
* ADI tags stored away for this page. Skip looking for
* stored tags
*/
if (pte_none(oldpte))
return;
if (adi_state.enabled && (pte_val(pte) & _PAGE_MCD_4V))
adi_restore_tags(mm, vma, addr, pte);
}
#define __HAVE_ARCH_UNMAP_ONE
static inline int arch_unmap_one(struct mm_struct *mm,
struct vm_area_struct *vma,
unsigned long addr, pte_t oldpte)
{
if (adi_state.enabled && (pte_val(oldpte) & _PAGE_MCD_4V))
return adi_save_tags(mm, vma, addr, oldpte);
return 0;
}
static inline int io_remap_pfn_range(struct vm_area_struct *vma,
unsigned long from, unsigned long pfn,
unsigned long size, pgprot_t prot)
......
......@@ -188,7 +188,7 @@ register struct thread_info *current_thread_info_reg asm("g6");
* in using in assembly, else we can't use the mask as
* an immediate value in instructions such as andcc.
*/
/* flag bit 12 is available */
#define TIF_MCDPER 12 /* Precise MCD exception */
#define TIF_MEMDIE 13 /* is terminating due to OOM killer */
#define TIF_POLLING_NRFLAG 14
......
......@@ -76,6 +76,8 @@ extern struct sun4v_1insn_patch_entry __sun4v_1insn_patch,
__sun4v_1insn_patch_end;
extern struct sun4v_1insn_patch_entry __fast_win_ctrl_1insn_patch,
__fast_win_ctrl_1insn_patch_end;
extern struct sun4v_1insn_patch_entry __sun_m7_1insn_patch,
__sun_m7_1insn_patch_end;
struct sun4v_2insn_patch_entry {
unsigned int addr;
......
......@@ -219,6 +219,16 @@
nop; \
nop;
#define SUN4V_MCD_PRECISE \
ldxa [%g0] ASI_SCRATCHPAD, %g2; \
ldx [%g2 + HV_FAULT_D_ADDR_OFFSET], %g4; \
ldx [%g2 + HV_FAULT_D_CTX_OFFSET], %g5; \
ba,pt %xcc, etrap; \
rd %pc, %g7; \
ba,pt %xcc, sun4v_mcd_detect_precise; \
nop; \
nop;
/* Before touching these macros, you owe it to yourself to go and
* see how arch/sparc64/kernel/winfixup.S works... -DaveM
*
......
......@@ -145,6 +145,8 @@
* ASIs, "(4V)" designates SUN4V specific ASIs. "(NG4)" designates SPARC-T4
* and later ASIs.
*/
#define ASI_MCD_PRIV_PRIMARY 0x02 /* (NG7) Privileged MCD version VA */
#define ASI_MCD_REAL 0x05 /* (NG7) Privileged MCD version PA */
#define ASI_PHYS_USE_EC 0x14 /* PADDR, E-cachable */
#define ASI_PHYS_BYPASS_EC_E 0x15 /* PADDR, E-bit */
#define ASI_BLK_AIUP_4V 0x16 /* (4V) Prim, user, block ld/st */
......@@ -245,6 +247,9 @@
#define ASI_UDBL_CONTROL_R 0x7f /* External UDB control regs rd low*/
#define ASI_INTR_R 0x7f /* IRQ vector dispatch read */
#define ASI_INTR_DATAN_R 0x7f /* (III) In irq vector data reg N */
#define ASI_MCD_PRIMARY 0x90 /* (NG7) MCD version load/store */
#define ASI_MCD_ST_BLKINIT_PRIMARY \
0x92 /* (NG7) MCD store BLKINIT primary */
#define ASI_PIC 0xb0 /* (NG4) PIC registers */
#define ASI_PST8_P 0xc0 /* Primary, 8 8-bit, partial */
#define ASI_PST8_S 0xc1 /* Secondary, 8 8-bit, partial */
......
......@@ -3,6 +3,17 @@
#define AT_SYSINFO_EHDR 33
#ifdef CONFIG_SPARC64
/* Avoid overlap with other AT_* values since they are consolidated in
* glibc and any overlaps can cause problems
*/
#define AT_ADI_BLKSZ 48
#define AT_ADI_NBITS 49
#define AT_ADI_UEONADI 50
#define AT_VECTOR_SIZE_ARCH 4
#else
#define AT_VECTOR_SIZE_ARCH 1
#endif
#endif /* !(__ASMSPARC_AUXVEC_H) */
......@@ -6,6 +6,8 @@
/* SunOS'ified... */
#define PROT_ADI 0x10 /* ADI enabled */
#define MAP_RENAME MAP_ANONYMOUS /* In SunOS terminology */
#define MAP_NORESERVE 0x40 /* don't reserve swap pages */
#define MAP_INHERIT 0x80 /* SunOS doesn't do this, but... */
......
......@@ -11,7 +11,12 @@
* -----------------------------------------------------------------------
* 63 12 11 10 9 8 7 6 5 4 3 2 1 0
*/
/* IG on V9 conflicts with MCDE on M7. PSTATE_MCDE will only be used on
* processors that support ADI which do not use IG, hence there is no
* functional conflict
*/
#define PSTATE_IG _AC(0x0000000000000800,UL) /* Interrupt Globals. */
#define PSTATE_MCDE _AC(0x0000000000000800,UL) /* MCD Enable */
#define PSTATE_MG _AC(0x0000000000000400,UL) /* MMU Globals. */
#define PSTATE_CLE _AC(0x0000000000000200,UL) /* Current Little Endian.*/
#define PSTATE_TLE _AC(0x0000000000000100,UL) /* Trap Little Endian. */
......@@ -48,7 +53,12 @@
#define TSTATE_ASI _AC(0x00000000ff000000,UL) /* AddrSpace ID. */
#define TSTATE_PIL _AC(0x0000000000f00000,UL) /* %pil (Linux traps)*/
#define TSTATE_PSTATE _AC(0x00000000000fff00,UL) /* PSTATE. */
/* IG on V9 conflicts with MCDE on M7. TSTATE_MCDE will only be used on
* processors that support ADI which do not support IG, hence there is
* no functional conflict
*/
#define TSTATE_IG _AC(0x0000000000080000,UL) /* Interrupt Globals.*/
#define TSTATE_MCDE _AC(0x0000000000080000,UL) /* MCD enable. */
#define TSTATE_MG _AC(0x0000000000040000,UL) /* MMU Globals. */
#define TSTATE_CLE _AC(0x0000000000020000,UL) /* CurrLittleEndian. */
#define TSTATE_TLE _AC(0x0000000000010000,UL) /* TrapLittleEndian. */
......
......@@ -69,6 +69,7 @@ obj-$(CONFIG_SPARC64) += visemul.o
obj-$(CONFIG_SPARC64) += hvapi.o
obj-$(CONFIG_SPARC64) += sstate.o
obj-$(CONFIG_SPARC64) += mdesc.o
obj-$(CONFIG_SPARC64) += adi_64.o
obj-$(CONFIG_SPARC64) += pcr.o
obj-$(CONFIG_SPARC64) += nmi.o
obj-$(CONFIG_SPARC64_SMP) += cpumap.o
......
This diff is collapsed.
......@@ -160,6 +160,9 @@ void sun4v_resum_overflow(struct pt_regs *regs);
void sun4v_nonresum_error(struct pt_regs *regs,
unsigned long offset);
void sun4v_nonresum_overflow(struct pt_regs *regs);
void sun4v_mem_corrupt_detect_precise(struct pt_regs *regs,
unsigned long addr,
unsigned long context);
extern unsigned long sun4v_err_itlb_vaddr;
extern unsigned long sun4v_err_itlb_ctx;
......
......@@ -151,7 +151,32 @@ etrap_save: save %g2, -STACK_BIAS, %sp
stx %g6, [%sp + PTREGS_OFF + PT_V9_G6]
stx %g7, [%sp + PTREGS_OFF + PT_V9_G7]
or %l7, %l0, %l7
sethi %hi(TSTATE_TSO | TSTATE_PEF), %l0
661: sethi %hi(TSTATE_TSO | TSTATE_PEF), %l0
/* If userspace is using ADI, it could potentially pass
* a pointer with version tag embedded in it. To maintain
* the ADI security, we must enable PSTATE.mcde. Userspace
* would have already set TTE.mcd in an earlier call to
* kernel and set the version tag for the address being
* dereferenced. Setting PSTATE.mcde would ensure any
* access to userspace data through a system call honors
* ADI and does not allow a rogue app to bypass ADI by
* using system calls. Setting PSTATE.mcde only affects
* accesses to virtual addresses that have TTE.mcd set.
* Set PMCDPER to ensure any exceptions caused by ADI
* version tag mismatch are exposed before system call
* returns to userspace. Setting PMCDPER affects only
* writes to virtual addresses that have TTE.mcd set and
* have a version tag set as well.
*/
.section .sun_m7_1insn_patch, "ax"
.word 661b
sethi %hi(TSTATE_TSO | TSTATE_PEF | TSTATE_MCDE), %l0
.previous
661: nop
.section .sun_m7_1insn_patch, "ax"
.word 661b
.word 0xaf902001 /* wrpr %g0, 1, %pmcdper */
.previous
or %l7, %l0, %l7
wrpr %l2, %tnpc
wrpr %l7, (TSTATE_PRIV | TSTATE_IE), %tstate
......
......@@ -897,6 +897,7 @@ sparc64_boot_end:
#include "syscalls.S"
#include "helpers.S"
#include "sun4v_tlb_miss.S"
#include "sun4v_mcd.S"
#include "sun4v_ivec.S"
#include "ktlb.S"
#include "tsb.S"
......
......@@ -22,6 +22,7 @@
#include <linux/uaccess.h>
#include <asm/oplib.h>
#include <asm/smp.h>
#include <asm/adi.h>
/* Unlike the OBP device tree, the machine description is a full-on
* DAG. An arbitrary number of ARCs are possible from one
......@@ -1345,5 +1346,6 @@ void __init sun4v_mdesc_init(void)
cur_mdesc = hp;
mdesc_adi_init();
report_platform_properties();
}
......@@ -670,6 +670,31 @@ int copy_thread(unsigned long clone_flags, unsigned long sp,
return 0;
}
/* TIF_MCDPER in thread info flags for current task is updated lazily upon
* a context switch. Update this flag in current task's thread flags
* before dup so the dup'd task will inherit the current TIF_MCDPER flag.
*/
int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
{
if (adi_capable()) {
register unsigned long tmp_mcdper;
__asm__ __volatile__(
".word 0x83438000\n\t" /* rd %mcdper, %g1 */
"mov %%g1, %0\n\t"
: "=r" (tmp_mcdper)
:
: "g1");
if (tmp_mcdper)
set_thread_flag(TIF_MCDPER);
else
clear_thread_flag(TIF_MCDPER);
}
*dst = *src;
return 0;
}
typedef struct {
union {
unsigned int pr_regs[32];
......
......@@ -25,13 +25,31 @@
.align 32
__handle_preemption:
call SCHEDULE_USER
wrpr %g0, RTRAP_PSTATE, %pstate
661: wrpr %g0, RTRAP_PSTATE, %pstate
/* If userspace is using ADI, it could potentially pass
* a pointer with version tag embedded in it. To maintain
* the ADI security, we must re-enable PSTATE.mcde before
* we continue execution in the kernel for another thread.
*/
.section .sun_m7_1insn_patch, "ax"
.word 661b
wrpr %g0, RTRAP_PSTATE|PSTATE_MCDE, %pstate
.previous
ba,pt %xcc, __handle_preemption_continue
wrpr %g0, RTRAP_PSTATE_IRQOFF, %pstate
__handle_user_windows:
call fault_in_user_windows
wrpr %g0, RTRAP_PSTATE, %pstate
661: wrpr %g0, RTRAP_PSTATE, %pstate
/* If userspace is using ADI, it could potentially pass
* a pointer with version tag embedded in it. To maintain
* the ADI security, we must re-enable PSTATE.mcde before
* we continue execution in the kernel for another thread.
*/
.section .sun_m7_1insn_patch, "ax"
.word 661b
wrpr %g0, RTRAP_PSTATE|PSTATE_MCDE, %pstate
.previous
ba,pt %xcc, __handle_preemption_continue
wrpr %g0, RTRAP_PSTATE_IRQOFF, %pstate
......@@ -48,7 +66,16 @@ __handle_signal:
add %sp, PTREGS_OFF, %o0
mov %l0, %o2
call do_notify_resume
wrpr %g0, RTRAP_PSTATE, %pstate
661: wrpr %g0, RTRAP_PSTATE, %pstate
/* If userspace is using ADI, it could potentially pass
* a pointer with version tag embedded in it. To maintain
* the ADI security, we must re-enable PSTATE.mcde before
* we continue execution in the kernel for another thread.
*/
.section .sun_m7_1insn_patch, "ax"
.word 661b
wrpr %g0, RTRAP_PSTATE|PSTATE_MCDE, %pstate
.previous
wrpr %g0, RTRAP_PSTATE_IRQOFF, %pstate
/* Signal delivery can modify pt_regs tstate, so we must
......
......@@ -294,6 +294,8 @@ static void __init sun4v_patch(void)
case SUN4V_CHIP_SPARC_M7:
case SUN4V_CHIP_SPARC_M8:
case SUN4V_CHIP_SPARC_SN:
sun4v_patch_1insn_range(&__sun_m7_1insn_patch,
&__sun_m7_1insn_patch_end);
sun_m7_patch_2insn_range(&__sun_m7_2insn_patch,
&__sun_m7_2insn_patch_end);
break;
......
/* sun4v_mcd.S: Sun4v memory corruption detected precise exception handler
*
* Copyright (c) 2015 Oracle and/or its affiliates. All rights reserved.
* Authors: Bob Picco <bob.picco@oracle.com>,
* Khalid Aziz <khalid.aziz@oracle.com>
*
* This work is licensed under the terms of the GNU GPL, version 2.
*/
.text
.align 32
sun4v_mcd_detect_precise:
mov %l4, %o1
mov %l5, %o2
call sun4v_mem_corrupt_detect_precise
add %sp, PTREGS_OFF, %o0
ba,a,pt %xcc, rtrap
nop
......@@ -362,7 +362,6 @@ void sun4v_data_access_exception(struct pt_regs *regs, unsigned long addr, unsig
{
unsigned short type = (type_ctx >> 16);
unsigned short ctx = (type_ctx & 0xffff);
siginfo_t info;
if (notify_die(DIE_TRAP, "data access exception", regs,
0, 0x8, SIGTRAP) == NOTIFY_STOP)
......@@ -397,12 +396,29 @@ void sun4v_data_access_exception(struct pt_regs *regs, unsigned long addr, unsig
if (is_no_fault_exception(regs))
return;
info.si_signo = SIGSEGV;
info.si_errno = 0;
info.si_code = SEGV_MAPERR;
info.si_addr = (void __user *) addr;
info.si_trapno = 0;
force_sig_info(SIGSEGV, &info, current);
/* MCD (Memory Corruption Detection) disabled trap (TT=0x19) in HV
* is vectored thorugh data access exception trap with fault type
* set to HV_FAULT_TYPE_MCD_DIS. Check for MCD disabled trap.
* Accessing an address with invalid ASI for the address, for
* example setting an ADI tag on an address with ASI_MCD_PRIMARY
* when TTE.mcd is not set for the VA, is also vectored into
* kerbel by HV as data access exception with fault type set to
* HV_FAULT_TYPE_INV_ASI.
*/
switch (type) {
case HV_FAULT_TYPE_INV_ASI:
force_sig_fault(SIGILL, ILL_ILLADR, (void __user *)addr, 0,
current);
break;
case HV_FAULT_TYPE_MCD_DIS:
force_sig_fault(SIGSEGV, SEGV_ACCADI, (void __user *)addr, 0,
current);
break;
default:
force_sig_fault(SIGSEGV, SEGV_MAPERR, (void __user *)addr, 0,
current);
break;
}
}
void sun4v_data_access_exception_tl1(struct pt_regs *regs, unsigned long addr, unsigned long type_ctx)
......@@ -1847,6 +1863,7 @@ struct sun4v_error_entry {
#define SUN4V_ERR_ATTRS_ASI 0x00000080
#define SUN4V_ERR_ATTRS_PRIV_REG 0x00000100
#define SUN4V_ERR_ATTRS_SPSTATE_MSK 0x00000600
#define SUN4V_ERR_ATTRS_MCD 0x00000800
#define SUN4V_ERR_ATTRS_SPSTATE_SHFT 9
#define SUN4V_ERR_ATTRS_MODE_MSK 0x03000000
#define SUN4V_ERR_ATTRS_MODE_SHFT 24
......@@ -2044,6 +2061,50 @@ static void sun4v_log_error(struct pt_regs *regs, struct sun4v_error_entry *ent,
}
}
/* Handle memory corruption detected error which is vectored in
* through resumable error trap.
*/
void do_mcd_err(struct pt_regs *regs, struct sun4v_error_entry ent)
{
if (notify_die(DIE_TRAP, "MCD error", regs, 0, 0x34,
SIGSEGV) == NOTIFY_STOP)
return;
if (regs->tstate & TSTATE_PRIV) {
/* MCD exception could happen because the task was
* running a system call with MCD enabled and passed a
* non-versioned pointer or pointer with bad version
* tag to the system call. In such cases, hypervisor
* places the address of offending instruction in the
* resumable error report. This is a deferred error,
* so the read/write that caused the trap was potentially
* retired long time back and we may have no choice
* but to send SIGSEGV to the process.
*/
const struct exception_table_entry *entry;
entry = search_exception_tables(regs->tpc);
if (entry) {
/* Looks like a bad syscall parameter */
#ifdef DEBUG_EXCEPTIONS
pr_emerg("Exception: PC<%016lx> faddr<UNKNOWN>\n",
regs->tpc);
pr_emerg("EX_TABLE: insn<%016lx> fixup<%016lx>\n",
ent.err_raddr, entry->fixup);
#endif
regs->tpc = entry->fixup;
regs->tnpc = regs->tpc + 4;
return;
}
}
/* Send SIGSEGV to the userspace process with the right signal
* code
*/
force_sig_fault(SIGSEGV, SEGV_ADIDERR, (void __user *)ent.err_raddr,
0, current);
}
/* We run with %pil set to PIL_NORMAL_MAX and PSTATE_IE enabled in %pstate.
* Log the event and clear the first word of the entry.
*/
......@@ -2081,6 +2142,14 @@ void sun4v_resum_error(struct pt_regs *regs, unsigned long offset)
goto out;
}
/* If this is a memory corruption detected error vectored in
* by HV through resumable error trap, call the handler
*/
if (local_copy.err_attrs & SUN4V_ERR_ATTRS_MCD) {
do_mcd_err(regs, local_copy);
return;
}
sun4v_log_error(regs, &local_copy, cpu,
KERN_ERR "RESUMABLE ERROR",
&sun4v_resum_oflow_cnt);
......@@ -2656,6 +2725,53 @@ void sun4v_do_mna(struct pt_regs *regs, unsigned long addr, unsigned long type_c
force_sig_info(SIGBUS, &info, current);
}
/* sun4v_mem_corrupt_detect_precise() - Handle precise exception on an ADI
* tag mismatch.
*
* ADI version tag mismatch on a load from memory always results in a
* precise exception. Tag mismatch on a store to memory will result in
* precise exception if MCDPER or PMCDPER is set to 1.
*/
void sun4v_mem_corrupt_detect_precise(struct pt_regs *regs, unsigned long addr,
unsigned long context)
{
if (notify_die(DIE_TRAP, "memory corruption precise exception", regs,
0, 0x8, SIGSEGV) == NOTIFY_STOP)
return;
if (regs->tstate & TSTATE_PRIV) {
/* MCD exception could happen because the task was running
* a system call with MCD enabled and passed a non-versioned
* pointer or pointer with bad version tag to the system
* call.
*/
const struct exception_table_entry *entry;
entry = search_exception_tables(regs->tpc);
if (entry) {
/* Looks like a bad syscall parameter */
#ifdef DEBUG_EXCEPTIONS
pr_emerg("Exception: PC<%016lx> faddr<UNKNOWN>\n",
regs->tpc);
pr_emerg("EX_TABLE: insn<%016lx> fixup<%016lx>\n",
regs->tpc, entry->fixup);
#endif
regs->tpc = entry->fixup;
regs->tnpc = regs->tpc + 4;
return;
}
pr_emerg("%s: ADDR[%016lx] CTX[%lx], going.\n",
__func__, addr, context);
die_if_kernel("MCD precise", regs);
}
if (test_thread_flag(TIF_32BIT)) {
regs->tpc &= 0xffffffff;
regs->tnpc &= 0xffffffff;
}
force_sig_fault(SIGSEGV, SEGV_ADIPERR, (void __user *)addr, 0, current);
}
void do_privop(struct pt_regs *regs)
{
enum ctx_state prev_state = exception_enter();
......
......@@ -26,8 +26,10 @@ tl0_ill: membar #Sync
TRAP_7INSNS(do_illegal_instruction)
tl0_privop: TRAP(do_privop)
tl0_resv012: BTRAP(0x12) BTRAP(0x13) BTRAP(0x14) BTRAP(0x15) BTRAP(0x16) BTRAP(0x17)
tl0_resv018: BTRAP(0x18) BTRAP(0x19) BTRAP(0x1a) BTRAP(0x1b) BTRAP(0x1c) BTRAP(0x1d)
tl0_resv01e: BTRAP(0x1e) BTRAP(0x1f)
tl0_resv018: BTRAP(0x18) BTRAP(0x19)
tl0_mcd: SUN4V_MCD_PRECISE
tl0_resv01b: BTRAP(0x1b)
tl0_resv01c: BTRAP(0x1c) BTRAP(0x1d) BTRAP(0x1e) BTRAP(0x1f)
tl0_fpdis: TRAP_NOSAVE(do_fpdis)
tl0_fpieee: TRAP_SAVEFPU(do_fpieee)
tl0_fpother: TRAP_NOSAVE(do_fpother_check_fitos)
......
......@@ -50,7 +50,12 @@ user_rtt_fill_fixup_common:
SET_GL(0)
.previous
wrpr %g0, RTRAP_PSTATE, %pstate
661: wrpr %g0, RTRAP_PSTATE, %pstate
.section .sun_m7_1insn_patch, "ax"
.word 661b
/* Re-enable PSTATE.mcde to maintain ADI security */
wrpr %g0, RTRAP_PSTATE|PSTATE_MCDE, %pstate
.previous
mov %l1, %g6
ldx [%g6 + TI_TASK], %g4
......
......@@ -145,6 +145,11 @@ SECTIONS
*(.pause_3insn_patch)
__pause_3insn_patch_end = .;
}
.sun_m7_1insn_patch : {
__sun_m7_1insn_patch = .;
*(.sun_m7_1insn_patch)
__sun_m7_1insn_patch_end = .;
}
.sun_m7_2insn_patch : {
__sun_m7_2insn_patch = .;
*(.sun_m7_2insn_patch)
......
......@@ -12,6 +12,7 @@
#include <linux/pagemap.h>
#include <linux/rwsem.h>
#include <asm/pgtable.h>
#include <asm/adi.h>
/*
* The performance critical leaf functions are made noinline otherwise gcc
......@@ -201,6 +202,24 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,
pgd_t *pgdp;
int nr = 0;
#ifdef CONFIG_SPARC64
if (adi_capable()) {
long addr = start;
/* If userspace has passed a versioned address, kernel
* will not find it in the VMAs since it does not store
* the version tags in the list of VMAs. Storing version
* tags in list of VMAs is impractical since they can be
* changed any time from userspace without dropping into
* kernel. Any address search in VMAs will be done with
* non-versioned addresses. Ensure the ADI version bits
* are dropped here by sign extending the last bit before
* ADI bits. IOMMU does not implement version tags.
*/
addr = (addr << (long)adi_nbits()) >> (long)adi_nbits();
start = addr;
}
#endif
start &= PAGE_MASK;
addr = start;
len = (unsigned long) nr_pages << PAGE_SHIFT;
......@@ -231,6 +250,24 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write,
pgd_t *pgdp;
int nr = 0;
#ifdef CONFIG_SPARC64
if (adi_capable()) {
long addr = start;
/* If userspace has passed a versioned address, kernel
* will not find it in the VMAs since it does not store
* the version tags in the list of VMAs. Storing version
* tags in list of VMAs is impractical since they can be
* changed any time from userspace without dropping into
* kernel. Any address search in VMAs will be done with
* non-versioned addresses. Ensure the ADI version bits
* are dropped here by sign extending the last bit before
* ADI bits. IOMMU does not implements version tags,
*/
addr = (addr << (long)adi_nbits()) >> (long)adi_nbits();
start = addr;
}
#endif
start &= PAGE_MASK;
addr = start;
len = (unsigned long) nr_pages << PAGE_SHIFT;
......
......@@ -182,8 +182,20 @@ pte_t arch_make_huge_pte(pte_t entry, struct vm_area_struct *vma,
struct page *page, int writeable)
{
unsigned int shift = huge_page_shift(hstate_vma(vma));
pte_t pte;
return hugepage_shift_to_tte(entry, shift);
pte = hugepage_shift_to_tte(entry, shift);
#ifdef CONFIG_SPARC64
/* If this vma has ADI enabled on it, turn on TTE.mcd
*/
if (vma->vm_flags & VM_SPARC_ADI)
return pte_mkmcd(pte);
else
return pte_mknotmcd(pte);
#else
return pte;
#endif
}
static unsigned int sun4v_huge_tte_to_shift(pte_t entry)
......
......@@ -3160,3 +3160,72 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
do_flush_tlb_kernel_range(start, end);
}
}
void copy_user_highpage(struct page *to, struct page *from,
unsigned long vaddr, struct vm_area_struct *vma)
{
char *vfrom, *vto;
vfrom = kmap_atomic(from);
vto = kmap_atomic(to);
copy_user_page(vto, vfrom, vaddr, to);
kunmap_atomic(vto);
kunmap_atomic(vfrom);
/* If this page has ADI enabled, copy over any ADI tags
* as well
*/
if (vma->vm_flags & VM_SPARC_ADI) {
unsigned long pfrom, pto, i, adi_tag;
pfrom = page_to_phys(from);
pto = page_to_phys(to);
for (i = pfrom; i < (pfrom + PAGE_SIZE); i += adi_blksize()) {
asm volatile("ldxa [%1] %2, %0\n\t"
: "=r" (adi_tag)
: "r" (i), "i" (ASI_MCD_REAL));
asm volatile("stxa %0, [%1] %2\n\t"
:
: "r" (adi_tag), "r" (pto),
"i" (ASI_MCD_REAL));
pto += adi_blksize();
}
asm volatile("membar #Sync\n\t");
}
}
EXPORT_SYMBOL(copy_user_highpage);
void copy_highpage(struct page *to, struct page *from)
{
char *vfrom, *vto;
vfrom = kmap_atomic(from);
vto = kmap_atomic(to);
copy_page(vto, vfrom);
kunmap_atomic(vto);
kunmap_atomic(vfrom);
/* If this platform is ADI enabled, copy any ADI tags
* as well
*/
if (adi_capable()) {
unsigned long pfrom, pto, i, adi_tag;
pfrom = page_to_phys(from);
pto = page_to_phys(to);
for (i = pfrom; i < (pfrom + PAGE_SIZE); i += adi_blksize()) {
asm volatile("ldxa [%1] %2, %0\n\t"
: "=r" (adi_tag)
: "r" (i), "i" (ASI_MCD_REAL));
asm volatile("stxa %0, [%1] %2\n\t"
:
: "r" (adi_tag), "r" (pto),
"i" (ASI_MCD_REAL));
pto += adi_blksize();
}
asm volatile("membar #Sync\n\t");
}
}
EXPORT_SYMBOL(copy_highpage);
......@@ -546,6 +546,9 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)
mm->context.sparc64_ctx_val = 0UL;
mm->context.tag_store = NULL;
spin_lock_init(&mm->context.tag_lock);
#if defined(CONFIG_HUGETLB_PAGE) || defined(CONFIG_TRANSPARENT_HUGEPAGE)
/* We reset them to zero because the fork() page copying
* will re-increment the counters as the parent PTEs are
......@@ -611,4 +614,22 @@ void destroy_context(struct mm_struct *mm)
}
spin_unlock_irqrestore(&ctx_alloc_lock, flags);
/* If ADI tag storage was allocated for this task, free it */
if (mm->context.tag_store) {
tag_storage_desc_t *tag_desc;
unsigned long max_desc;
unsigned char *tags;
tag_desc = mm->context.tag_store;
max_desc = PAGE_SIZE/sizeof(tag_storage_desc_t);
for (i = 0; i < max_desc; i++) {
tags = tag_desc->tags;
tag_desc->tags = NULL;
kfree(tags);
tag_desc++;
}
kfree(mm->context.tag_store);
mm->context.tag_store = NULL;
}
}
......@@ -27,7 +27,7 @@ static inline void signal_compat_build_tests(void)
*/
BUILD_BUG_ON(NSIGILL != 11);
BUILD_BUG_ON(NSIGFPE != 13);
BUILD_BUG_ON(NSIGSEGV != 4);
BUILD_BUG_ON(NSIGSEGV != 7);
BUILD_BUG_ON(NSIGBUS != 5);
BUILD_BUG_ON(NSIGTRAP != 4);
BUILD_BUG_ON(NSIGCHLD != 6);
......
......@@ -400,6 +400,42 @@ static inline int pud_same(pud_t pud_a, pud_t pud_b)
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
#endif
#ifndef __HAVE_ARCH_DO_SWAP_PAGE
/*
* Some architectures support metadata associated with a page. When a
* page is being swapped out, this metadata must be saved so it can be
* restored when the page is swapped back in. SPARC M7 and newer
* processors support an ADI (Application Data Integrity) tag for the
* page as metadata for the page. arch_do_swap_page() can restore this
* metadata when a page is swapped back in.
*/
static inline void arch_do_swap_page(struct mm_struct *mm,
struct vm_area_struct *vma,
unsigned long addr,
pte_t pte, pte_t oldpte)
{
}
#endif
#ifndef __HAVE_ARCH_UNMAP_ONE
/*
* Some architectures support metadata associated with a page. When a
* page is being swapped out, this metadata must be saved so it can be
* restored when the page is swapped back in. SPARC M7 and newer
* processors support an ADI (Application Data Integrity) tag for the
* page as metadata for the page. arch_unmap_one() can save this
* metadata on a swap-out of a page.
*/
static inline int arch_unmap_one(struct mm_struct *mm,
struct vm_area_struct *vma,
unsigned long addr,
pte_t orig_pte)
{
return 0;
}
#endif
#ifndef __HAVE_ARCH_PGD_OFFSET_GATE
#define pgd_offset_gate(mm, addr) pgd_offset(mm, addr)
#endif
......
......@@ -237,6 +237,8 @@ static inline void copy_user_highpage(struct page *to, struct page *from,
#endif
#ifndef __HAVE_ARCH_COPY_HIGHPAGE
static inline void copy_highpage(struct page *to, struct page *from)
{
char *vfrom, *vto;
......@@ -248,4 +250,6 @@ static inline void copy_highpage(struct page *to, struct page *from)
kunmap_atomic(vfrom);
}
#endif
#endif /* _LINUX_HIGHMEM_H */
......@@ -245,6 +245,9 @@ extern unsigned int kobjsize(const void *objp);
# define VM_GROWSUP VM_ARCH_1
#elif defined(CONFIG_IA64)
# define VM_GROWSUP VM_ARCH_1
#elif defined(CONFIG_SPARC64)
# define VM_SPARC_ADI VM_ARCH_1 /* Uses ADI tag for access control */
# define VM_ARCH_CLEAR VM_SPARC_ADI
#elif !defined(CONFIG_MMU)
# define VM_MAPPED_COPY VM_ARCH_1 /* T if mapped copy of data (nommu mmap) */
#endif
......@@ -287,6 +290,12 @@ extern unsigned int kobjsize(const void *objp);
/* This mask is used to clear all the VMA flags used by mlock */
#define VM_LOCKED_CLEAR_MASK (~(VM_LOCKED | VM_LOCKONFAULT))
/* Arch-specific flags to clear when updating VM flags on protection change */
#ifndef VM_ARCH_CLEAR
# define VM_ARCH_CLEAR VM_NONE
#endif
#define VM_FLAGS_CLEAR (ARCH_VM_PKEY_FLAGS | VM_ARCH_CLEAR)
/*
* mapping from the currently active vm_flags protection bits (the
* low four bits) to a page protection mask..
......
......@@ -92,7 +92,7 @@ static inline void vm_unacct_memory(long pages)
*
* Returns true if the prot flags are valid
*/
static inline bool arch_validate_prot(unsigned long prot)
static inline bool arch_validate_prot(unsigned long prot, unsigned long addr)
{
return (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM)) == 0;
}
......
......@@ -246,7 +246,10 @@ typedef struct siginfo {
#else
# define SEGV_PKUERR 4 /* failed protection key checks */
#endif
#define NSIGSEGV 4
#define SEGV_ACCADI 5 /* ADI not enabled for mapped object */
#define SEGV_ADIDERR 6 /* Disrupting MCD error */
#define SEGV_ADIPERR 7 /* Precise MCD exception */
#define NSIGSEGV 7
/*
* SIGBUS si_codes
......
......@@ -2369,6 +2369,10 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start,
if (*vm_flags & VM_SAO)
return 0;
#endif
#ifdef VM_SPARC_ADI
if (*vm_flags & VM_SPARC_ADI)
return 0;
#endif
if (!test_bit(MMF_VM_MERGEABLE, &mm->flags)) {
err = __ksm_enter(mm);
......
......@@ -3053,6 +3053,7 @@ int do_swap_page(struct vm_fault *vmf)
if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte);
vmf->orig_pte = pte;
/* ksm created a completely new copy */
......
......@@ -417,7 +417,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len,
end = start + len;
if (end <= start)
return -ENOMEM;
if (!arch_validate_prot(prot))
if (!arch_validate_prot(prot, start))
return -EINVAL;
reqprot = prot;
......@@ -475,7 +475,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len,
* cleared from the VMA.
*/
mask_off_old_flags = VM_READ | VM_WRITE | VM_EXEC |
ARCH_VM_PKEY_FLAGS;
VM_FLAGS_CLEAR;
new_vma_pkey = arch_override_mprotect_pkey(vma, prot, pkey);
newflags = calc_vm_prot_bits(prot, new_vma_pkey);
......
......@@ -1497,6 +1497,14 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
(flags & (TTU_MIGRATION|TTU_SPLIT_FREEZE))) {
swp_entry_t entry;
pte_t swp_pte;
if (arch_unmap_one(mm, vma, address, pteval) < 0) {
set_pte_at(mm, address, pvmw.pte, pteval);
ret = false;
page_vma_mapped_walk_done(&pvmw);
break;
}
/*
* Store the pfn of the page in a special migration
* pte. do_swap_page() will wait until the migration
......@@ -1556,6 +1564,12 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
page_vma_mapped_walk_done(&pvmw);
break;
}
if (arch_unmap_one(mm, vma, address, pteval) < 0) {
set_pte_at(mm, address, pvmw.pte, pteval);
ret = false;
page_vma_mapped_walk_done(&pvmw);
break;
}
if (list_empty(&mm->mmlist)) {
spin_lock(&mmlist_lock);
if (list_empty(&mm->mmlist))
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment