Commit 76f1948a authored by Linus Torvalds's avatar Linus Torvalds

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/livepatching

Pull livepatch updates from Jiri Kosina:

 - a per-task consistency model is being added for architectures that
   support reliable stack dumping (extending this, currently rather
   trivial set, is currently in the works).

   This extends the nature of the types of patches that can be applied
   by live patching infrastructure. The code stems from the design
   proposal made [1] back in November 2014. It's a hybrid of SUSE's
   kGraft and RH's kpatch, combining advantages of both: it uses
   kGraft's per-task consistency and syscall barrier switching combined
   with kpatch's stack trace switching. There are also a number of
   fallback options which make it quite flexible.

   Most of the heavy lifting done by Josh Poimboeuf with help from
   Miroslav Benes and Petr Mladek

   [1] https://lkml.kernel.org/r/20141107140458.GA21774@suse.cz

 - module load time patch optimization from Zhou Chengming

 - a few assorted small fixes

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/livepatching:
  livepatch: add missing printk newlines
  livepatch: Cancel transition a safe way for immediate patches
  livepatch: Reduce the time of finding module symbols
  livepatch: make klp_mutex proper part of API
  livepatch: allow removal of a disabled patch
  livepatch: add /proc/<pid>/patch_state
  livepatch: change to a per-task consistency model
  livepatch: store function sizes
  livepatch: use kstrtobool() in enabled_store()
  livepatch: move patching functions into patch.c
  livepatch: remove unnecessary object loaded check
  livepatch: separate enabled and patched states
  livepatch/s390: add TIF_PATCH_PENDING thread flag
  livepatch/s390: reorganize TIF thread flag bits
  livepatch/powerpc: add TIF_PATCH_PENDING thread flag
  livepatch/x86: add TIF_PATCH_PENDING thread flag
  livepatch: create temporary klp_update_patch_state() stub
  x86/entry: define _TIF_ALLWORK_MASK flags explicitly
  stacktrace/x86: add function for detecting reliable stack traces
parents 7af4c727 a0841609
...@@ -25,6 +25,14 @@ Description: ...@@ -25,6 +25,14 @@ Description:
code is currently applied. Writing 0 will disable the patch code is currently applied. Writing 0 will disable the patch
while writing 1 will re-enable the patch. while writing 1 will re-enable the patch.
What: /sys/kernel/livepatch/<patch>/transition
Date: Feb 2017
KernelVersion: 4.12.0
Contact: live-patching@vger.kernel.org
Description:
An attribute which indicates whether the patch is currently in
transition.
What: /sys/kernel/livepatch/<patch>/<object> What: /sys/kernel/livepatch/<patch>/<object>
Date: Nov 2014 Date: Nov 2014
KernelVersion: 3.19.0 KernelVersion: 3.19.0
......
...@@ -44,6 +44,7 @@ Table of Contents ...@@ -44,6 +44,7 @@ Table of Contents
3.8 /proc/<pid>/fdinfo/<fd> - Information about opened file 3.8 /proc/<pid>/fdinfo/<fd> - Information about opened file
3.9 /proc/<pid>/map_files - Information about memory mapped files 3.9 /proc/<pid>/map_files - Information about memory mapped files
3.10 /proc/<pid>/timerslack_ns - Task timerslack value 3.10 /proc/<pid>/timerslack_ns - Task timerslack value
3.11 /proc/<pid>/patch_state - Livepatch patch operation state
4 Configuring procfs 4 Configuring procfs
4.1 Mount options 4.1 Mount options
...@@ -1887,6 +1888,23 @@ Valid values are from 0 - ULLONG_MAX ...@@ -1887,6 +1888,23 @@ Valid values are from 0 - ULLONG_MAX
An application setting the value must have PTRACE_MODE_ATTACH_FSCREDS level An application setting the value must have PTRACE_MODE_ATTACH_FSCREDS level
permissions on the task specified to change its timerslack_ns value. permissions on the task specified to change its timerslack_ns value.
3.11 /proc/<pid>/patch_state - Livepatch patch operation state
-----------------------------------------------------------------
When CONFIG_LIVEPATCH is enabled, this file displays the value of the
patch state for the task.
A value of '-1' indicates that no patch is in transition.
A value of '0' indicates that a patch is in transition and the task is
unpatched. If the patch is being enabled, then the task hasn't been
patched yet. If the patch is being disabled, then the task has already
been unpatched.
A value of '1' indicates that a patch is in transition and the task is
patched. If the patch is being enabled, then the task has already been
patched. If the patch is being disabled, then the task hasn't been
unpatched yet.
------------------------------------------------------------------------------ ------------------------------------------------------------------------------
Configuring procfs Configuring procfs
......
This diff is collapsed.
...@@ -720,6 +720,12 @@ config HAVE_STACK_VALIDATION ...@@ -720,6 +720,12 @@ config HAVE_STACK_VALIDATION
Architecture supports the 'objtool check' host tool command, which Architecture supports the 'objtool check' host tool command, which
performs compile-time stack metadata validation. performs compile-time stack metadata validation.
config HAVE_RELIABLE_STACKTRACE
bool
help
Architecture has a save_stack_trace_tsk_reliable() function which
only returns a stack trace if it can guarantee the trace is reliable.
config HAVE_ARCH_HASH config HAVE_ARCH_HASH
bool bool
default n default n
......
...@@ -92,6 +92,7 @@ static inline struct thread_info *current_thread_info(void) ...@@ -92,6 +92,7 @@ static inline struct thread_info *current_thread_info(void)
TIF_NEED_RESCHED */ TIF_NEED_RESCHED */
#define TIF_32BIT 4 /* 32 bit binary */ #define TIF_32BIT 4 /* 32 bit binary */
#define TIF_RESTORE_TM 5 /* need to restore TM FP/VEC/VSX */ #define TIF_RESTORE_TM 5 /* need to restore TM FP/VEC/VSX */
#define TIF_PATCH_PENDING 6 /* pending live patching update */
#define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */ #define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */
#define TIF_SINGLESTEP 8 /* singlestepping active */ #define TIF_SINGLESTEP 8 /* singlestepping active */
#define TIF_NOHZ 9 /* in adaptive nohz mode */ #define TIF_NOHZ 9 /* in adaptive nohz mode */
...@@ -115,6 +116,7 @@ static inline struct thread_info *current_thread_info(void) ...@@ -115,6 +116,7 @@ static inline struct thread_info *current_thread_info(void)
#define _TIF_POLLING_NRFLAG (1<<TIF_POLLING_NRFLAG) #define _TIF_POLLING_NRFLAG (1<<TIF_POLLING_NRFLAG)
#define _TIF_32BIT (1<<TIF_32BIT) #define _TIF_32BIT (1<<TIF_32BIT)
#define _TIF_RESTORE_TM (1<<TIF_RESTORE_TM) #define _TIF_RESTORE_TM (1<<TIF_RESTORE_TM)
#define _TIF_PATCH_PENDING (1<<TIF_PATCH_PENDING)
#define _TIF_SYSCALL_AUDIT (1<<TIF_SYSCALL_AUDIT) #define _TIF_SYSCALL_AUDIT (1<<TIF_SYSCALL_AUDIT)
#define _TIF_SINGLESTEP (1<<TIF_SINGLESTEP) #define _TIF_SINGLESTEP (1<<TIF_SINGLESTEP)
#define _TIF_SECCOMP (1<<TIF_SECCOMP) #define _TIF_SECCOMP (1<<TIF_SECCOMP)
...@@ -131,7 +133,7 @@ static inline struct thread_info *current_thread_info(void) ...@@ -131,7 +133,7 @@ static inline struct thread_info *current_thread_info(void)
#define _TIF_USER_WORK_MASK (_TIF_SIGPENDING | _TIF_NEED_RESCHED | \ #define _TIF_USER_WORK_MASK (_TIF_SIGPENDING | _TIF_NEED_RESCHED | \
_TIF_NOTIFY_RESUME | _TIF_UPROBE | \ _TIF_NOTIFY_RESUME | _TIF_UPROBE | \
_TIF_RESTORE_TM) _TIF_RESTORE_TM | _TIF_PATCH_PENDING)
#define _TIF_PERSYSCALL_MASK (_TIF_RESTOREALL|_TIF_NOERROR) #define _TIF_PERSYSCALL_MASK (_TIF_RESTOREALL|_TIF_NOERROR)
/* Bits in local_flags */ /* Bits in local_flags */
......
...@@ -14,6 +14,7 @@ ...@@ -14,6 +14,7 @@
#include <linux/uprobes.h> #include <linux/uprobes.h>
#include <linux/key.h> #include <linux/key.h>
#include <linux/context_tracking.h> #include <linux/context_tracking.h>
#include <linux/livepatch.h>
#include <asm/hw_breakpoint.h> #include <asm/hw_breakpoint.h>
#include <linux/uaccess.h> #include <linux/uaccess.h>
#include <asm/unistd.h> #include <asm/unistd.h>
...@@ -162,6 +163,9 @@ void do_notify_resume(struct pt_regs *regs, unsigned long thread_info_flags) ...@@ -162,6 +163,9 @@ void do_notify_resume(struct pt_regs *regs, unsigned long thread_info_flags)
tracehook_notify_resume(regs); tracehook_notify_resume(regs);
} }
if (thread_info_flags & _TIF_PATCH_PENDING)
klp_update_patch_state(current);
user_enter(); user_enter();
} }
......
...@@ -51,15 +51,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); ...@@ -51,15 +51,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
/* /*
* thread information flags bit numbers * thread information flags bit numbers
*/ */
/* _TIF_WORK bits */
#define TIF_NOTIFY_RESUME 0 /* callback before returning to user */ #define TIF_NOTIFY_RESUME 0 /* callback before returning to user */
#define TIF_SIGPENDING 1 /* signal pending */ #define TIF_SIGPENDING 1 /* signal pending */
#define TIF_NEED_RESCHED 2 /* rescheduling necessary */ #define TIF_NEED_RESCHED 2 /* rescheduling necessary */
#define TIF_UPROBE 3 /* breakpointed or single-stepping */ #define TIF_UPROBE 3 /* breakpointed or single-stepping */
#define TIF_GUARDED_STORAGE 4 /* load guarded storage control block */ #define TIF_GUARDED_STORAGE 4 /* load guarded storage control block */
#define TIF_SYSCALL_TRACE 8 /* syscall trace active */ #define TIF_PATCH_PENDING 5 /* pending live patching update */
#define TIF_SYSCALL_AUDIT 9 /* syscall auditing active */
#define TIF_SECCOMP 10 /* secure computing */
#define TIF_SYSCALL_TRACEPOINT 11 /* syscall tracepoint instrumentation */
#define TIF_31BIT 16 /* 32bit process */ #define TIF_31BIT 16 /* 32bit process */
#define TIF_MEMDIE 17 /* is terminating due to OOM killer */ #define TIF_MEMDIE 17 /* is terminating due to OOM killer */
#define TIF_RESTORE_SIGMASK 18 /* restore signal mask in do_signal() */ #define TIF_RESTORE_SIGMASK 18 /* restore signal mask in do_signal() */
...@@ -67,16 +66,25 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); ...@@ -67,16 +66,25 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
#define TIF_BLOCK_STEP 20 /* This task is block stepped */ #define TIF_BLOCK_STEP 20 /* This task is block stepped */
#define TIF_UPROBE_SINGLESTEP 21 /* This task is uprobe single stepped */ #define TIF_UPROBE_SINGLESTEP 21 /* This task is uprobe single stepped */
/* _TIF_TRACE bits */
#define TIF_SYSCALL_TRACE 24 /* syscall trace active */
#define TIF_SYSCALL_AUDIT 25 /* syscall auditing active */
#define TIF_SECCOMP 26 /* secure computing */
#define TIF_SYSCALL_TRACEPOINT 27 /* syscall tracepoint instrumentation */
#define _TIF_NOTIFY_RESUME _BITUL(TIF_NOTIFY_RESUME) #define _TIF_NOTIFY_RESUME _BITUL(TIF_NOTIFY_RESUME)
#define _TIF_SIGPENDING _BITUL(TIF_SIGPENDING) #define _TIF_SIGPENDING _BITUL(TIF_SIGPENDING)
#define _TIF_NEED_RESCHED _BITUL(TIF_NEED_RESCHED) #define _TIF_NEED_RESCHED _BITUL(TIF_NEED_RESCHED)
#define _TIF_UPROBE _BITUL(TIF_UPROBE)
#define _TIF_GUARDED_STORAGE _BITUL(TIF_GUARDED_STORAGE)
#define _TIF_PATCH_PENDING _BITUL(TIF_PATCH_PENDING)
#define _TIF_31BIT _BITUL(TIF_31BIT)
#define _TIF_SINGLE_STEP _BITUL(TIF_SINGLE_STEP)
#define _TIF_SYSCALL_TRACE _BITUL(TIF_SYSCALL_TRACE) #define _TIF_SYSCALL_TRACE _BITUL(TIF_SYSCALL_TRACE)
#define _TIF_SYSCALL_AUDIT _BITUL(TIF_SYSCALL_AUDIT) #define _TIF_SYSCALL_AUDIT _BITUL(TIF_SYSCALL_AUDIT)
#define _TIF_SECCOMP _BITUL(TIF_SECCOMP) #define _TIF_SECCOMP _BITUL(TIF_SECCOMP)
#define _TIF_SYSCALL_TRACEPOINT _BITUL(TIF_SYSCALL_TRACEPOINT) #define _TIF_SYSCALL_TRACEPOINT _BITUL(TIF_SYSCALL_TRACEPOINT)
#define _TIF_UPROBE _BITUL(TIF_UPROBE)
#define _TIF_31BIT _BITUL(TIF_31BIT)
#define _TIF_SINGLE_STEP _BITUL(TIF_SINGLE_STEP)
#define _TIF_GUARDED_STORAGE _BITUL(TIF_GUARDED_STORAGE)
#endif /* _ASM_THREAD_INFO_H */ #endif /* _ASM_THREAD_INFO_H */
...@@ -47,7 +47,7 @@ STACK_SIZE = 1 << STACK_SHIFT ...@@ -47,7 +47,7 @@ STACK_SIZE = 1 << STACK_SHIFT
STACK_INIT = STACK_SIZE - STACK_FRAME_OVERHEAD - __PT_SIZE STACK_INIT = STACK_SIZE - STACK_FRAME_OVERHEAD - __PT_SIZE
_TIF_WORK = (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_NEED_RESCHED | \ _TIF_WORK = (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_NEED_RESCHED | \
_TIF_UPROBE | _TIF_GUARDED_STORAGE) _TIF_UPROBE | _TIF_GUARDED_STORAGE | _TIF_PATCH_PENDING)
_TIF_TRACE = (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | _TIF_SECCOMP | \ _TIF_TRACE = (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | _TIF_SECCOMP | \
_TIF_SYSCALL_TRACEPOINT) _TIF_SYSCALL_TRACEPOINT)
_CIF_WORK = (_CIF_MCCK_PENDING | _CIF_ASCE_PRIMARY | \ _CIF_WORK = (_CIF_MCCK_PENDING | _CIF_ASCE_PRIMARY | \
...@@ -334,6 +334,11 @@ ENTRY(system_call) ...@@ -334,6 +334,11 @@ ENTRY(system_call)
jo .Lsysc_guarded_storage jo .Lsysc_guarded_storage
TSTMSK __PT_FLAGS(%r11),_PIF_PER_TRAP TSTMSK __PT_FLAGS(%r11),_PIF_PER_TRAP
jo .Lsysc_singlestep jo .Lsysc_singlestep
#ifdef CONFIG_LIVEPATCH
TSTMSK __TI_flags(%r12),_TIF_PATCH_PENDING
jo .Lsysc_patch_pending # handle live patching just before
# signals and possible syscall restart
#endif
TSTMSK __TI_flags(%r12),_TIF_SIGPENDING TSTMSK __TI_flags(%r12),_TIF_SIGPENDING
jo .Lsysc_sigpending jo .Lsysc_sigpending
TSTMSK __TI_flags(%r12),_TIF_NOTIFY_RESUME TSTMSK __TI_flags(%r12),_TIF_NOTIFY_RESUME
...@@ -415,6 +420,15 @@ ENTRY(system_call) ...@@ -415,6 +420,15 @@ ENTRY(system_call)
lgr %r2,%r11 # pass pointer to pt_regs lgr %r2,%r11 # pass pointer to pt_regs
larl %r14,.Lsysc_return larl %r14,.Lsysc_return
jg gs_load_bc_cb jg gs_load_bc_cb
#
# _TIF_PATCH_PENDING is set, call klp_update_patch_state
#
#ifdef CONFIG_LIVEPATCH
.Lsysc_patch_pending:
lg %r2,__LC_CURRENT # pass pointer to task struct
larl %r14,.Lsysc_return
jg klp_update_patch_state
#endif
# #
# _PIF_PER_TRAP is set, call do_per_trap # _PIF_PER_TRAP is set, call do_per_trap
...@@ -667,6 +681,10 @@ ENTRY(io_int_handler) ...@@ -667,6 +681,10 @@ ENTRY(io_int_handler)
jo .Lio_mcck_pending jo .Lio_mcck_pending
TSTMSK __TI_flags(%r12),_TIF_NEED_RESCHED TSTMSK __TI_flags(%r12),_TIF_NEED_RESCHED
jo .Lio_reschedule jo .Lio_reschedule
#ifdef CONFIG_LIVEPATCH
TSTMSK __TI_flags(%r12),_TIF_PATCH_PENDING
jo .Lio_patch_pending
#endif
TSTMSK __TI_flags(%r12),_TIF_SIGPENDING TSTMSK __TI_flags(%r12),_TIF_SIGPENDING
jo .Lio_sigpending jo .Lio_sigpending
TSTMSK __TI_flags(%r12),_TIF_NOTIFY_RESUME TSTMSK __TI_flags(%r12),_TIF_NOTIFY_RESUME
...@@ -729,6 +747,16 @@ ENTRY(io_int_handler) ...@@ -729,6 +747,16 @@ ENTRY(io_int_handler)
TRACE_IRQS_OFF TRACE_IRQS_OFF
j .Lio_return j .Lio_return
#
# _TIF_PATCH_PENDING is set, call klp_update_patch_state
#
#ifdef CONFIG_LIVEPATCH
.Lio_patch_pending:
lg %r2,__LC_CURRENT # pass pointer to task struct
larl %r14,.Lio_return
jg klp_update_patch_state
#endif
# #
# _TIF_SIGPENDING or is set, call do_signal # _TIF_SIGPENDING or is set, call do_signal
# #
......
...@@ -160,6 +160,7 @@ config X86 ...@@ -160,6 +160,7 @@ config X86
select HAVE_PERF_REGS select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP select HAVE_PERF_USER_STACK_DUMP
select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if X86_64 && FRAME_POINTER && STACK_VALIDATION
select HAVE_STACK_VALIDATION if X86_64 select HAVE_STACK_VALIDATION if X86_64
select HAVE_SYSCALL_TRACEPOINTS select HAVE_SYSCALL_TRACEPOINTS
select HAVE_UNSTABLE_SCHED_CLOCK select HAVE_UNSTABLE_SCHED_CLOCK
......
...@@ -22,6 +22,7 @@ ...@@ -22,6 +22,7 @@
#include <linux/context_tracking.h> #include <linux/context_tracking.h>
#include <linux/user-return-notifier.h> #include <linux/user-return-notifier.h>
#include <linux/uprobes.h> #include <linux/uprobes.h>
#include <linux/livepatch.h>
#include <asm/desc.h> #include <asm/desc.h>
#include <asm/traps.h> #include <asm/traps.h>
...@@ -130,14 +131,13 @@ static long syscall_trace_enter(struct pt_regs *regs) ...@@ -130,14 +131,13 @@ static long syscall_trace_enter(struct pt_regs *regs)
#define EXIT_TO_USERMODE_LOOP_FLAGS \ #define EXIT_TO_USERMODE_LOOP_FLAGS \
(_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE | \
_TIF_NEED_RESCHED | _TIF_USER_RETURN_NOTIFY) _TIF_NEED_RESCHED | _TIF_USER_RETURN_NOTIFY | _TIF_PATCH_PENDING)
static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags) static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags)
{ {
/* /*
* In order to return to user mode, we need to have IRQs off with * In order to return to user mode, we need to have IRQs off with
* none of _TIF_SIGPENDING, _TIF_NOTIFY_RESUME, _TIF_USER_RETURN_NOTIFY, * none of EXIT_TO_USERMODE_LOOP_FLAGS set. Several of these flags
* _TIF_UPROBE, or _TIF_NEED_RESCHED set. Several of these flags
* can be set at any time on preemptable kernels if we have IRQs on, * can be set at any time on preemptable kernels if we have IRQs on,
* so we need to loop. Disabling preemption wouldn't help: doing the * so we need to loop. Disabling preemption wouldn't help: doing the
* work to clear some of the flags can sleep. * work to clear some of the flags can sleep.
...@@ -164,6 +164,9 @@ static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags) ...@@ -164,6 +164,9 @@ static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags)
if (cached_flags & _TIF_USER_RETURN_NOTIFY) if (cached_flags & _TIF_USER_RETURN_NOTIFY)
fire_user_return_notifiers(); fire_user_return_notifiers();
if (cached_flags & _TIF_PATCH_PENDING)
klp_update_patch_state(current);
/* Disable IRQs and retry */ /* Disable IRQs and retry */
local_irq_disable(); local_irq_disable();
......
...@@ -73,9 +73,6 @@ struct thread_info { ...@@ -73,9 +73,6 @@ struct thread_info {
* thread information flags * thread information flags
* - these are process state flags that various assembly files * - these are process state flags that various assembly files
* may need to access * may need to access
* - pending work-to-be-done flags are in LSW
* - other flags in MSW
* Warning: layout of LSW is hardcoded in entry.S
*/ */
#define TIF_SYSCALL_TRACE 0 /* syscall trace active */ #define TIF_SYSCALL_TRACE 0 /* syscall trace active */
#define TIF_NOTIFY_RESUME 1 /* callback before returning to user */ #define TIF_NOTIFY_RESUME 1 /* callback before returning to user */
...@@ -87,6 +84,7 @@ struct thread_info { ...@@ -87,6 +84,7 @@ struct thread_info {
#define TIF_SECCOMP 8 /* secure computing */ #define TIF_SECCOMP 8 /* secure computing */
#define TIF_USER_RETURN_NOTIFY 11 /* notify kernel of userspace return */ #define TIF_USER_RETURN_NOTIFY 11 /* notify kernel of userspace return */
#define TIF_UPROBE 12 /* breakpointed or singlestepping */ #define TIF_UPROBE 12 /* breakpointed or singlestepping */
#define TIF_PATCH_PENDING 13 /* pending live patching update */
#define TIF_NOCPUID 15 /* CPUID is not accessible in userland */ #define TIF_NOCPUID 15 /* CPUID is not accessible in userland */
#define TIF_NOTSC 16 /* TSC is not accessible in userland */ #define TIF_NOTSC 16 /* TSC is not accessible in userland */
#define TIF_IA32 17 /* IA32 compatibility process */ #define TIF_IA32 17 /* IA32 compatibility process */
...@@ -104,13 +102,14 @@ struct thread_info { ...@@ -104,13 +102,14 @@ struct thread_info {
#define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE) #define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE)
#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)
#define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING)
#define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP)
#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)
#define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP)
#define _TIF_SYSCALL_EMU (1 << TIF_SYSCALL_EMU) #define _TIF_SYSCALL_EMU (1 << TIF_SYSCALL_EMU)
#define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT) #define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT)
#define _TIF_SECCOMP (1 << TIF_SECCOMP) #define _TIF_SECCOMP (1 << TIF_SECCOMP)
#define _TIF_USER_RETURN_NOTIFY (1 << TIF_USER_RETURN_NOTIFY) #define _TIF_USER_RETURN_NOTIFY (1 << TIF_USER_RETURN_NOTIFY)
#define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_UPROBE (1 << TIF_UPROBE)
#define _TIF_PATCH_PENDING (1 << TIF_PATCH_PENDING)
#define _TIF_NOCPUID (1 << TIF_NOCPUID) #define _TIF_NOCPUID (1 << TIF_NOCPUID)
#define _TIF_NOTSC (1 << TIF_NOTSC) #define _TIF_NOTSC (1 << TIF_NOTSC)
#define _TIF_IA32 (1 << TIF_IA32) #define _TIF_IA32 (1 << TIF_IA32)
...@@ -135,8 +134,10 @@ struct thread_info { ...@@ -135,8 +134,10 @@ struct thread_info {
/* work to do on any return to user space */ /* work to do on any return to user space */
#define _TIF_ALLWORK_MASK \ #define _TIF_ALLWORK_MASK \
((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT | \ (_TIF_SYSCALL_TRACE | _TIF_NOTIFY_RESUME | _TIF_SIGPENDING | \
_TIF_NOHZ) _TIF_NEED_RESCHED | _TIF_SINGLESTEP | _TIF_SYSCALL_EMU | \
_TIF_SYSCALL_AUDIT | _TIF_USER_RETURN_NOTIFY | _TIF_UPROBE | \
_TIF_PATCH_PENDING | _TIF_NOHZ | _TIF_SYSCALL_TRACEPOINT)
/* flags to check in __switch_to() */ /* flags to check in __switch_to() */
#define _TIF_WORK_CTXSW \ #define _TIF_WORK_CTXSW \
......
...@@ -11,6 +11,7 @@ struct unwind_state { ...@@ -11,6 +11,7 @@ struct unwind_state {
unsigned long stack_mask; unsigned long stack_mask;
struct task_struct *task; struct task_struct *task;
int graph_idx; int graph_idx;
bool error;
#ifdef CONFIG_FRAME_POINTER #ifdef CONFIG_FRAME_POINTER
bool got_irq; bool got_irq;
unsigned long *bp, *orig_sp; unsigned long *bp, *orig_sp;
...@@ -42,6 +43,11 @@ void unwind_start(struct unwind_state *state, struct task_struct *task, ...@@ -42,6 +43,11 @@ void unwind_start(struct unwind_state *state, struct task_struct *task,
__unwind_start(state, task, regs, first_frame); __unwind_start(state, task, regs, first_frame);
} }
static inline bool unwind_error(struct unwind_state *state)
{
return state->error;
}
#ifdef CONFIG_FRAME_POINTER #ifdef CONFIG_FRAME_POINTER
static inline static inline
......
...@@ -76,6 +76,101 @@ void save_stack_trace_tsk(struct task_struct *tsk, struct stack_trace *trace) ...@@ -76,6 +76,101 @@ void save_stack_trace_tsk(struct task_struct *tsk, struct stack_trace *trace)
} }
EXPORT_SYMBOL_GPL(save_stack_trace_tsk); EXPORT_SYMBOL_GPL(save_stack_trace_tsk);
#ifdef CONFIG_HAVE_RELIABLE_STACKTRACE
#define STACKTRACE_DUMP_ONCE(task) ({ \
static bool __section(.data.unlikely) __dumped; \
\
if (!__dumped) { \
__dumped = true; \
WARN_ON(1); \
show_stack(task, NULL); \
} \
})
static int __save_stack_trace_reliable(struct stack_trace *trace,
struct task_struct *task)
{
struct unwind_state state;
struct pt_regs *regs;
unsigned long addr;
for (unwind_start(&state, task, NULL, NULL); !unwind_done(&state);
unwind_next_frame(&state)) {
regs = unwind_get_entry_regs(&state);
if (regs) {
/*
* Kernel mode registers on the stack indicate an
* in-kernel interrupt or exception (e.g., preemption
* or a page fault), which can make frame pointers
* unreliable.
*/
if (!user_mode(regs))
return -EINVAL;
/*
* The last frame contains the user mode syscall
* pt_regs. Skip it and finish the unwind.
*/
unwind_next_frame(&state);
if (!unwind_done(&state)) {
STACKTRACE_DUMP_ONCE(task);
return -EINVAL;
}
break;
}
addr = unwind_get_return_address(&state);
/*
* A NULL or invalid return address probably means there's some
* generated code which __kernel_text_address() doesn't know
* about.
*/
if (!addr) {
STACKTRACE_DUMP_ONCE(task);
return -EINVAL;
}
if (save_stack_address(trace, addr, false))
return -EINVAL;
}
/* Check for stack corruption */
if (unwind_error(&state)) {
STACKTRACE_DUMP_ONCE(task);
return -EINVAL;
}
if (trace->nr_entries < trace->max_entries)
trace->entries[trace->nr_entries++] = ULONG_MAX;
return 0;
}
/*
* This function returns an error if it detects any unreliable features of the
* stack. Otherwise it guarantees that the stack trace is reliable.
*
* If the task is not 'current', the caller *must* ensure the task is inactive.
*/
int save_stack_trace_tsk_reliable(struct task_struct *tsk,
struct stack_trace *trace)
{
int ret;
if (!try_get_task_stack(tsk))
return -EINVAL;
ret = __save_stack_trace_reliable(trace, tsk);
put_task_stack(tsk);
return ret;
}
#endif /* CONFIG_HAVE_RELIABLE_STACKTRACE */
/* Userspace stacktrace - based on kernel/trace/trace_sysprof.c */ /* Userspace stacktrace - based on kernel/trace/trace_sysprof.c */
struct stack_frame_user { struct stack_frame_user {
...@@ -138,4 +233,3 @@ void save_stack_trace_user(struct stack_trace *trace) ...@@ -138,4 +233,3 @@ void save_stack_trace_user(struct stack_trace *trace)
if (trace->nr_entries < trace->max_entries) if (trace->nr_entries < trace->max_entries)
trace->entries[trace->nr_entries++] = ULONG_MAX; trace->entries[trace->nr_entries++] = ULONG_MAX;
} }
...@@ -283,6 +283,8 @@ bool unwind_next_frame(struct unwind_state *state) ...@@ -283,6 +283,8 @@ bool unwind_next_frame(struct unwind_state *state)
return true; return true;
bad_address: bad_address:
state->error = true;
/* /*
* When unwinding a non-current task, the task might actually be * When unwinding a non-current task, the task might actually be
* running on another CPU, in which case it could be modifying its * running on another CPU, in which case it could be modifying its
......
...@@ -2834,6 +2834,15 @@ static int proc_pid_personality(struct seq_file *m, struct pid_namespace *ns, ...@@ -2834,6 +2834,15 @@ static int proc_pid_personality(struct seq_file *m, struct pid_namespace *ns,
return err; return err;
} }
#ifdef CONFIG_LIVEPATCH
static int proc_pid_patch_state(struct seq_file *m, struct pid_namespace *ns,
struct pid *pid, struct task_struct *task)
{
seq_printf(m, "%d\n", task->patch_state);
return 0;
}
#endif /* CONFIG_LIVEPATCH */
/* /*
* Thread groups * Thread groups
*/ */
...@@ -2933,6 +2942,9 @@ static const struct pid_entry tgid_base_stuff[] = { ...@@ -2933,6 +2942,9 @@ static const struct pid_entry tgid_base_stuff[] = {
REG("timers", S_IRUGO, proc_timers_operations), REG("timers", S_IRUGO, proc_timers_operations),
#endif #endif
REG("timerslack_ns", S_IRUGO|S_IWUGO, proc_pid_set_timerslack_ns_operations), REG("timerslack_ns", S_IRUGO|S_IWUGO, proc_pid_set_timerslack_ns_operations),
#ifdef CONFIG_LIVEPATCH
ONE("patch_state", S_IRUSR, proc_pid_patch_state),
#endif
}; };
static int proc_tgid_base_readdir(struct file *file, struct dir_context *ctx) static int proc_tgid_base_readdir(struct file *file, struct dir_context *ctx)
...@@ -3315,6 +3327,9 @@ static const struct pid_entry tid_base_stuff[] = { ...@@ -3315,6 +3327,9 @@ static const struct pid_entry tid_base_stuff[] = {
REG("projid_map", S_IRUGO|S_IWUSR, proc_projid_map_operations), REG("projid_map", S_IRUGO|S_IWUSR, proc_projid_map_operations),
REG("setgroups", S_IRUGO|S_IWUSR, proc_setgroups_operations), REG("setgroups", S_IRUGO|S_IWUSR, proc_setgroups_operations),
#endif #endif
#ifdef CONFIG_LIVEPATCH
ONE("patch_state", S_IRUSR, proc_pid_patch_state),
#endif
}; };
static int proc_tid_base_readdir(struct file *file, struct dir_context *ctx) static int proc_tid_base_readdir(struct file *file, struct dir_context *ctx)
......
...@@ -15,6 +15,7 @@ ...@@ -15,6 +15,7 @@
#include <linux/sched/autogroup.h> #include <linux/sched/autogroup.h>
#include <net/net_namespace.h> #include <net/net_namespace.h>
#include <linux/sched/rt.h> #include <linux/sched/rt.h>
#include <linux/livepatch.h>
#include <linux/mm_types.h> #include <linux/mm_types.h>
#include <asm/thread_info.h> #include <asm/thread_info.h>
...@@ -203,6 +204,13 @@ extern struct cred init_cred; ...@@ -203,6 +204,13 @@ extern struct cred init_cred;
# define INIT_KASAN(tsk) # define INIT_KASAN(tsk)
#endif #endif
#ifdef CONFIG_LIVEPATCH
# define INIT_LIVEPATCH(tsk) \
.patch_state = KLP_UNDEFINED,
#else
# define INIT_LIVEPATCH(tsk)
#endif
#ifdef CONFIG_THREAD_INFO_IN_TASK #ifdef CONFIG_THREAD_INFO_IN_TASK
# define INIT_TASK_TI(tsk) \ # define INIT_TASK_TI(tsk) \
.thread_info = INIT_THREAD_INFO(tsk), \ .thread_info = INIT_THREAD_INFO(tsk), \
...@@ -289,6 +297,7 @@ extern struct cred init_cred; ...@@ -289,6 +297,7 @@ extern struct cred init_cred;
INIT_VTIME(tsk) \ INIT_VTIME(tsk) \
INIT_NUMA_BALANCING(tsk) \ INIT_NUMA_BALANCING(tsk) \
INIT_KASAN(tsk) \ INIT_KASAN(tsk) \
INIT_LIVEPATCH(tsk) \
} }
......
...@@ -23,15 +23,16 @@ ...@@ -23,15 +23,16 @@
#include <linux/module.h> #include <linux/module.h>
#include <linux/ftrace.h> #include <linux/ftrace.h>
#include <linux/completion.h>
#if IS_ENABLED(CONFIG_LIVEPATCH) #if IS_ENABLED(CONFIG_LIVEPATCH)
#include <asm/livepatch.h> #include <asm/livepatch.h>
enum klp_state { /* task patch states */
KLP_DISABLED, #define KLP_UNDEFINED -1
KLP_ENABLED #define KLP_UNPATCHED 0
}; #define KLP_PATCHED 1
/** /**
* struct klp_func - function structure for live patching * struct klp_func - function structure for live patching
...@@ -39,10 +40,29 @@ enum klp_state { ...@@ -39,10 +40,29 @@ enum klp_state {
* @new_func: pointer to the patched function code * @new_func: pointer to the patched function code
* @old_sympos: a hint indicating which symbol position the old function * @old_sympos: a hint indicating which symbol position the old function
* can be found (optional) * can be found (optional)
* @immediate: patch the func immediately, bypassing safety mechanisms
* @old_addr: the address of the function being patched * @old_addr: the address of the function being patched
* @kobj: kobject for sysfs resources * @kobj: kobject for sysfs resources
* @state: tracks function-level patch application state
* @stack_node: list node for klp_ops func_stack list * @stack_node: list node for klp_ops func_stack list
* @old_size: size of the old function
* @new_size: size of the new function
* @patched: the func has been added to the klp_ops list
* @transition: the func is currently being applied or reverted
*
* The patched and transition variables define the func's patching state. When
* patching, a func is always in one of the following states:
*
* patched=0 transition=0: unpatched
* patched=0 transition=1: unpatched, temporary starting state
* patched=1 transition=1: patched, may be visible to some tasks
* patched=1 transition=0: patched, visible to all tasks
*
* And when unpatching, it goes in the reverse order:
*
* patched=1 transition=0: patched, visible to all tasks
* patched=1 transition=1: patched, may be visible to some tasks
* patched=0 transition=1: unpatched, temporary ending state
* patched=0 transition=0: unpatched
*/ */
struct klp_func { struct klp_func {
/* external */ /* external */
...@@ -56,12 +76,15 @@ struct klp_func { ...@@ -56,12 +76,15 @@ struct klp_func {
* in kallsyms for the given object is used. * in kallsyms for the given object is used.
*/ */
unsigned long old_sympos; unsigned long old_sympos;
bool immediate;
/* internal */ /* internal */
unsigned long old_addr; unsigned long old_addr;
struct kobject kobj; struct kobject kobj;
enum klp_state state;
struct list_head stack_node; struct list_head stack_node;
unsigned long old_size, new_size;
bool patched;
bool transition;
}; };
/** /**
...@@ -71,7 +94,7 @@ struct klp_func { ...@@ -71,7 +94,7 @@ struct klp_func {
* @kobj: kobject for sysfs resources * @kobj: kobject for sysfs resources
* @mod: kernel module associated with the patched object * @mod: kernel module associated with the patched object
* (NULL for vmlinux) * (NULL for vmlinux)
* @state: tracks object-level patch application state * @patched: the object's funcs have been added to the klp_ops list
*/ */
struct klp_object { struct klp_object {
/* external */ /* external */
...@@ -81,26 +104,30 @@ struct klp_object { ...@@ -81,26 +104,30 @@ struct klp_object {
/* internal */ /* internal */
struct kobject kobj; struct kobject kobj;
struct module *mod; struct module *mod;
enum klp_state state; bool patched;
}; };
/** /**
* struct klp_patch - patch structure for live patching * struct klp_patch - patch structure for live patching
* @mod: reference to the live patch module * @mod: reference to the live patch module
* @objs: object entries for kernel objects to be patched * @objs: object entries for kernel objects to be patched
* @immediate: patch all funcs immediately, bypassing safety mechanisms
* @list: list node for global list of registered patches * @list: list node for global list of registered patches
* @kobj: kobject for sysfs resources * @kobj: kobject for sysfs resources
* @state: tracks patch-level application state * @enabled: the patch is enabled (but operation may be incomplete)
* @finish: for waiting till it is safe to remove the patch module
*/ */
struct klp_patch { struct klp_patch {
/* external */ /* external */
struct module *mod; struct module *mod;
struct klp_object *objs; struct klp_object *objs;
bool immediate;
/* internal */ /* internal */
struct list_head list; struct list_head list;
struct kobject kobj; struct kobject kobj;
enum klp_state state; bool enabled;
struct completion finish;
}; };
#define klp_for_each_object(patch, obj) \ #define klp_for_each_object(patch, obj) \
...@@ -123,10 +150,27 @@ void arch_klp_init_object_loaded(struct klp_patch *patch, ...@@ -123,10 +150,27 @@ void arch_klp_init_object_loaded(struct klp_patch *patch,
int klp_module_coming(struct module *mod); int klp_module_coming(struct module *mod);
void klp_module_going(struct module *mod); void klp_module_going(struct module *mod);
void klp_copy_process(struct task_struct *child);
void klp_update_patch_state(struct task_struct *task);
static inline bool klp_patch_pending(struct task_struct *task)
{
return test_tsk_thread_flag(task, TIF_PATCH_PENDING);
}
static inline bool klp_have_reliable_stack(void)
{
return IS_ENABLED(CONFIG_STACKTRACE) &&
IS_ENABLED(CONFIG_HAVE_RELIABLE_STACKTRACE);
}
#else /* !CONFIG_LIVEPATCH */ #else /* !CONFIG_LIVEPATCH */
static inline int klp_module_coming(struct module *mod) { return 0; } static inline int klp_module_coming(struct module *mod) { return 0; }
static inline void klp_module_going(struct module *mod) { } static inline void klp_module_going(struct module *mod) {}
static inline bool klp_patch_pending(struct task_struct *task) { return false; }
static inline void klp_update_patch_state(struct task_struct *task) {}
static inline void klp_copy_process(struct task_struct *child) {}
#endif /* CONFIG_LIVEPATCH */ #endif /* CONFIG_LIVEPATCH */
......
...@@ -1043,6 +1043,9 @@ struct task_struct { ...@@ -1043,6 +1043,9 @@ struct task_struct {
#ifdef CONFIG_THREAD_INFO_IN_TASK #ifdef CONFIG_THREAD_INFO_IN_TASK
/* A live task holds one reference: */ /* A live task holds one reference: */
atomic_t stack_refcount; atomic_t stack_refcount;
#endif
#ifdef CONFIG_LIVEPATCH
int patch_state;
#endif #endif
/* CPU-specific state of this task: */ /* CPU-specific state of this task: */
struct thread_struct thread; struct thread_struct thread;
......
...@@ -18,6 +18,8 @@ extern void save_stack_trace_regs(struct pt_regs *regs, ...@@ -18,6 +18,8 @@ extern void save_stack_trace_regs(struct pt_regs *regs,
struct stack_trace *trace); struct stack_trace *trace);
extern void save_stack_trace_tsk(struct task_struct *tsk, extern void save_stack_trace_tsk(struct task_struct *tsk,
struct stack_trace *trace); struct stack_trace *trace);
extern int save_stack_trace_tsk_reliable(struct task_struct *tsk,
struct stack_trace *trace);
extern void print_stack_trace(struct stack_trace *trace, int spaces); extern void print_stack_trace(struct stack_trace *trace, int spaces);
extern int snprint_stack_trace(char *buf, size_t size, extern int snprint_stack_trace(char *buf, size_t size,
...@@ -29,12 +31,13 @@ extern void save_stack_trace_user(struct stack_trace *trace); ...@@ -29,12 +31,13 @@ extern void save_stack_trace_user(struct stack_trace *trace);
# define save_stack_trace_user(trace) do { } while (0) # define save_stack_trace_user(trace) do { } while (0)
#endif #endif
#else #else /* !CONFIG_STACKTRACE */
# define save_stack_trace(trace) do { } while (0) # define save_stack_trace(trace) do { } while (0)
# define save_stack_trace_tsk(tsk, trace) do { } while (0) # define save_stack_trace_tsk(tsk, trace) do { } while (0)
# define save_stack_trace_user(trace) do { } while (0) # define save_stack_trace_user(trace) do { } while (0)
# define print_stack_trace(trace, spaces) do { } while (0) # define print_stack_trace(trace, spaces) do { } while (0)
# define snprint_stack_trace(buf, size, trace, spaces) do { } while (0) # define snprint_stack_trace(buf, size, trace, spaces) do { } while (0)
#endif # define save_stack_trace_tsk_reliable(tsk, trace) ({ -ENOSYS; })
#endif /* CONFIG_STACKTRACE */
#endif #endif /* __LINUX_STACKTRACE_H */
...@@ -87,6 +87,7 @@ ...@@ -87,6 +87,7 @@
#include <linux/compiler.h> #include <linux/compiler.h>
#include <linux/sysctl.h> #include <linux/sysctl.h>
#include <linux/kcov.h> #include <linux/kcov.h>
#include <linux/livepatch.h>
#include <asm/pgtable.h> #include <asm/pgtable.h>
#include <asm/pgalloc.h> #include <asm/pgalloc.h>
...@@ -1798,6 +1799,8 @@ static __latent_entropy struct task_struct *copy_process( ...@@ -1798,6 +1799,8 @@ static __latent_entropy struct task_struct *copy_process(
p->parent_exec_id = current->self_exec_id; p->parent_exec_id = current->self_exec_id;
} }
klp_copy_process(p);
spin_lock(&current->sighand->siglock); spin_lock(&current->sighand->siglock);
/* /*
......
obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-$(CONFIG_LIVEPATCH) += livepatch.o
livepatch-objs := core.o livepatch-objs := core.o patch.o transition.o
This diff is collapsed.
#ifndef _LIVEPATCH_CORE_H
#define _LIVEPATCH_CORE_H
extern struct mutex klp_mutex;
#endif /* _LIVEPATCH_CORE_H */
/*
* patch.c - livepatch patching functions
*
* Copyright (C) 2014 Seth Jennings <sjenning@redhat.com>
* Copyright (C) 2014 SUSE
* Copyright (C) 2015 Josh Poimboeuf <jpoimboe@redhat.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/livepatch.h>
#include <linux/list.h>
#include <linux/ftrace.h>
#include <linux/rculist.h>
#include <linux/slab.h>
#include <linux/bug.h>
#include <linux/printk.h>
#include "patch.h"
#include "transition.h"
static LIST_HEAD(klp_ops);
struct klp_ops *klp_find_ops(unsigned long old_addr)
{
struct klp_ops *ops;
struct klp_func *func;
list_for_each_entry(ops, &klp_ops, node) {
func = list_first_entry(&ops->func_stack, struct klp_func,
stack_node);
if (func->old_addr == old_addr)
return ops;
}
return NULL;
}
static void notrace klp_ftrace_handler(unsigned long ip,
unsigned long parent_ip,
struct ftrace_ops *fops,
struct pt_regs *regs)
{
struct klp_ops *ops;
struct klp_func *func;
int patch_state;
ops = container_of(fops, struct klp_ops, fops);
rcu_read_lock();
func = list_first_or_null_rcu(&ops->func_stack, struct klp_func,
stack_node);
/*
* func should never be NULL because preemption should be disabled here
* and unregister_ftrace_function() does the equivalent of a
* synchronize_sched() before the func_stack removal.
*/
if (WARN_ON_ONCE(!func))
goto unlock;
/*
* In the enable path, enforce the order of the ops->func_stack and
* func->transition reads. The corresponding write barrier is in
* __klp_enable_patch().
*
* (Note that this barrier technically isn't needed in the disable
* path. In the rare case where klp_update_patch_state() runs before
* this handler, its TIF_PATCH_PENDING read and this func->transition
* read need to be ordered. But klp_update_patch_state() already
* enforces that.)
*/
smp_rmb();
if (unlikely(func->transition)) {
/*
* Enforce the order of the func->transition and
* current->patch_state reads. Otherwise we could read an
* out-of-date task state and pick the wrong function. The
* corresponding write barrier is in klp_init_transition().
*/
smp_rmb();
patch_state = current->patch_state;
WARN_ON_ONCE(patch_state == KLP_UNDEFINED);
if (patch_state == KLP_UNPATCHED) {
/*
* Use the previously patched version of the function.
* If no previous patches exist, continue with the
* original function.
*/
func = list_entry_rcu(func->stack_node.next,
struct klp_func, stack_node);
if (&func->stack_node == &ops->func_stack)
goto unlock;
}
}
klp_arch_set_pc(regs, (unsigned long)func->new_func);
unlock:
rcu_read_unlock();
}
/*
* Convert a function address into the appropriate ftrace location.
*
* Usually this is just the address of the function, but on some architectures
* it's more complicated so allow them to provide a custom behaviour.
*/
#ifndef klp_get_ftrace_location
static unsigned long klp_get_ftrace_location(unsigned long faddr)
{
return faddr;
}
#endif
static void klp_unpatch_func(struct klp_func *func)
{
struct klp_ops *ops;
if (WARN_ON(!func->patched))
return;
if (WARN_ON(!func->old_addr))
return;
ops = klp_find_ops(func->old_addr);
if (WARN_ON(!ops))
return;
if (list_is_singular(&ops->func_stack)) {
unsigned long ftrace_loc;
ftrace_loc = klp_get_ftrace_location(func->old_addr);
if (WARN_ON(!ftrace_loc))
return;
WARN_ON(unregister_ftrace_function(&ops->fops));
WARN_ON(ftrace_set_filter_ip(&ops->fops, ftrace_loc, 1, 0));
list_del_rcu(&func->stack_node);
list_del(&ops->node);
kfree(ops);
} else {
list_del_rcu(&func->stack_node);
}
func->patched = false;
}
static int klp_patch_func(struct klp_func *func)
{
struct klp_ops *ops;
int ret;
if (WARN_ON(!func->old_addr))
return -EINVAL;
if (WARN_ON(func->patched))
return -EINVAL;
ops = klp_find_ops(func->old_addr);
if (!ops) {
unsigned long ftrace_loc;
ftrace_loc = klp_get_ftrace_location(func->old_addr);
if (!ftrace_loc) {
pr_err("failed to find location for function '%s'\n",
func->old_name);
return -EINVAL;
}
ops = kzalloc(sizeof(*ops), GFP_KERNEL);
if (!ops)
return -ENOMEM;
ops->fops.func = klp_ftrace_handler;
ops->fops.flags = FTRACE_OPS_FL_SAVE_REGS |
FTRACE_OPS_FL_DYNAMIC |
FTRACE_OPS_FL_IPMODIFY;
list_add(&ops->node, &klp_ops);
INIT_LIST_HEAD(&ops->func_stack);
list_add_rcu(&func->stack_node, &ops->func_stack);
ret = ftrace_set_filter_ip(&ops->fops, ftrace_loc, 0, 0);
if (ret) {
pr_err("failed to set ftrace filter for function '%s' (%d)\n",
func->old_name, ret);
goto err;
}
ret = register_ftrace_function(&ops->fops);
if (ret) {
pr_err("failed to register ftrace handler for function '%s' (%d)\n",
func->old_name, ret);
ftrace_set_filter_ip(&ops->fops, ftrace_loc, 1, 0);
goto err;
}
} else {
list_add_rcu(&func->stack_node, &ops->func_stack);
}
func->patched = true;
return 0;
err:
list_del_rcu(&func->stack_node);
list_del(&ops->node);
kfree(ops);
return ret;
}
void klp_unpatch_object(struct klp_object *obj)
{
struct klp_func *func;
klp_for_each_func(obj, func)
if (func->patched)
klp_unpatch_func(func);
obj->patched = false;
}
int klp_patch_object(struct klp_object *obj)
{
struct klp_func *func;
int ret;
if (WARN_ON(obj->patched))
return -EINVAL;
klp_for_each_func(obj, func) {
ret = klp_patch_func(func);
if (ret) {
klp_unpatch_object(obj);
return ret;
}
}
obj->patched = true;
return 0;
}
void klp_unpatch_objects(struct klp_patch *patch)
{
struct klp_object *obj;
klp_for_each_object(patch, obj)
if (obj->patched)
klp_unpatch_object(obj);
}
#ifndef _LIVEPATCH_PATCH_H
#define _LIVEPATCH_PATCH_H
#include <linux/livepatch.h>
#include <linux/list.h>
#include <linux/ftrace.h>
/**
* struct klp_ops - structure for tracking registered ftrace ops structs
*
* A single ftrace_ops is shared between all enabled replacement functions
* (klp_func structs) which have the same old_addr. This allows the switch
* between function versions to happen instantaneously by updating the klp_ops
* struct's func_stack list. The winner is the klp_func at the top of the
* func_stack (front of the list).
*
* @node: node for the global klp_ops list
* @func_stack: list head for the stack of klp_func's (active func is on top)
* @fops: registered ftrace ops struct
*/
struct klp_ops {
struct list_head node;
struct list_head func_stack;
struct ftrace_ops fops;
};
struct klp_ops *klp_find_ops(unsigned long old_addr);
int klp_patch_object(struct klp_object *obj);
void klp_unpatch_object(struct klp_object *obj);
void klp_unpatch_objects(struct klp_patch *patch);
#endif /* _LIVEPATCH_PATCH_H */
This diff is collapsed.
#ifndef _LIVEPATCH_TRANSITION_H
#define _LIVEPATCH_TRANSITION_H
#include <linux/livepatch.h>
extern struct klp_patch *klp_transition_patch;
void klp_init_transition(struct klp_patch *patch, int state);
void klp_cancel_transition(void);
void klp_start_transition(void);
void klp_try_complete_transition(void);
void klp_reverse_transition(void);
#endif /* _LIVEPATCH_TRANSITION_H */
...@@ -10,6 +10,7 @@ ...@@ -10,6 +10,7 @@
#include <linux/mm.h> #include <linux/mm.h>
#include <linux/stackprotector.h> #include <linux/stackprotector.h>
#include <linux/suspend.h> #include <linux/suspend.h>
#include <linux/livepatch.h>
#include <asm/tlb.h> #include <asm/tlb.h>
...@@ -265,6 +266,9 @@ static void do_idle(void) ...@@ -265,6 +266,9 @@ static void do_idle(void)
sched_ttwu_pending(); sched_ttwu_pending();
schedule_preempt_disabled(); schedule_preempt_disabled();
if (unlikely(klp_patch_pending(current)))
klp_update_patch_state(current);
} }
bool cpu_in_idle(unsigned long pc) bool cpu_in_idle(unsigned long pc)
......
...@@ -54,8 +54,8 @@ int snprint_stack_trace(char *buf, size_t size, ...@@ -54,8 +54,8 @@ int snprint_stack_trace(char *buf, size_t size,
EXPORT_SYMBOL_GPL(snprint_stack_trace); EXPORT_SYMBOL_GPL(snprint_stack_trace);
/* /*
* Architectures that do not implement save_stack_trace_tsk or * Architectures that do not implement save_stack_trace_*()
* save_stack_trace_regs get this weak alias and a once-per-bootup warning * get these weak aliases and once-per-bootup warnings
* (whenever this facility is utilized - for example by procfs): * (whenever this facility is utilized - for example by procfs):
*/ */
__weak void __weak void
...@@ -69,3 +69,11 @@ save_stack_trace_regs(struct pt_regs *regs, struct stack_trace *trace) ...@@ -69,3 +69,11 @@ save_stack_trace_regs(struct pt_regs *regs, struct stack_trace *trace)
{ {
WARN_ONCE(1, KERN_INFO "save_stack_trace_regs() not implemented yet.\n"); WARN_ONCE(1, KERN_INFO "save_stack_trace_regs() not implemented yet.\n");
} }
__weak int
save_stack_trace_tsk_reliable(struct task_struct *tsk,
struct stack_trace *trace)
{
WARN_ONCE(1, KERN_INFO "save_stack_tsk_reliable() not implemented yet.\n");
return -ENOSYS;
}
...@@ -17,6 +17,8 @@ ...@@ -17,6 +17,8 @@
* along with this program; if not, see <http://www.gnu.org/licenses/>. * along with this program; if not, see <http://www.gnu.org/licenses/>.
*/ */
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/module.h> #include <linux/module.h>
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/livepatch.h> #include <linux/livepatch.h>
...@@ -69,6 +71,21 @@ static int livepatch_init(void) ...@@ -69,6 +71,21 @@ static int livepatch_init(void)
{ {
int ret; int ret;
if (!klp_have_reliable_stack() && !patch.immediate) {
/*
* WARNING: Be very careful when using 'patch.immediate' in
* your patches. It's ok to use it for simple patches like
* this, but for more complex patches which change function
* semantics, locking semantics, or data structures, it may not
* be safe. Use of this option will also prevent removal of
* the patch.
*
* See Documentation/livepatch/livepatch.txt for more details.
*/
patch.immediate = true;
pr_notice("The consistency model isn't supported for your architecture. Bypassing safety mechanisms and applying the patch immediately.\n");
}
ret = klp_register_patch(&patch); ret = klp_register_patch(&patch);
if (ret) if (ret)
return ret; return ret;
...@@ -82,7 +99,6 @@ static int livepatch_init(void) ...@@ -82,7 +99,6 @@ static int livepatch_init(void)
static void livepatch_exit(void) static void livepatch_exit(void)
{ {
WARN_ON(klp_disable_patch(&patch));
WARN_ON(klp_unregister_patch(&patch)); WARN_ON(klp_unregister_patch(&patch));
} }
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment