Commit de5d1b39 authored by Linus Torvalds's avatar Linus Torvalds

Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull locking/atomics update from Thomas Gleixner:
 "The locking, atomics and memory model brains delivered:

   - A larger update to the atomics code which reworks the ordering
     barriers, consolidates the atomic primitives, provides the new
     atomic64_fetch_add_unless() primitive and cleans up the include
     hell.

   - Simplify cmpxchg() instrumentation and add instrumentation for
     xchg() and cmpxchg_double().

   - Updates to the memory model and documentation"

* 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (48 commits)
  locking/atomics: Rework ordering barriers
  locking/atomics: Instrument cmpxchg_double*()
  locking/atomics: Instrument xchg()
  locking/atomics: Simplify cmpxchg() instrumentation
  locking/atomics/x86: Reduce arch_cmpxchg64*() instrumentation
  tools/memory-model: Rename litmus tests to comply to norm7
  tools/memory-model/Documentation: Fix typo, smb->smp
  sched/Documentation: Update wake_up() & co. memory-barrier guarantees
  locking/spinlock, sched/core: Clarify requirements for smp_mb__after_spinlock()
  sched/core: Use smp_mb() in wake_woken_function()
  tools/memory-model: Add informal LKMM documentation to MAINTAINERS
  locking/atomics/Documentation: Describe atomic_set() as a write operation
  tools/memory-model: Make scripts executable
  tools/memory-model: Remove ACCESS_ONCE() from model
  tools/memory-model: Remove ACCESS_ONCE() from recipes
  locking/memory-barriers.txt/kokr: Update Korean translation to fix broken DMA vs. MMIO ordering example
  MAINTAINERS: Add Daniel Lustig as an LKMM reviewer
  tools/memory-model: Fix ISA2+pooncelock+pooncelock+pombonce name
  tools/memory-model: Add litmus test for full multicopy atomicity
  locking/refcount: Always allow checked forms
  ...
parents 1c594774 fd2efaa4
...@@ -29,7 +29,7 @@ updated by one CPU, local_t is probably more appropriate. Please see ...@@ -29,7 +29,7 @@ updated by one CPU, local_t is probably more appropriate. Please see
local_t. local_t.
The first operations to implement for atomic_t's are the initializers and The first operations to implement for atomic_t's are the initializers and
plain reads. :: plain writes. ::
#define ATOMIC_INIT(i) { (i) } #define ATOMIC_INIT(i) { (i) }
#define atomic_set(v, i) ((v)->counter = (i)) #define atomic_set(v, i) ((v)->counter = (i))
......
...@@ -2179,32 +2179,41 @@ or: ...@@ -2179,32 +2179,41 @@ or:
event_indicated = 1; event_indicated = 1;
wake_up_process(event_daemon); wake_up_process(event_daemon);
A write memory barrier is implied by wake_up() and co. if and only if they A general memory barrier is executed by wake_up() if it wakes something up.
wake something up. The barrier occurs before the task state is cleared, and so If it doesn't wake anything up then a memory barrier may or may not be
sits between the STORE to indicate the event and the STORE to set TASK_RUNNING: executed; you must not rely on it. The barrier occurs before the task state
is accessed, in particular, it sits between the STORE to indicate the event
and the STORE to set TASK_RUNNING:
CPU 1 CPU 2 CPU 1 (Sleeper) CPU 2 (Waker)
=============================== =============================== =============================== ===============================
set_current_state(); STORE event_indicated set_current_state(); STORE event_indicated
smp_store_mb(); wake_up(); smp_store_mb(); wake_up();
STORE current->state <write barrier> STORE current->state ...
<general barrier> STORE current->state <general barrier> <general barrier>
LOAD event_indicated LOAD event_indicated if ((LOAD task->state) & TASK_NORMAL)
STORE task->state
To repeat, this write memory barrier is present if and only if something where "task" is the thread being woken up and it equals CPU 1's "current".
is actually awakened. To see this, consider the following sequence of
events, where X and Y are both initially zero: To repeat, a general memory barrier is guaranteed to be executed by wake_up()
if something is actually awakened, but otherwise there is no such guarantee.
To see this, consider the following sequence of events, where X and Y are both
initially zero:
CPU 1 CPU 2 CPU 1 CPU 2
=============================== =============================== =============================== ===============================
X = 1; STORE event_indicated X = 1; Y = 1;
smp_mb(); wake_up(); smp_mb(); wake_up();
Y = 1; wait_event(wq, Y == 1); LOAD Y LOAD X
wake_up(); load from Y sees 1, no memory barrier
load from X might see 0 If a wakeup does occur, one (at least) of the two loads must see 1. If, on
the other hand, a wakeup does not occur, both loads might see 0.
In contrast, if a wakeup does occur, CPU 2's load from X would be guaranteed wake_up_process() always executes a general memory barrier. The barrier again
to see 1. occurs before the task state is accessed. In particular, if the wake_up() in
the previous snippet were replaced by a call to wake_up_process() then one of
the two loads would be guaranteed to see 1.
The available waker functions include: The available waker functions include:
...@@ -2224,6 +2233,8 @@ The available waker functions include: ...@@ -2224,6 +2233,8 @@ The available waker functions include:
wake_up_poll(); wake_up_poll();
wake_up_process(); wake_up_process();
In terms of memory ordering, these functions all provide the same guarantees of
a wake_up() (or stronger).
[!] Note that the memory barriers implied by the sleeper and the waker do _not_ [!] Note that the memory barriers implied by the sleeper and the waker do _not_
order multiple stores before the wake-up with respect to loads of those stored order multiple stores before the wake-up with respect to loads of those stored
......
...@@ -1891,22 +1891,22 @@ Mandatory 배리어들은 SMP 시스템에서도 UP 시스템에서도 SMP 효 ...@@ -1891,22 +1891,22 @@ Mandatory 배리어들은 SMP 시스템에서도 UP 시스템에서도 SMP 효
/* 소유권을 수정 */ /* 소유권을 수정 */
desc->status = DEVICE_OWN; desc->status = DEVICE_OWN;
/* MMIO 를 통해 디바이스에 공지를 하기 전에 메모리를 동기화 */
wmb();
/* 업데이트된 디스크립터의 디바이스에 공지 */ /* 업데이트된 디스크립터의 디바이스에 공지 */
writel(DESC_NOTIFY, doorbell); writel(DESC_NOTIFY, doorbell);
} }
dma_rmb() 는 디스크립터로부터 데이터를 읽어오기 전에 디바이스가 소유권을 dma_rmb() 는 디스크립터로부터 데이터를 읽어오기 전에 디바이스가 소유권을
내놓았음을 보장하게 하고, dma_wmb() 는 디바이스가 자신이 소유권을 다시 내려놓았을 것을 보장하고, dma_wmb() 는 디바이스가 자신이 소유권을 다시
가졌음을 보기 전에 디스크립터에 데이터가 쓰였음을 보장합니다. wmb() 는 가졌음을 보기 전에 디스크립터에 데이터가 쓰였을 것을 보장합니다. 참고로,
캐시 일관성이 없는 (cache incoherent) MMIO 영역에 쓰기를 시도하기 전에 writel() 을 사용하면 캐시 일관성이 있는 메모리 (cache coherent memory)
캐시 일관성이 있는 메모리 (cache coherent memory) 쓰기가 완료되었음을 쓰기가 MMIO 영역에의 쓰기 전에 완료되었을 것을 보장하므로 writel() 앞에
보장해주기 위해 필요합니다. wmb() 를 실행할 필요가 없음을 알아두시기 바랍니다. writel() 보다 비용이
저렴한 writel_relaxed() 는 이런 보장을 제공하지 않으므로 여기선 사용되지
consistent memory 에 대한 자세한 내용을 위해선 Documentation/DMA-API.txt 않아야 합니다.
문서를 참고하세요.
writel_relaxed() 와 같은 완화된 I/O 접근자들에 대한 자세한 내용을 위해서는
"커널 I/O 배리어의 효과" 섹션을, consistent memory 에 대한 자세한 내용을
위해선 Documentation/DMA-API.txt 문서를 참고하세요.
MMIO 쓰기 배리어 MMIO 쓰기 배리어
......
...@@ -8317,10 +8317,16 @@ M: Jade Alglave <j.alglave@ucl.ac.uk> ...@@ -8317,10 +8317,16 @@ M: Jade Alglave <j.alglave@ucl.ac.uk>
M: Luc Maranget <luc.maranget@inria.fr> M: Luc Maranget <luc.maranget@inria.fr>
M: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> M: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
R: Akira Yokosawa <akiyks@gmail.com> R: Akira Yokosawa <akiyks@gmail.com>
R: Daniel Lustig <dlustig@nvidia.com>
L: linux-kernel@vger.kernel.org L: linux-kernel@vger.kernel.org
L: linux-arch@vger.kernel.org
S: Supported S: Supported
T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git T: git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
F: tools/memory-model/ F: tools/memory-model/
F: Documentation/atomic_bitops.txt
F: Documentation/atomic_t.txt
F: Documentation/core-api/atomic_ops.rst
F: Documentation/core-api/refcount-vs-atomic.rst
F: Documentation/memory-barriers.txt F: Documentation/memory-barriers.txt
LINUX SECURITY MODULE (LSM) FRAMEWORK LINUX SECURITY MODULE (LSM) FRAMEWORK
......
...@@ -18,11 +18,11 @@ ...@@ -18,11 +18,11 @@
* To ensure dependency ordering is preserved for the _relaxed and * To ensure dependency ordering is preserved for the _relaxed and
* _release atomics, an smp_read_barrier_depends() is unconditionally * _release atomics, an smp_read_barrier_depends() is unconditionally
* inserted into the _relaxed variants, which are used to build the * inserted into the _relaxed variants, which are used to build the
* barriered versions. To avoid redundant back-to-back fences, we can * barriered versions. Avoid redundant back-to-back fences in the
* define the _acquire and _fence versions explicitly. * _acquire and _fence versions.
*/ */
#define __atomic_op_acquire(op, args...) op##_relaxed(args) #define __atomic_acquire_fence()
#define __atomic_op_fence __atomic_op_release #define __atomic_post_full_fence()
#define ATOMIC_INIT(i) { (i) } #define ATOMIC_INIT(i) { (i) }
#define ATOMIC64_INIT(i) { (i) } #define ATOMIC64_INIT(i) { (i) }
...@@ -206,7 +206,7 @@ ATOMIC_OPS(xor, xor) ...@@ -206,7 +206,7 @@ ATOMIC_OPS(xor, xor)
#define atomic_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic_xchg(v, new) (xchg(&((v)->counter), new))
/** /**
* __atomic_add_unless - add unless the number is a given value * atomic_fetch_add_unless - add unless the number is a given value
* @v: pointer of type atomic_t * @v: pointer of type atomic_t
* @a: the amount to add to v... * @a: the amount to add to v...
* @u: ...unless v is equal to u. * @u: ...unless v is equal to u.
...@@ -214,7 +214,7 @@ ATOMIC_OPS(xor, xor) ...@@ -214,7 +214,7 @@ ATOMIC_OPS(xor, xor)
* Atomically adds @a to @v, so long as it was not @u. * Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v. * Returns the old value of @v.
*/ */
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u) static __inline__ int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{ {
int c, new, old; int c, new, old;
smp_mb(); smp_mb();
...@@ -235,38 +235,39 @@ static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u) ...@@ -235,38 +235,39 @@ static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
smp_mb(); smp_mb();
return old; return old;
} }
#define atomic_fetch_add_unless atomic_fetch_add_unless
/** /**
* atomic64_add_unless - add unless the number is a given value * atomic64_fetch_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t * @v: pointer of type atomic64_t
* @a: the amount to add to v... * @a: the amount to add to v...
* @u: ...unless v is equal to u. * @u: ...unless v is equal to u.
* *
* Atomically adds @a to @v, so long as it was not @u. * Atomically adds @a to @v, so long as it was not @u.
* Returns true iff @v was not @u. * Returns the old value of @v.
*/ */
static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u) static __inline__ long atomic64_fetch_add_unless(atomic64_t *v, long a, long u)
{ {
long c, tmp; long c, new, old;
smp_mb(); smp_mb();
__asm__ __volatile__( __asm__ __volatile__(
"1: ldq_l %[tmp],%[mem]\n" "1: ldq_l %[old],%[mem]\n"
" cmpeq %[tmp],%[u],%[c]\n" " cmpeq %[old],%[u],%[c]\n"
" addq %[tmp],%[a],%[tmp]\n" " addq %[old],%[a],%[new]\n"
" bne %[c],2f\n" " bne %[c],2f\n"
" stq_c %[tmp],%[mem]\n" " stq_c %[new],%[mem]\n"
" beq %[tmp],3f\n" " beq %[new],3f\n"
"2:\n" "2:\n"
".subsection 2\n" ".subsection 2\n"
"3: br 1b\n" "3: br 1b\n"
".previous" ".previous"
: [tmp] "=&r"(tmp), [c] "=&r"(c) : [old] "=&r"(old), [new] "=&r"(new), [c] "=&r"(c)
: [mem] "m"(*v), [a] "rI"(a), [u] "rI"(u) : [mem] "m"(*v), [a] "rI"(a), [u] "rI"(u)
: "memory"); : "memory");
smp_mb(); smp_mb();
return !c; return old;
} }
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
/* /*
* atomic64_dec_if_positive - decrement by 1 if old value positive * atomic64_dec_if_positive - decrement by 1 if old value positive
...@@ -295,31 +296,6 @@ static inline long atomic64_dec_if_positive(atomic64_t *v) ...@@ -295,31 +296,6 @@ static inline long atomic64_dec_if_positive(atomic64_t *v)
smp_mb(); smp_mb();
return old - 1; return old - 1;
} }
#define atomic64_dec_if_positive atomic64_dec_if_positive
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
#define atomic_dec_return(v) atomic_sub_return(1,(v))
#define atomic64_dec_return(v) atomic64_sub_return(1,(v))
#define atomic_inc_return(v) atomic_add_return(1,(v))
#define atomic64_inc_return(v) atomic64_add_return(1,(v))
#define atomic_sub_and_test(i,v) (atomic_sub_return((i), (v)) == 0)
#define atomic64_sub_and_test(i,v) (atomic64_sub_return((i), (v)) == 0)
#define atomic_inc_and_test(v) (atomic_add_return(1, (v)) == 0)
#define atomic64_inc_and_test(v) (atomic64_add_return(1, (v)) == 0)
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
#define atomic64_dec_and_test(v) (atomic64_sub_return(1, (v)) == 0)
#define atomic_inc(v) atomic_add(1,(v))
#define atomic64_inc(v) atomic64_add(1,(v))
#define atomic_dec(v) atomic_sub(1,(v))
#define atomic64_dec(v) atomic64_sub(1,(v))
#endif /* _ALPHA_ATOMIC_H */ #endif /* _ALPHA_ATOMIC_H */
...@@ -188,6 +188,7 @@ ATOMIC_OPS(add, +=, add) ...@@ -188,6 +188,7 @@ ATOMIC_OPS(add, +=, add)
ATOMIC_OPS(sub, -=, sub) ATOMIC_OPS(sub, -=, sub)
#define atomic_andnot atomic_andnot #define atomic_andnot atomic_andnot
#define atomic_fetch_andnot atomic_fetch_andnot
#undef ATOMIC_OPS #undef ATOMIC_OPS
#define ATOMIC_OPS(op, c_op, asm_op) \ #define ATOMIC_OPS(op, c_op, asm_op) \
...@@ -296,8 +297,6 @@ ATOMIC_OPS(add, +=, CTOP_INST_AADD_DI_R2_R2_R3) ...@@ -296,8 +297,6 @@ ATOMIC_OPS(add, +=, CTOP_INST_AADD_DI_R2_R2_R3)
ATOMIC_FETCH_OP(op, c_op, asm_op) ATOMIC_FETCH_OP(op, c_op, asm_op)
ATOMIC_OPS(and, &=, CTOP_INST_AAND_DI_R2_R2_R3) ATOMIC_OPS(and, &=, CTOP_INST_AAND_DI_R2_R2_R3)
#define atomic_andnot(mask, v) atomic_and(~(mask), (v))
#define atomic_fetch_andnot(mask, v) atomic_fetch_and(~(mask), (v))
ATOMIC_OPS(or, |=, CTOP_INST_AOR_DI_R2_R2_R3) ATOMIC_OPS(or, |=, CTOP_INST_AOR_DI_R2_R2_R3)
ATOMIC_OPS(xor, ^=, CTOP_INST_AXOR_DI_R2_R2_R3) ATOMIC_OPS(xor, ^=, CTOP_INST_AXOR_DI_R2_R2_R3)
...@@ -308,48 +307,6 @@ ATOMIC_OPS(xor, ^=, CTOP_INST_AXOR_DI_R2_R2_R3) ...@@ -308,48 +307,6 @@ ATOMIC_OPS(xor, ^=, CTOP_INST_AXOR_DI_R2_R2_R3)
#undef ATOMIC_OP_RETURN #undef ATOMIC_OP_RETURN
#undef ATOMIC_OP #undef ATOMIC_OP
/**
* __atomic_add_unless - add unless the number is a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v
*/
#define __atomic_add_unless(v, a, u) \
({ \
int c, old; \
\
/* \
* Explicit full memory barrier needed before/after as \
* LLOCK/SCOND thmeselves don't provide any such semantics \
*/ \
smp_mb(); \
\
c = atomic_read(v); \
while (c != (u) && (old = atomic_cmpxchg((v), c, c + (a))) != c)\
c = old; \
\
smp_mb(); \
\
c; \
})
#define atomic_inc_not_zero(v) atomic_add_unless((v), 1, 0)
#define atomic_inc(v) atomic_add(1, v)
#define atomic_dec(v) atomic_sub(1, v)
#define atomic_inc_and_test(v) (atomic_add_return(1, v) == 0)
#define atomic_dec_and_test(v) (atomic_sub_return(1, v) == 0)
#define atomic_inc_return(v) atomic_add_return(1, (v))
#define atomic_dec_return(v) atomic_sub_return(1, (v))
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
#define atomic_add_negative(i, v) (atomic_add_return(i, v) < 0)
#ifdef CONFIG_GENERIC_ATOMIC64 #ifdef CONFIG_GENERIC_ATOMIC64
#include <asm-generic/atomic64.h> #include <asm-generic/atomic64.h>
...@@ -473,6 +430,7 @@ static inline long long atomic64_fetch_##op(long long a, atomic64_t *v) \ ...@@ -473,6 +430,7 @@ static inline long long atomic64_fetch_##op(long long a, atomic64_t *v) \
ATOMIC64_FETCH_OP(op, op1, op2) ATOMIC64_FETCH_OP(op, op1, op2)
#define atomic64_andnot atomic64_andnot #define atomic64_andnot atomic64_andnot
#define atomic64_fetch_andnot atomic64_fetch_andnot
ATOMIC64_OPS(add, add.f, adc) ATOMIC64_OPS(add, add.f, adc)
ATOMIC64_OPS(sub, sub.f, sbc) ATOMIC64_OPS(sub, sub.f, sbc)
...@@ -559,53 +517,43 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v) ...@@ -559,53 +517,43 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
return val; return val;
} }
#define atomic64_dec_if_positive atomic64_dec_if_positive
/** /**
* atomic64_add_unless - add unless the number is a given value * atomic64_fetch_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t * @v: pointer of type atomic64_t
* @a: the amount to add to v... * @a: the amount to add to v...
* @u: ...unless v is equal to u. * @u: ...unless v is equal to u.
* *
* if (v != u) { v += a; ret = 1} else {ret = 0} * Atomically adds @a to @v, if it was not @u.
* Returns 1 iff @v was not @u (i.e. if add actually happened) * Returns the old value of @v
*/ */
static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) static inline long long atomic64_fetch_add_unless(atomic64_t *v, long long a,
long long u)
{ {
long long val; long long old, temp;
int op_done;
smp_mb(); smp_mb();
__asm__ __volatile__( __asm__ __volatile__(
"1: llockd %0, [%2] \n" "1: llockd %0, [%2] \n"
" mov %1, 1 \n"
" brne %L0, %L4, 2f # continue to add since v != u \n" " brne %L0, %L4, 2f # continue to add since v != u \n"
" breq.d %H0, %H4, 3f # return since v == u \n" " breq.d %H0, %H4, 3f # return since v == u \n"
" mov %1, 0 \n"
"2: \n" "2: \n"
" add.f %L0, %L0, %L3 \n" " add.f %L1, %L0, %L3 \n"
" adc %H0, %H0, %H3 \n" " adc %H1, %H0, %H3 \n"
" scondd %0, [%2] \n" " scondd %1, [%2] \n"
" bnz 1b \n" " bnz 1b \n"
"3: \n" "3: \n"
: "=&r"(val), "=&r" (op_done) : "=&r"(old), "=&r" (temp)
: "r"(&v->counter), "r"(a), "r"(u) : "r"(&v->counter), "r"(a), "r"(u)
: "cc"); /* memory clobber comes from smp_mb() */ : "cc"); /* memory clobber comes from smp_mb() */
smp_mb(); smp_mb();
return op_done; return old;
} }
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
#define atomic64_inc(v) atomic64_add(1LL, (v))
#define atomic64_inc_return(v) atomic64_add_return(1LL, (v))
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
#define atomic64_sub_and_test(a, v) (atomic64_sub_return((a), (v)) == 0)
#define atomic64_dec(v) atomic64_sub(1LL, (v))
#define atomic64_dec_return(v) atomic64_sub_return(1LL, (v))
#define atomic64_dec_and_test(v) (atomic64_dec_return((v)) == 0)
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1LL, 0LL)
#endif /* !CONFIG_GENERIC_ATOMIC64 */ #endif /* !CONFIG_GENERIC_ATOMIC64 */
......
...@@ -130,7 +130,7 @@ static inline int atomic_cmpxchg_relaxed(atomic_t *ptr, int old, int new) ...@@ -130,7 +130,7 @@ static inline int atomic_cmpxchg_relaxed(atomic_t *ptr, int old, int new)
} }
#define atomic_cmpxchg_relaxed atomic_cmpxchg_relaxed #define atomic_cmpxchg_relaxed atomic_cmpxchg_relaxed
static inline int __atomic_add_unless(atomic_t *v, int a, int u) static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{ {
int oldval, newval; int oldval, newval;
unsigned long tmp; unsigned long tmp;
...@@ -156,6 +156,7 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u) ...@@ -156,6 +156,7 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u)
return oldval; return oldval;
} }
#define atomic_fetch_add_unless atomic_fetch_add_unless
#else /* ARM_ARCH_6 */ #else /* ARM_ARCH_6 */
...@@ -215,15 +216,7 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new) ...@@ -215,15 +216,7 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new)
return ret; return ret;
} }
static inline int __atomic_add_unless(atomic_t *v, int a, int u) #define atomic_fetch_andnot atomic_fetch_andnot
{
int c, old;
c = atomic_read(v);
while (c != u && (old = atomic_cmpxchg((v), c, c + a)) != c)
c = old;
return c;
}
#endif /* __LINUX_ARM_ARCH__ */ #endif /* __LINUX_ARM_ARCH__ */
...@@ -254,17 +247,6 @@ ATOMIC_OPS(xor, ^=, eor) ...@@ -254,17 +247,6 @@ ATOMIC_OPS(xor, ^=, eor)
#define atomic_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic_xchg(v, new) (xchg(&((v)->counter), new))
#define atomic_inc(v) atomic_add(1, v)
#define atomic_dec(v) atomic_sub(1, v)
#define atomic_inc_and_test(v) (atomic_add_return(1, v) == 0)
#define atomic_dec_and_test(v) (atomic_sub_return(1, v) == 0)
#define atomic_inc_return_relaxed(v) (atomic_add_return_relaxed(1, v))
#define atomic_dec_return_relaxed(v) (atomic_sub_return_relaxed(1, v))
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
#define atomic_add_negative(i,v) (atomic_add_return(i, v) < 0)
#ifndef CONFIG_GENERIC_ATOMIC64 #ifndef CONFIG_GENERIC_ATOMIC64
typedef struct { typedef struct {
long long counter; long long counter;
...@@ -494,12 +476,13 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v) ...@@ -494,12 +476,13 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
return result; return result;
} }
#define atomic64_dec_if_positive atomic64_dec_if_positive
static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) static inline long long atomic64_fetch_add_unless(atomic64_t *v, long long a,
long long u)
{ {
long long val; long long oldval, newval;
unsigned long tmp; unsigned long tmp;
int ret = 1;
smp_mb(); smp_mb();
prefetchw(&v->counter); prefetchw(&v->counter);
...@@ -508,33 +491,23 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) ...@@ -508,33 +491,23 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
"1: ldrexd %0, %H0, [%4]\n" "1: ldrexd %0, %H0, [%4]\n"
" teq %0, %5\n" " teq %0, %5\n"
" teqeq %H0, %H5\n" " teqeq %H0, %H5\n"
" moveq %1, #0\n"
" beq 2f\n" " beq 2f\n"
" adds %Q0, %Q0, %Q6\n" " adds %Q1, %Q0, %Q6\n"
" adc %R0, %R0, %R6\n" " adc %R1, %R0, %R6\n"
" strexd %2, %0, %H0, [%4]\n" " strexd %2, %1, %H1, [%4]\n"
" teq %2, #0\n" " teq %2, #0\n"
" bne 1b\n" " bne 1b\n"
"2:" "2:"
: "=&r" (val), "+r" (ret), "=&r" (tmp), "+Qo" (v->counter) : "=&r" (oldval), "=&r" (newval), "=&r" (tmp), "+Qo" (v->counter)
: "r" (&v->counter), "r" (u), "r" (a) : "r" (&v->counter), "r" (u), "r" (a)
: "cc"); : "cc");
if (ret) if (oldval != u)
smp_mb(); smp_mb();
return ret; return oldval;
} }
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
#define atomic64_inc(v) atomic64_add(1LL, (v))
#define atomic64_inc_return_relaxed(v) atomic64_add_return_relaxed(1LL, (v))
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
#define atomic64_sub_and_test(a, v) (atomic64_sub_return((a), (v)) == 0)
#define atomic64_dec(v) atomic64_sub(1LL, (v))
#define atomic64_dec_return_relaxed(v) atomic64_sub_return_relaxed(1LL, (v))
#define atomic64_dec_and_test(v) (atomic64_dec_return((v)) == 0)
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1LL, 0LL)
#endif /* !CONFIG_GENERIC_ATOMIC64 */ #endif /* !CONFIG_GENERIC_ATOMIC64 */
#endif #endif
......
...@@ -40,17 +40,6 @@ ...@@ -40,17 +40,6 @@
#include <asm/cmpxchg.h> #include <asm/cmpxchg.h>
#define ___atomic_add_unless(v, a, u, sfx) \
({ \
typeof((v)->counter) c, old; \
\
c = atomic##sfx##_read(v); \
while (c != (u) && \
(old = atomic##sfx##_cmpxchg((v), c, c + (a))) != c) \
c = old; \
c; \
})
#define ATOMIC_INIT(i) { (i) } #define ATOMIC_INIT(i) { (i) }
#define atomic_read(v) READ_ONCE((v)->counter) #define atomic_read(v) READ_ONCE((v)->counter)
...@@ -61,21 +50,11 @@ ...@@ -61,21 +50,11 @@
#define atomic_add_return_release atomic_add_return_release #define atomic_add_return_release atomic_add_return_release
#define atomic_add_return atomic_add_return #define atomic_add_return atomic_add_return
#define atomic_inc_return_relaxed(v) atomic_add_return_relaxed(1, (v))
#define atomic_inc_return_acquire(v) atomic_add_return_acquire(1, (v))
#define atomic_inc_return_release(v) atomic_add_return_release(1, (v))
#define atomic_inc_return(v) atomic_add_return(1, (v))
#define atomic_sub_return_relaxed atomic_sub_return_relaxed #define atomic_sub_return_relaxed atomic_sub_return_relaxed
#define atomic_sub_return_acquire atomic_sub_return_acquire #define atomic_sub_return_acquire atomic_sub_return_acquire
#define atomic_sub_return_release atomic_sub_return_release #define atomic_sub_return_release atomic_sub_return_release
#define atomic_sub_return atomic_sub_return #define atomic_sub_return atomic_sub_return
#define atomic_dec_return_relaxed(v) atomic_sub_return_relaxed(1, (v))
#define atomic_dec_return_acquire(v) atomic_sub_return_acquire(1, (v))
#define atomic_dec_return_release(v) atomic_sub_return_release(1, (v))
#define atomic_dec_return(v) atomic_sub_return(1, (v))
#define atomic_fetch_add_relaxed atomic_fetch_add_relaxed #define atomic_fetch_add_relaxed atomic_fetch_add_relaxed
#define atomic_fetch_add_acquire atomic_fetch_add_acquire #define atomic_fetch_add_acquire atomic_fetch_add_acquire
#define atomic_fetch_add_release atomic_fetch_add_release #define atomic_fetch_add_release atomic_fetch_add_release
...@@ -119,13 +98,6 @@ ...@@ -119,13 +98,6 @@
cmpxchg_release(&((v)->counter), (old), (new)) cmpxchg_release(&((v)->counter), (old), (new))
#define atomic_cmpxchg(v, old, new) cmpxchg(&((v)->counter), (old), (new)) #define atomic_cmpxchg(v, old, new) cmpxchg(&((v)->counter), (old), (new))
#define atomic_inc(v) atomic_add(1, (v))
#define atomic_dec(v) atomic_sub(1, (v))
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
#define atomic_sub_and_test(i, v) (atomic_sub_return((i), (v)) == 0)
#define atomic_add_negative(i, v) (atomic_add_return((i), (v)) < 0)
#define __atomic_add_unless(v, a, u) ___atomic_add_unless(v, a, u,)
#define atomic_andnot atomic_andnot #define atomic_andnot atomic_andnot
/* /*
...@@ -140,21 +112,11 @@ ...@@ -140,21 +112,11 @@
#define atomic64_add_return_release atomic64_add_return_release #define atomic64_add_return_release atomic64_add_return_release
#define atomic64_add_return atomic64_add_return #define atomic64_add_return atomic64_add_return
#define atomic64_inc_return_relaxed(v) atomic64_add_return_relaxed(1, (v))
#define atomic64_inc_return_acquire(v) atomic64_add_return_acquire(1, (v))
#define atomic64_inc_return_release(v) atomic64_add_return_release(1, (v))
#define atomic64_inc_return(v) atomic64_add_return(1, (v))
#define atomic64_sub_return_relaxed atomic64_sub_return_relaxed #define atomic64_sub_return_relaxed atomic64_sub_return_relaxed
#define atomic64_sub_return_acquire atomic64_sub_return_acquire #define atomic64_sub_return_acquire atomic64_sub_return_acquire
#define atomic64_sub_return_release atomic64_sub_return_release #define atomic64_sub_return_release atomic64_sub_return_release
#define atomic64_sub_return atomic64_sub_return #define atomic64_sub_return atomic64_sub_return
#define atomic64_dec_return_relaxed(v) atomic64_sub_return_relaxed(1, (v))
#define atomic64_dec_return_acquire(v) atomic64_sub_return_acquire(1, (v))
#define atomic64_dec_return_release(v) atomic64_sub_return_release(1, (v))
#define atomic64_dec_return(v) atomic64_sub_return(1, (v))
#define atomic64_fetch_add_relaxed atomic64_fetch_add_relaxed #define atomic64_fetch_add_relaxed atomic64_fetch_add_relaxed
#define atomic64_fetch_add_acquire atomic64_fetch_add_acquire #define atomic64_fetch_add_acquire atomic64_fetch_add_acquire
#define atomic64_fetch_add_release atomic64_fetch_add_release #define atomic64_fetch_add_release atomic64_fetch_add_release
...@@ -195,16 +157,9 @@ ...@@ -195,16 +157,9 @@
#define atomic64_cmpxchg_release atomic_cmpxchg_release #define atomic64_cmpxchg_release atomic_cmpxchg_release
#define atomic64_cmpxchg atomic_cmpxchg #define atomic64_cmpxchg atomic_cmpxchg
#define atomic64_inc(v) atomic64_add(1, (v))
#define atomic64_dec(v) atomic64_sub(1, (v))
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
#define atomic64_dec_and_test(v) (atomic64_dec_return(v) == 0)
#define atomic64_sub_and_test(i, v) (atomic64_sub_return((i), (v)) == 0)
#define atomic64_add_negative(i, v) (atomic64_add_return((i), (v)) < 0)
#define atomic64_add_unless(v, a, u) (___atomic_add_unless(v, a, u, 64) != u)
#define atomic64_andnot atomic64_andnot #define atomic64_andnot atomic64_andnot
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0) #define atomic64_dec_if_positive atomic64_dec_if_positive
#endif #endif
#endif #endif
...@@ -17,22 +17,11 @@ ...@@ -17,22 +17,11 @@
#define __ASM_BITOPS_H #define __ASM_BITOPS_H
#include <linux/compiler.h> #include <linux/compiler.h>
#include <asm/barrier.h>
#ifndef _LINUX_BITOPS_H #ifndef _LINUX_BITOPS_H
#error only <linux/bitops.h> can be included directly #error only <linux/bitops.h> can be included directly
#endif #endif
/*
* Little endian assembly atomic bitops.
*/
extern void set_bit(int nr, volatile unsigned long *p);
extern void clear_bit(int nr, volatile unsigned long *p);
extern void change_bit(int nr, volatile unsigned long *p);
extern int test_and_set_bit(int nr, volatile unsigned long *p);
extern int test_and_clear_bit(int nr, volatile unsigned long *p);
extern int test_and_change_bit(int nr, volatile unsigned long *p);
#include <asm-generic/bitops/builtin-__ffs.h> #include <asm-generic/bitops/builtin-__ffs.h>
#include <asm-generic/bitops/builtin-ffs.h> #include <asm-generic/bitops/builtin-ffs.h>
#include <asm-generic/bitops/builtin-__fls.h> #include <asm-generic/bitops/builtin-__fls.h>
...@@ -44,15 +33,11 @@ extern int test_and_change_bit(int nr, volatile unsigned long *p); ...@@ -44,15 +33,11 @@ extern int test_and_change_bit(int nr, volatile unsigned long *p);
#include <asm-generic/bitops/sched.h> #include <asm-generic/bitops/sched.h>
#include <asm-generic/bitops/hweight.h> #include <asm-generic/bitops/hweight.h>
#include <asm-generic/bitops/lock.h>
#include <asm-generic/bitops/atomic.h>
#include <asm-generic/bitops/lock.h>
#include <asm-generic/bitops/non-atomic.h> #include <asm-generic/bitops/non-atomic.h>
#include <asm-generic/bitops/le.h> #include <asm-generic/bitops/le.h>
#include <asm-generic/bitops/ext2-atomic-setbit.h>
/*
* Ext2 is defined to use little-endian byte ordering.
*/
#define ext2_set_bit_atomic(lock, nr, p) test_and_set_bit_le(nr, p)
#define ext2_clear_bit_atomic(lock, nr, p) test_and_clear_bit_le(nr, p)
#endif /* __ASM_BITOPS_H */ #endif /* __ASM_BITOPS_H */
# SPDX-License-Identifier: GPL-2.0 # SPDX-License-Identifier: GPL-2.0
lib-y := bitops.o clear_user.o delay.o copy_from_user.o \ lib-y := clear_user.o delay.o copy_from_user.o \
copy_to_user.o copy_in_user.o copy_page.o \ copy_to_user.o copy_in_user.o copy_page.o \
clear_page.o memchr.o memcpy.o memmove.o memset.o \ clear_page.o memchr.o memcpy.o memmove.o memset.o \
memcmp.o strcmp.o strncmp.o strlen.o strnlen.o \ memcmp.o strcmp.o strncmp.o strlen.o strnlen.o \
......
/*
* Based on arch/arm/lib/bitops.h
*
* Copyright (C) 2013 ARM Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <linux/linkage.h>
#include <asm/assembler.h>
#include <asm/lse.h>
/*
* x0: bits 5:0 bit offset
* bits 31:6 word offset
* x1: address
*/
.macro bitop, name, llsc, lse
ENTRY( \name )
and w3, w0, #63 // Get bit offset
eor w0, w0, w3 // Clear low bits
mov x2, #1
add x1, x1, x0, lsr #3 // Get word offset
alt_lse " prfm pstl1strm, [x1]", "nop"
lsl x3, x2, x3 // Create mask
alt_lse "1: ldxr x2, [x1]", "\lse x3, [x1]"
alt_lse " \llsc x2, x2, x3", "nop"
alt_lse " stxr w0, x2, [x1]", "nop"
alt_lse " cbnz w0, 1b", "nop"
ret
ENDPROC(\name )
.endm
.macro testop, name, llsc, lse
ENTRY( \name )
and w3, w0, #63 // Get bit offset
eor w0, w0, w3 // Clear low bits
mov x2, #1
add x1, x1, x0, lsr #3 // Get word offset
alt_lse " prfm pstl1strm, [x1]", "nop"
lsl x4, x2, x3 // Create mask
alt_lse "1: ldxr x2, [x1]", "\lse x4, x2, [x1]"
lsr x0, x2, x3
alt_lse " \llsc x2, x2, x4", "nop"
alt_lse " stlxr w5, x2, [x1]", "nop"
alt_lse " cbnz w5, 1b", "nop"
alt_lse " dmb ish", "nop"
and x0, x0, #1
ret
ENDPROC(\name )
.endm
/*
* Atomic bit operations.
*/
bitop change_bit, eor, steor
bitop clear_bit, bic, stclr
bitop set_bit, orr, stset
testop test_and_change_bit, eor, ldeoral
testop test_and_clear_bit, bic, ldclral
testop test_and_set_bit, orr, ldsetal
...@@ -2,8 +2,10 @@ ...@@ -2,8 +2,10 @@
#ifndef __ARCH_H8300_ATOMIC__ #ifndef __ARCH_H8300_ATOMIC__
#define __ARCH_H8300_ATOMIC__ #define __ARCH_H8300_ATOMIC__
#include <linux/compiler.h>
#include <linux/types.h> #include <linux/types.h>
#include <asm/cmpxchg.h> #include <asm/cmpxchg.h>
#include <asm/irqflags.h>
/* /*
* Atomic operations that C can't guarantee us. Useful for * Atomic operations that C can't guarantee us. Useful for
...@@ -15,8 +17,6 @@ ...@@ -15,8 +17,6 @@
#define atomic_read(v) READ_ONCE((v)->counter) #define atomic_read(v) READ_ONCE((v)->counter)
#define atomic_set(v, i) WRITE_ONCE(((v)->counter), (i)) #define atomic_set(v, i) WRITE_ONCE(((v)->counter), (i))
#include <linux/kernel.h>
#define ATOMIC_OP_RETURN(op, c_op) \ #define ATOMIC_OP_RETURN(op, c_op) \
static inline int atomic_##op##_return(int i, atomic_t *v) \ static inline int atomic_##op##_return(int i, atomic_t *v) \
{ \ { \
...@@ -69,18 +69,6 @@ ATOMIC_OPS(sub, -=) ...@@ -69,18 +69,6 @@ ATOMIC_OPS(sub, -=)
#undef ATOMIC_OP_RETURN #undef ATOMIC_OP_RETURN
#undef ATOMIC_OP #undef ATOMIC_OP
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
#define atomic_inc_return(v) atomic_add_return(1, v)
#define atomic_dec_return(v) atomic_sub_return(1, v)
#define atomic_inc(v) (void)atomic_inc_return(v)
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
#define atomic_dec(v) (void)atomic_dec_return(v)
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
static inline int atomic_cmpxchg(atomic_t *v, int old, int new) static inline int atomic_cmpxchg(atomic_t *v, int old, int new)
{ {
int ret; int ret;
...@@ -94,7 +82,7 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new) ...@@ -94,7 +82,7 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new)
return ret; return ret;
} }
static inline int __atomic_add_unless(atomic_t *v, int a, int u) static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{ {
int ret; int ret;
h8300flags flags; h8300flags flags;
...@@ -106,5 +94,6 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u) ...@@ -106,5 +94,6 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u)
arch_local_irq_restore(flags); arch_local_irq_restore(flags);
return ret; return ret;
} }
#define atomic_fetch_add_unless atomic_fetch_add_unless
#endif /* __ARCH_H8300_ATOMIC __ */ #endif /* __ARCH_H8300_ATOMIC __ */
...@@ -164,7 +164,7 @@ ATOMIC_OPS(xor) ...@@ -164,7 +164,7 @@ ATOMIC_OPS(xor)
#undef ATOMIC_OP #undef ATOMIC_OP
/** /**
* __atomic_add_unless - add unless the number is a given value * atomic_fetch_add_unless - add unless the number is a given value
* @v: pointer to value * @v: pointer to value
* @a: amount to add * @a: amount to add
* @u: unless value is equal to u * @u: unless value is equal to u
...@@ -173,7 +173,7 @@ ATOMIC_OPS(xor) ...@@ -173,7 +173,7 @@ ATOMIC_OPS(xor)
* *
*/ */
static inline int __atomic_add_unless(atomic_t *v, int a, int u) static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{ {
int __oldval; int __oldval;
register int tmp; register int tmp;
...@@ -196,18 +196,6 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u) ...@@ -196,18 +196,6 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u)
); );
return __oldval; return __oldval;
} }
#define atomic_fetch_add_unless atomic_fetch_add_unless
#define atomic_inc_not_zero(v) atomic_add_unless((v), 1, 0)
#define atomic_inc(v) atomic_add(1, (v))
#define atomic_dec(v) atomic_sub(1, (v))
#define atomic_inc_and_test(v) (atomic_add_return(1, (v)) == 0)
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, (v)) == 0)
#define atomic_add_negative(i, v) (atomic_add_return(i, (v)) < 0)
#define atomic_inc_return(v) (atomic_add_return(1, v))
#define atomic_dec_return(v) (atomic_sub_return(1, v))
#endif #endif
...@@ -215,91 +215,10 @@ ATOMIC64_FETCH_OP(xor, ^) ...@@ -215,91 +215,10 @@ ATOMIC64_FETCH_OP(xor, ^)
(cmpxchg(&((v)->counter), old, new)) (cmpxchg(&((v)->counter), old, new))
#define atomic64_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic64_xchg(v, new) (xchg(&((v)->counter), new))
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c, old;
c = atomic_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c;
}
static __inline__ long atomic64_add_unless(atomic64_t *v, long a, long u)
{
long c, old;
c = atomic64_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic64_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c != (u);
}
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
static __inline__ long atomic64_dec_if_positive(atomic64_t *v)
{
long c, old, dec;
c = atomic64_read(v);
for (;;) {
dec = c - 1;
if (unlikely(dec < 0))
break;
old = atomic64_cmpxchg((v), c, dec);
if (likely(old == c))
break;
c = old;
}
return dec;
}
/*
* Atomically add I to V and return TRUE if the resulting value is
* negative.
*/
static __inline__ int
atomic_add_negative (int i, atomic_t *v)
{
return atomic_add_return(i, v) < 0;
}
static __inline__ long
atomic64_add_negative (__s64 i, atomic64_t *v)
{
return atomic64_add_return(i, v) < 0;
}
#define atomic_dec_return(v) atomic_sub_return(1, (v))
#define atomic_inc_return(v) atomic_add_return(1, (v))
#define atomic64_dec_return(v) atomic64_sub_return(1, (v))
#define atomic64_inc_return(v) atomic64_add_return(1, (v))
#define atomic_sub_and_test(i,v) (atomic_sub_return((i), (v)) == 0)
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
#define atomic_inc_and_test(v) (atomic_add_return(1, (v)) == 0)
#define atomic64_sub_and_test(i,v) (atomic64_sub_return((i), (v)) == 0)
#define atomic64_dec_and_test(v) (atomic64_sub_return(1, (v)) == 0)
#define atomic64_inc_and_test(v) (atomic64_add_return(1, (v)) == 0)
#define atomic_add(i,v) (void)atomic_add_return((i), (v)) #define atomic_add(i,v) (void)atomic_add_return((i), (v))
#define atomic_sub(i,v) (void)atomic_sub_return((i), (v)) #define atomic_sub(i,v) (void)atomic_sub_return((i), (v))
#define atomic_inc(v) atomic_add(1, (v))
#define atomic_dec(v) atomic_sub(1, (v))
#define atomic64_add(i,v) (void)atomic64_add_return((i), (v)) #define atomic64_add(i,v) (void)atomic64_add_return((i), (v))
#define atomic64_sub(i,v) (void)atomic64_sub_return((i), (v)) #define atomic64_sub(i,v) (void)atomic64_sub_return((i), (v))
#define atomic64_inc(v) atomic64_add(1, (v))
#define atomic64_dec(v) atomic64_sub(1, (v))
#endif /* _ASM_IA64_ATOMIC_H */ #endif /* _ASM_IA64_ATOMIC_H */
...@@ -126,11 +126,13 @@ static inline void atomic_inc(atomic_t *v) ...@@ -126,11 +126,13 @@ static inline void atomic_inc(atomic_t *v)
{ {
__asm__ __volatile__("addql #1,%0" : "+m" (*v)); __asm__ __volatile__("addql #1,%0" : "+m" (*v));
} }
#define atomic_inc atomic_inc
static inline void atomic_dec(atomic_t *v) static inline void atomic_dec(atomic_t *v)
{ {
__asm__ __volatile__("subql #1,%0" : "+m" (*v)); __asm__ __volatile__("subql #1,%0" : "+m" (*v));
} }
#define atomic_dec atomic_dec
static inline int atomic_dec_and_test(atomic_t *v) static inline int atomic_dec_and_test(atomic_t *v)
{ {
...@@ -138,6 +140,7 @@ static inline int atomic_dec_and_test(atomic_t *v) ...@@ -138,6 +140,7 @@ static inline int atomic_dec_and_test(atomic_t *v)
__asm__ __volatile__("subql #1,%1; seq %0" : "=d" (c), "+m" (*v)); __asm__ __volatile__("subql #1,%1; seq %0" : "=d" (c), "+m" (*v));
return c != 0; return c != 0;
} }
#define atomic_dec_and_test atomic_dec_and_test
static inline int atomic_dec_and_test_lt(atomic_t *v) static inline int atomic_dec_and_test_lt(atomic_t *v)
{ {
...@@ -155,6 +158,7 @@ static inline int atomic_inc_and_test(atomic_t *v) ...@@ -155,6 +158,7 @@ static inline int atomic_inc_and_test(atomic_t *v)
__asm__ __volatile__("addql #1,%1; seq %0" : "=d" (c), "+m" (*v)); __asm__ __volatile__("addql #1,%1; seq %0" : "=d" (c), "+m" (*v));
return c != 0; return c != 0;
} }
#define atomic_inc_and_test atomic_inc_and_test
#ifdef CONFIG_RMW_INSNS #ifdef CONFIG_RMW_INSNS
...@@ -190,9 +194,6 @@ static inline int atomic_xchg(atomic_t *v, int new) ...@@ -190,9 +194,6 @@ static inline int atomic_xchg(atomic_t *v, int new)
#endif /* !CONFIG_RMW_INSNS */ #endif /* !CONFIG_RMW_INSNS */
#define atomic_dec_return(v) atomic_sub_return(1, (v))
#define atomic_inc_return(v) atomic_add_return(1, (v))
static inline int atomic_sub_and_test(int i, atomic_t *v) static inline int atomic_sub_and_test(int i, atomic_t *v)
{ {
char c; char c;
...@@ -201,6 +202,7 @@ static inline int atomic_sub_and_test(int i, atomic_t *v) ...@@ -201,6 +202,7 @@ static inline int atomic_sub_and_test(int i, atomic_t *v)
: ASM_DI (i)); : ASM_DI (i));
return c != 0; return c != 0;
} }
#define atomic_sub_and_test atomic_sub_and_test
static inline int atomic_add_negative(int i, atomic_t *v) static inline int atomic_add_negative(int i, atomic_t *v)
{ {
...@@ -210,20 +212,6 @@ static inline int atomic_add_negative(int i, atomic_t *v) ...@@ -210,20 +212,6 @@ static inline int atomic_add_negative(int i, atomic_t *v)
: ASM_DI (i)); : ASM_DI (i));
return c != 0; return c != 0;
} }
#define atomic_add_negative atomic_add_negative
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c, old;
c = atomic_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c;
}
#endif /* __ARCH_M68K_ATOMIC __ */ #endif /* __ARCH_M68K_ATOMIC __ */
...@@ -519,12 +519,16 @@ static inline int __fls(int x) ...@@ -519,12 +519,16 @@ static inline int __fls(int x)
#endif #endif
/* Simple test-and-set bit locks */
#define test_and_set_bit_lock test_and_set_bit
#define clear_bit_unlock clear_bit
#define __clear_bit_unlock clear_bit_unlock
#include <asm-generic/bitops/ext2-atomic.h> #include <asm-generic/bitops/ext2-atomic.h>
#include <asm-generic/bitops/le.h> #include <asm-generic/bitops/le.h>
#include <asm-generic/bitops/fls64.h> #include <asm-generic/bitops/fls64.h>
#include <asm-generic/bitops/sched.h> #include <asm-generic/bitops/sched.h>
#include <asm-generic/bitops/hweight.h> #include <asm-generic/bitops/hweight.h>
#include <asm-generic/bitops/lock.h>
#endif /* __KERNEL__ */ #endif /* __KERNEL__ */
#endif /* _M68K_BITOPS_H */ #endif /* _M68K_BITOPS_H */
...@@ -274,97 +274,12 @@ static __inline__ int atomic_sub_if_positive(int i, atomic_t * v) ...@@ -274,97 +274,12 @@ static __inline__ int atomic_sub_if_positive(int i, atomic_t * v)
#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n))) #define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
#define atomic_xchg(v, new) (xchg(&((v)->counter), (new))) #define atomic_xchg(v, new) (xchg(&((v)->counter), (new)))
/**
* __atomic_add_unless - add unless the number is a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v.
*/
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c, old;
c = atomic_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c;
}
#define atomic_dec_return(v) atomic_sub_return(1, (v))
#define atomic_inc_return(v) atomic_add_return(1, (v))
/*
* atomic_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer of type atomic_t
*
* Atomically subtracts @i from @v and returns
* true if the result is zero, or false for all
* other cases.
*/
#define atomic_sub_and_test(i, v) (atomic_sub_return((i), (v)) == 0)
/*
* atomic_inc_and_test - increment and test
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
/*
* atomic_dec_and_test - decrement by 1 and test
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
/* /*
* atomic_dec_if_positive - decrement by 1 if old value positive * atomic_dec_if_positive - decrement by 1 if old value positive
* @v: pointer of type atomic_t * @v: pointer of type atomic_t
*/ */
#define atomic_dec_if_positive(v) atomic_sub_if_positive(1, v) #define atomic_dec_if_positive(v) atomic_sub_if_positive(1, v)
/*
* atomic_inc - increment atomic variable
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1.
*/
#define atomic_inc(v) atomic_add(1, (v))
/*
* atomic_dec - decrement and test
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1.
*/
#define atomic_dec(v) atomic_sub(1, (v))
/*
* atomic_add_negative - add and test if negative
* @v: pointer of type atomic_t
* @i: integer value to add
*
* Atomically adds @i to @v and returns true
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
#define atomic_add_negative(i, v) (atomic_add_return(i, (v)) < 0)
#ifdef CONFIG_64BIT #ifdef CONFIG_64BIT
#define ATOMIC64_INIT(i) { (i) } #define ATOMIC64_INIT(i) { (i) }
...@@ -620,99 +535,12 @@ static __inline__ long atomic64_sub_if_positive(long i, atomic64_t * v) ...@@ -620,99 +535,12 @@ static __inline__ long atomic64_sub_if_positive(long i, atomic64_t * v)
((__typeof__((v)->counter))cmpxchg(&((v)->counter), (o), (n))) ((__typeof__((v)->counter))cmpxchg(&((v)->counter), (o), (n)))
#define atomic64_xchg(v, new) (xchg(&((v)->counter), (new))) #define atomic64_xchg(v, new) (xchg(&((v)->counter), (new)))
/**
* atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, so long as it was not @u.
* Returns true iff @v was not @u.
*/
static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u)
{
long c, old;
c = atomic64_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic64_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c != (u);
}
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
#define atomic64_dec_return(v) atomic64_sub_return(1, (v))
#define atomic64_inc_return(v) atomic64_add_return(1, (v))
/*
* atomic64_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer of type atomic64_t
*
* Atomically subtracts @i from @v and returns
* true if the result is zero, or false for all
* other cases.
*/
#define atomic64_sub_and_test(i, v) (atomic64_sub_return((i), (v)) == 0)
/*
* atomic64_inc_and_test - increment and test
* @v: pointer of type atomic64_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
/*
* atomic64_dec_and_test - decrement by 1 and test
* @v: pointer of type atomic64_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
#define atomic64_dec_and_test(v) (atomic64_sub_return(1, (v)) == 0)
/* /*
* atomic64_dec_if_positive - decrement by 1 if old value positive * atomic64_dec_if_positive - decrement by 1 if old value positive
* @v: pointer of type atomic64_t * @v: pointer of type atomic64_t
*/ */
#define atomic64_dec_if_positive(v) atomic64_sub_if_positive(1, v) #define atomic64_dec_if_positive(v) atomic64_sub_if_positive(1, v)
/*
* atomic64_inc - increment atomic variable
* @v: pointer of type atomic64_t
*
* Atomically increments @v by 1.
*/
#define atomic64_inc(v) atomic64_add(1, (v))
/*
* atomic64_dec - decrement and test
* @v: pointer of type atomic64_t
*
* Atomically decrements @v by 1.
*/
#define atomic64_dec(v) atomic64_sub(1, (v))
/*
* atomic64_add_negative - add and test if negative
* @v: pointer of type atomic64_t
* @i: integer value to add
*
* Atomically adds @i to @v and returns true
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
#define atomic64_add_negative(i, v) (atomic64_add_return(i, (v)) < 0)
#endif /* CONFIG_64BIT */ #endif /* CONFIG_64BIT */
#endif /* _ASM_ATOMIC_H */ #endif /* _ASM_ATOMIC_H */
...@@ -100,7 +100,7 @@ ATOMIC_OP(xor) ...@@ -100,7 +100,7 @@ ATOMIC_OP(xor)
* *
* This is often used through atomic_inc_not_zero() * This is often used through atomic_inc_not_zero()
*/ */
static inline int __atomic_add_unless(atomic_t *v, int a, int u) static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{ {
int old, tmp; int old, tmp;
...@@ -119,7 +119,7 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u) ...@@ -119,7 +119,7 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u)
return old; return old;
} }
#define __atomic_add_unless __atomic_add_unless #define atomic_fetch_add_unless atomic_fetch_add_unless
#include <asm-generic/atomic.h> #include <asm-generic/atomic.h>
......
...@@ -16,8 +16,9 @@ ...@@ -16,8 +16,9 @@
#ifndef __ASM_OPENRISC_CMPXCHG_H #ifndef __ASM_OPENRISC_CMPXCHG_H
#define __ASM_OPENRISC_CMPXCHG_H #define __ASM_OPENRISC_CMPXCHG_H
#include <linux/bits.h>
#include <linux/compiler.h>
#include <linux/types.h> #include <linux/types.h>
#include <linux/bitops.h>
#define __HAVE_ARCH_CMPXCHG 1 #define __HAVE_ARCH_CMPXCHG 1
......
...@@ -77,30 +77,6 @@ static __inline__ int atomic_read(const atomic_t *v) ...@@ -77,30 +77,6 @@ static __inline__ int atomic_read(const atomic_t *v)
#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n))) #define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
#define atomic_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic_xchg(v, new) (xchg(&((v)->counter), new))
/**
* __atomic_add_unless - add unless the number is a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v.
*/
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c, old;
c = atomic_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c;
}
#define ATOMIC_OP(op, c_op) \ #define ATOMIC_OP(op, c_op) \
static __inline__ void atomic_##op(int i, atomic_t *v) \ static __inline__ void atomic_##op(int i, atomic_t *v) \
{ \ { \
...@@ -160,28 +136,6 @@ ATOMIC_OPS(xor, ^=) ...@@ -160,28 +136,6 @@ ATOMIC_OPS(xor, ^=)
#undef ATOMIC_OP_RETURN #undef ATOMIC_OP_RETURN
#undef ATOMIC_OP #undef ATOMIC_OP
#define atomic_inc(v) (atomic_add( 1,(v)))
#define atomic_dec(v) (atomic_add( -1,(v)))
#define atomic_inc_return(v) (atomic_add_return( 1,(v)))
#define atomic_dec_return(v) (atomic_add_return( -1,(v)))
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
/*
* atomic_inc_and_test - increment and test
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
#define atomic_sub_and_test(i,v) (atomic_sub_return((i),(v)) == 0)
#define ATOMIC_INIT(i) { (i) } #define ATOMIC_INIT(i) { (i) }
#ifdef CONFIG_64BIT #ifdef CONFIG_64BIT
...@@ -264,72 +218,11 @@ atomic64_read(const atomic64_t *v) ...@@ -264,72 +218,11 @@ atomic64_read(const atomic64_t *v)
return READ_ONCE((v)->counter); return READ_ONCE((v)->counter);
} }
#define atomic64_inc(v) (atomic64_add( 1,(v)))
#define atomic64_dec(v) (atomic64_add( -1,(v)))
#define atomic64_inc_return(v) (atomic64_add_return( 1,(v)))
#define atomic64_dec_return(v) (atomic64_add_return( -1,(v)))
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
#define atomic64_dec_and_test(v) (atomic64_dec_return(v) == 0)
#define atomic64_sub_and_test(i,v) (atomic64_sub_return((i),(v)) == 0)
/* exported interface */ /* exported interface */
#define atomic64_cmpxchg(v, o, n) \ #define atomic64_cmpxchg(v, o, n) \
((__typeof__((v)->counter))cmpxchg(&((v)->counter), (o), (n))) ((__typeof__((v)->counter))cmpxchg(&((v)->counter), (o), (n)))
#define atomic64_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic64_xchg(v, new) (xchg(&((v)->counter), new))
/**
* atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v.
*/
static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u)
{
long c, old;
c = atomic64_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic64_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c != (u);
}
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
/*
* atomic64_dec_if_positive - decrement by 1 if old value positive
* @v: pointer of type atomic_t
*
* The function returns the old value of *v minus 1, even if
* the atomic variable, v, was not decremented.
*/
static inline long atomic64_dec_if_positive(atomic64_t *v)
{
long c, old, dec;
c = atomic64_read(v);
for (;;) {
dec = c - 1;
if (unlikely(dec < 0))
break;
old = atomic64_cmpxchg((v), c, dec);
if (likely(old == c))
break;
c = old;
}
return dec;
}
#endif /* !CONFIG_64BIT */ #endif /* !CONFIG_64BIT */
......
...@@ -18,18 +18,11 @@ ...@@ -18,18 +18,11 @@
* a "bne-" instruction at the end, so an isync is enough as a acquire barrier * a "bne-" instruction at the end, so an isync is enough as a acquire barrier
* on the platform without lwsync. * on the platform without lwsync.
*/ */
#define __atomic_op_acquire(op, args...) \ #define __atomic_acquire_fence() \
({ \ __asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory")
typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \
__asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory"); \ #define __atomic_release_fence() \
__ret; \ __asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory")
})
#define __atomic_op_release(op, args...) \
({ \
__asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory"); \
op##_relaxed(args); \
})
static __inline__ int atomic_read(const atomic_t *v) static __inline__ int atomic_read(const atomic_t *v)
{ {
...@@ -129,8 +122,6 @@ ATOMIC_OPS(xor, xor) ...@@ -129,8 +122,6 @@ ATOMIC_OPS(xor, xor)
#undef ATOMIC_OP_RETURN_RELAXED #undef ATOMIC_OP_RETURN_RELAXED
#undef ATOMIC_OP #undef ATOMIC_OP
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
static __inline__ void atomic_inc(atomic_t *v) static __inline__ void atomic_inc(atomic_t *v)
{ {
int t; int t;
...@@ -145,6 +136,7 @@ static __inline__ void atomic_inc(atomic_t *v) ...@@ -145,6 +136,7 @@ static __inline__ void atomic_inc(atomic_t *v)
: "r" (&v->counter) : "r" (&v->counter)
: "cc", "xer"); : "cc", "xer");
} }
#define atomic_inc atomic_inc
static __inline__ int atomic_inc_return_relaxed(atomic_t *v) static __inline__ int atomic_inc_return_relaxed(atomic_t *v)
{ {
...@@ -163,16 +155,6 @@ static __inline__ int atomic_inc_return_relaxed(atomic_t *v) ...@@ -163,16 +155,6 @@ static __inline__ int atomic_inc_return_relaxed(atomic_t *v)
return t; return t;
} }
/*
* atomic_inc_and_test - increment and test
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
static __inline__ void atomic_dec(atomic_t *v) static __inline__ void atomic_dec(atomic_t *v)
{ {
int t; int t;
...@@ -187,6 +169,7 @@ static __inline__ void atomic_dec(atomic_t *v) ...@@ -187,6 +169,7 @@ static __inline__ void atomic_dec(atomic_t *v)
: "r" (&v->counter) : "r" (&v->counter)
: "cc", "xer"); : "cc", "xer");
} }
#define atomic_dec atomic_dec
static __inline__ int atomic_dec_return_relaxed(atomic_t *v) static __inline__ int atomic_dec_return_relaxed(atomic_t *v)
{ {
...@@ -218,7 +201,7 @@ static __inline__ int atomic_dec_return_relaxed(atomic_t *v) ...@@ -218,7 +201,7 @@ static __inline__ int atomic_dec_return_relaxed(atomic_t *v)
#define atomic_xchg_relaxed(v, new) xchg_relaxed(&((v)->counter), (new)) #define atomic_xchg_relaxed(v, new) xchg_relaxed(&((v)->counter), (new))
/** /**
* __atomic_add_unless - add unless the number is a given value * atomic_fetch_add_unless - add unless the number is a given value
* @v: pointer of type atomic_t * @v: pointer of type atomic_t
* @a: the amount to add to v... * @a: the amount to add to v...
* @u: ...unless v is equal to u. * @u: ...unless v is equal to u.
...@@ -226,13 +209,13 @@ static __inline__ int atomic_dec_return_relaxed(atomic_t *v) ...@@ -226,13 +209,13 @@ static __inline__ int atomic_dec_return_relaxed(atomic_t *v)
* Atomically adds @a to @v, so long as it was not @u. * Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v. * Returns the old value of @v.
*/ */
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u) static __inline__ int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{ {
int t; int t;
__asm__ __volatile__ ( __asm__ __volatile__ (
PPC_ATOMIC_ENTRY_BARRIER PPC_ATOMIC_ENTRY_BARRIER
"1: lwarx %0,0,%1 # __atomic_add_unless\n\ "1: lwarx %0,0,%1 # atomic_fetch_add_unless\n\
cmpw 0,%0,%3 \n\ cmpw 0,%0,%3 \n\
beq 2f \n\ beq 2f \n\
add %0,%2,%0 \n" add %0,%2,%0 \n"
...@@ -248,6 +231,7 @@ static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u) ...@@ -248,6 +231,7 @@ static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
return t; return t;
} }
#define atomic_fetch_add_unless atomic_fetch_add_unless
/** /**
* atomic_inc_not_zero - increment unless the number is zero * atomic_inc_not_zero - increment unless the number is zero
...@@ -280,9 +264,6 @@ static __inline__ int atomic_inc_not_zero(atomic_t *v) ...@@ -280,9 +264,6 @@ static __inline__ int atomic_inc_not_zero(atomic_t *v)
} }
#define atomic_inc_not_zero(v) atomic_inc_not_zero((v)) #define atomic_inc_not_zero(v) atomic_inc_not_zero((v))
#define atomic_sub_and_test(a, v) (atomic_sub_return((a), (v)) == 0)
#define atomic_dec_and_test(v) (atomic_dec_return((v)) == 0)
/* /*
* Atomically test *v and decrement if it is greater than 0. * Atomically test *v and decrement if it is greater than 0.
* The function returns the old value of *v minus 1, even if * The function returns the old value of *v minus 1, even if
...@@ -412,8 +393,6 @@ ATOMIC64_OPS(xor, xor) ...@@ -412,8 +393,6 @@ ATOMIC64_OPS(xor, xor)
#undef ATOMIC64_OP_RETURN_RELAXED #undef ATOMIC64_OP_RETURN_RELAXED
#undef ATOMIC64_OP #undef ATOMIC64_OP
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
static __inline__ void atomic64_inc(atomic64_t *v) static __inline__ void atomic64_inc(atomic64_t *v)
{ {
long t; long t;
...@@ -427,6 +406,7 @@ static __inline__ void atomic64_inc(atomic64_t *v) ...@@ -427,6 +406,7 @@ static __inline__ void atomic64_inc(atomic64_t *v)
: "r" (&v->counter) : "r" (&v->counter)
: "cc", "xer"); : "cc", "xer");
} }
#define atomic64_inc atomic64_inc
static __inline__ long atomic64_inc_return_relaxed(atomic64_t *v) static __inline__ long atomic64_inc_return_relaxed(atomic64_t *v)
{ {
...@@ -444,16 +424,6 @@ static __inline__ long atomic64_inc_return_relaxed(atomic64_t *v) ...@@ -444,16 +424,6 @@ static __inline__ long atomic64_inc_return_relaxed(atomic64_t *v)
return t; return t;
} }
/*
* atomic64_inc_and_test - increment and test
* @v: pointer of type atomic64_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
static __inline__ void atomic64_dec(atomic64_t *v) static __inline__ void atomic64_dec(atomic64_t *v)
{ {
long t; long t;
...@@ -467,6 +437,7 @@ static __inline__ void atomic64_dec(atomic64_t *v) ...@@ -467,6 +437,7 @@ static __inline__ void atomic64_dec(atomic64_t *v)
: "r" (&v->counter) : "r" (&v->counter)
: "cc", "xer"); : "cc", "xer");
} }
#define atomic64_dec atomic64_dec
static __inline__ long atomic64_dec_return_relaxed(atomic64_t *v) static __inline__ long atomic64_dec_return_relaxed(atomic64_t *v)
{ {
...@@ -487,9 +458,6 @@ static __inline__ long atomic64_dec_return_relaxed(atomic64_t *v) ...@@ -487,9 +458,6 @@ static __inline__ long atomic64_dec_return_relaxed(atomic64_t *v)
#define atomic64_inc_return_relaxed atomic64_inc_return_relaxed #define atomic64_inc_return_relaxed atomic64_inc_return_relaxed
#define atomic64_dec_return_relaxed atomic64_dec_return_relaxed #define atomic64_dec_return_relaxed atomic64_dec_return_relaxed
#define atomic64_sub_and_test(a, v) (atomic64_sub_return((a), (v)) == 0)
#define atomic64_dec_and_test(v) (atomic64_dec_return((v)) == 0)
/* /*
* Atomically test *v and decrement if it is greater than 0. * Atomically test *v and decrement if it is greater than 0.
* The function returns the old value of *v minus 1. * The function returns the old value of *v minus 1.
...@@ -513,6 +481,7 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v) ...@@ -513,6 +481,7 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v)
return t; return t;
} }
#define atomic64_dec_if_positive atomic64_dec_if_positive
#define atomic64_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n))) #define atomic64_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
#define atomic64_cmpxchg_relaxed(v, o, n) \ #define atomic64_cmpxchg_relaxed(v, o, n) \
...@@ -524,7 +493,7 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v) ...@@ -524,7 +493,7 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v)
#define atomic64_xchg_relaxed(v, new) xchg_relaxed(&((v)->counter), (new)) #define atomic64_xchg_relaxed(v, new) xchg_relaxed(&((v)->counter), (new))
/** /**
* atomic64_add_unless - add unless the number is a given value * atomic64_fetch_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t * @v: pointer of type atomic64_t
* @a: the amount to add to v... * @a: the amount to add to v...
* @u: ...unless v is equal to u. * @u: ...unless v is equal to u.
...@@ -532,13 +501,13 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v) ...@@ -532,13 +501,13 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v)
* Atomically adds @a to @v, so long as it was not @u. * Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v. * Returns the old value of @v.
*/ */
static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u) static __inline__ long atomic64_fetch_add_unless(atomic64_t *v, long a, long u)
{ {
long t; long t;
__asm__ __volatile__ ( __asm__ __volatile__ (
PPC_ATOMIC_ENTRY_BARRIER PPC_ATOMIC_ENTRY_BARRIER
"1: ldarx %0,0,%1 # __atomic_add_unless\n\ "1: ldarx %0,0,%1 # atomic64_fetch_add_unless\n\
cmpd 0,%0,%3 \n\ cmpd 0,%0,%3 \n\
beq 2f \n\ beq 2f \n\
add %0,%2,%0 \n" add %0,%2,%0 \n"
...@@ -551,8 +520,9 @@ static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u) ...@@ -551,8 +520,9 @@ static __inline__ int atomic64_add_unless(atomic64_t *v, long a, long u)
: "r" (&v->counter), "r" (a), "r" (u) : "r" (&v->counter), "r" (a), "r" (u)
: "cc", "memory"); : "cc", "memory");
return t != u; return t;
} }
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
/** /**
* atomic_inc64_not_zero - increment unless the number is zero * atomic_inc64_not_zero - increment unless the number is zero
...@@ -582,6 +552,7 @@ static __inline__ int atomic64_inc_not_zero(atomic64_t *v) ...@@ -582,6 +552,7 @@ static __inline__ int atomic64_inc_not_zero(atomic64_t *v)
return t1 != 0; return t1 != 0;
} }
#define atomic64_inc_not_zero(v) atomic64_inc_not_zero((v))
#endif /* __powerpc64__ */ #endif /* __powerpc64__ */
......
...@@ -25,18 +25,11 @@ ...@@ -25,18 +25,11 @@
#define ATOMIC_INIT(i) { (i) } #define ATOMIC_INIT(i) { (i) }
#define __atomic_op_acquire(op, args...) \ #define __atomic_acquire_fence() \
({ \ __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory")
typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \
__asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory"); \ #define __atomic_release_fence() \
__ret; \ __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory");
})
#define __atomic_op_release(op, args...) \
({ \
__asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); \
op##_relaxed(args); \
})
static __always_inline int atomic_read(const atomic_t *v) static __always_inline int atomic_read(const atomic_t *v)
{ {
...@@ -209,130 +202,8 @@ ATOMIC_OPS(xor, xor, i) ...@@ -209,130 +202,8 @@ ATOMIC_OPS(xor, xor, i)
#undef ATOMIC_FETCH_OP #undef ATOMIC_FETCH_OP
#undef ATOMIC_OP_RETURN #undef ATOMIC_OP_RETURN
/*
* The extra atomic operations that are constructed from one of the core
* AMO-based operations above (aside from sub, which is easier to fit above).
* These are required to perform a full barrier, but they're OK this way
* because atomic_*_return is also required to perform a full barrier.
*
*/
#define ATOMIC_OP(op, func_op, comp_op, I, c_type, prefix) \
static __always_inline \
bool atomic##prefix##_##op(c_type i, atomic##prefix##_t *v) \
{ \
return atomic##prefix##_##func_op##_return(i, v) comp_op I; \
}
#ifdef CONFIG_GENERIC_ATOMIC64
#define ATOMIC_OPS(op, func_op, comp_op, I) \
ATOMIC_OP(op, func_op, comp_op, I, int, )
#else
#define ATOMIC_OPS(op, func_op, comp_op, I) \
ATOMIC_OP(op, func_op, comp_op, I, int, ) \
ATOMIC_OP(op, func_op, comp_op, I, long, 64)
#endif
ATOMIC_OPS(add_and_test, add, ==, 0)
ATOMIC_OPS(sub_and_test, sub, ==, 0)
ATOMIC_OPS(add_negative, add, <, 0)
#undef ATOMIC_OP
#undef ATOMIC_OPS
#define ATOMIC_OP(op, func_op, I, c_type, prefix) \
static __always_inline \
void atomic##prefix##_##op(atomic##prefix##_t *v) \
{ \
atomic##prefix##_##func_op(I, v); \
}
#define ATOMIC_FETCH_OP(op, func_op, I, c_type, prefix) \
static __always_inline \
c_type atomic##prefix##_fetch_##op##_relaxed(atomic##prefix##_t *v) \
{ \
return atomic##prefix##_fetch_##func_op##_relaxed(I, v); \
} \
static __always_inline \
c_type atomic##prefix##_fetch_##op(atomic##prefix##_t *v) \
{ \
return atomic##prefix##_fetch_##func_op(I, v); \
}
#define ATOMIC_OP_RETURN(op, asm_op, c_op, I, c_type, prefix) \
static __always_inline \
c_type atomic##prefix##_##op##_return_relaxed(atomic##prefix##_t *v) \
{ \
return atomic##prefix##_fetch_##op##_relaxed(v) c_op I; \
} \
static __always_inline \
c_type atomic##prefix##_##op##_return(atomic##prefix##_t *v) \
{ \
return atomic##prefix##_fetch_##op(v) c_op I; \
}
#ifdef CONFIG_GENERIC_ATOMIC64
#define ATOMIC_OPS(op, asm_op, c_op, I) \
ATOMIC_OP( op, asm_op, I, int, ) \
ATOMIC_FETCH_OP( op, asm_op, I, int, ) \
ATOMIC_OP_RETURN(op, asm_op, c_op, I, int, )
#else
#define ATOMIC_OPS(op, asm_op, c_op, I) \
ATOMIC_OP( op, asm_op, I, int, ) \
ATOMIC_FETCH_OP( op, asm_op, I, int, ) \
ATOMIC_OP_RETURN(op, asm_op, c_op, I, int, ) \
ATOMIC_OP( op, asm_op, I, long, 64) \
ATOMIC_FETCH_OP( op, asm_op, I, long, 64) \
ATOMIC_OP_RETURN(op, asm_op, c_op, I, long, 64)
#endif
ATOMIC_OPS(inc, add, +, 1)
ATOMIC_OPS(dec, add, +, -1)
#define atomic_inc_return_relaxed atomic_inc_return_relaxed
#define atomic_dec_return_relaxed atomic_dec_return_relaxed
#define atomic_inc_return atomic_inc_return
#define atomic_dec_return atomic_dec_return
#define atomic_fetch_inc_relaxed atomic_fetch_inc_relaxed
#define atomic_fetch_dec_relaxed atomic_fetch_dec_relaxed
#define atomic_fetch_inc atomic_fetch_inc
#define atomic_fetch_dec atomic_fetch_dec
#ifndef CONFIG_GENERIC_ATOMIC64
#define atomic64_inc_return_relaxed atomic64_inc_return_relaxed
#define atomic64_dec_return_relaxed atomic64_dec_return_relaxed
#define atomic64_inc_return atomic64_inc_return
#define atomic64_dec_return atomic64_dec_return
#define atomic64_fetch_inc_relaxed atomic64_fetch_inc_relaxed
#define atomic64_fetch_dec_relaxed atomic64_fetch_dec_relaxed
#define atomic64_fetch_inc atomic64_fetch_inc
#define atomic64_fetch_dec atomic64_fetch_dec
#endif
#undef ATOMIC_OPS
#undef ATOMIC_OP
#undef ATOMIC_FETCH_OP
#undef ATOMIC_OP_RETURN
#define ATOMIC_OP(op, func_op, comp_op, I, prefix) \
static __always_inline \
bool atomic##prefix##_##op(atomic##prefix##_t *v) \
{ \
return atomic##prefix##_##func_op##_return(v) comp_op I; \
}
ATOMIC_OP(inc_and_test, inc, ==, 0, )
ATOMIC_OP(dec_and_test, dec, ==, 0, )
#ifndef CONFIG_GENERIC_ATOMIC64
ATOMIC_OP(inc_and_test, inc, ==, 0, 64)
ATOMIC_OP(dec_and_test, dec, ==, 0, 64)
#endif
#undef ATOMIC_OP
/* This is required to provide a full barrier on success. */ /* This is required to provide a full barrier on success. */
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) static __always_inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{ {
int prev, rc; int prev, rc;
...@@ -349,9 +220,10 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) ...@@ -349,9 +220,10 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
: "memory"); : "memory");
return prev; return prev;
} }
#define atomic_fetch_add_unless atomic_fetch_add_unless
#ifndef CONFIG_GENERIC_ATOMIC64 #ifndef CONFIG_GENERIC_ATOMIC64
static __always_inline long __atomic64_add_unless(atomic64_t *v, long a, long u) static __always_inline long atomic64_fetch_add_unless(atomic64_t *v, long a, long u)
{ {
long prev, rc; long prev, rc;
...@@ -368,27 +240,7 @@ static __always_inline long __atomic64_add_unless(atomic64_t *v, long a, long u) ...@@ -368,27 +240,7 @@ static __always_inline long __atomic64_add_unless(atomic64_t *v, long a, long u)
: "memory"); : "memory");
return prev; return prev;
} }
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
static __always_inline int atomic64_add_unless(atomic64_t *v, long a, long u)
{
return __atomic64_add_unless(v, a, u) != u;
}
#endif
/*
* The extra atomic operations that are constructed from one of the core
* LR/SC-based operations above.
*/
static __always_inline int atomic_inc_not_zero(atomic_t *v)
{
return __atomic_add_unless(v, 1, 0);
}
#ifndef CONFIG_GENERIC_ATOMIC64
static __always_inline long atomic64_inc_not_zero(atomic64_t *v)
{
return atomic64_add_unless(v, 1, 0);
}
#endif #endif
/* /*
......
...@@ -55,17 +55,9 @@ static inline void atomic_add(int i, atomic_t *v) ...@@ -55,17 +55,9 @@ static inline void atomic_add(int i, atomic_t *v)
__atomic_add(i, &v->counter); __atomic_add(i, &v->counter);
} }
#define atomic_add_negative(_i, _v) (atomic_add_return(_i, _v) < 0)
#define atomic_inc(_v) atomic_add(1, _v)
#define atomic_inc_return(_v) atomic_add_return(1, _v)
#define atomic_inc_and_test(_v) (atomic_add_return(1, _v) == 0)
#define atomic_sub(_i, _v) atomic_add(-(int)(_i), _v) #define atomic_sub(_i, _v) atomic_add(-(int)(_i), _v)
#define atomic_sub_return(_i, _v) atomic_add_return(-(int)(_i), _v) #define atomic_sub_return(_i, _v) atomic_add_return(-(int)(_i), _v)
#define atomic_fetch_sub(_i, _v) atomic_fetch_add(-(int)(_i), _v) #define atomic_fetch_sub(_i, _v) atomic_fetch_add(-(int)(_i), _v)
#define atomic_sub_and_test(_i, _v) (atomic_sub_return(_i, _v) == 0)
#define atomic_dec(_v) atomic_sub(1, _v)
#define atomic_dec_return(_v) atomic_sub_return(1, _v)
#define atomic_dec_and_test(_v) (atomic_sub_return(1, _v) == 0)
#define ATOMIC_OPS(op) \ #define ATOMIC_OPS(op) \
static inline void atomic_##op(int i, atomic_t *v) \ static inline void atomic_##op(int i, atomic_t *v) \
...@@ -90,21 +82,6 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new) ...@@ -90,21 +82,6 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new)
return __atomic_cmpxchg(&v->counter, old, new); return __atomic_cmpxchg(&v->counter, old, new);
} }
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c, old;
c = atomic_read(v);
for (;;) {
if (unlikely(c == u))
break;
old = atomic_cmpxchg(v, c, c + a);
if (likely(old == c))
break;
c = old;
}
return c;
}
#define ATOMIC64_INIT(i) { (i) } #define ATOMIC64_INIT(i) { (i) }
static inline long atomic64_read(const atomic64_t *v) static inline long atomic64_read(const atomic64_t *v)
...@@ -168,50 +145,8 @@ ATOMIC64_OPS(xor) ...@@ -168,50 +145,8 @@ ATOMIC64_OPS(xor)
#undef ATOMIC64_OPS #undef ATOMIC64_OPS
static inline int atomic64_add_unless(atomic64_t *v, long i, long u)
{
long c, old;
c = atomic64_read(v);
for (;;) {
if (unlikely(c == u))
break;
old = atomic64_cmpxchg(v, c, c + i);
if (likely(old == c))
break;
c = old;
}
return c != u;
}
static inline long atomic64_dec_if_positive(atomic64_t *v)
{
long c, old, dec;
c = atomic64_read(v);
for (;;) {
dec = c - 1;
if (unlikely(dec < 0))
break;
old = atomic64_cmpxchg((v), c, dec);
if (likely(old == c))
break;
c = old;
}
return dec;
}
#define atomic64_add_negative(_i, _v) (atomic64_add_return(_i, _v) < 0)
#define atomic64_inc(_v) atomic64_add(1, _v)
#define atomic64_inc_return(_v) atomic64_add_return(1, _v)
#define atomic64_inc_and_test(_v) (atomic64_add_return(1, _v) == 0)
#define atomic64_sub_return(_i, _v) atomic64_add_return(-(long)(_i), _v) #define atomic64_sub_return(_i, _v) atomic64_add_return(-(long)(_i), _v)
#define atomic64_fetch_sub(_i, _v) atomic64_fetch_add(-(long)(_i), _v) #define atomic64_fetch_sub(_i, _v) atomic64_fetch_add(-(long)(_i), _v)
#define atomic64_sub(_i, _v) atomic64_add(-(long)(_i), _v) #define atomic64_sub(_i, _v) atomic64_add(-(long)(_i), _v)
#define atomic64_sub_and_test(_i, _v) (atomic64_sub_return(_i, _v) == 0)
#define atomic64_dec(_v) atomic64_sub(1, _v)
#define atomic64_dec_return(_v) atomic64_sub_return(1, _v)
#define atomic64_dec_and_test(_v) (atomic64_sub_return(1, _v) == 0)
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
#endif /* __ARCH_S390_ATOMIC__ */ #endif /* __ARCH_S390_ATOMIC__ */
...@@ -32,44 +32,9 @@ ...@@ -32,44 +32,9 @@
#include <asm/atomic-irq.h> #include <asm/atomic-irq.h>
#endif #endif
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
#define atomic_dec_return(v) atomic_sub_return(1, (v))
#define atomic_inc_return(v) atomic_add_return(1, (v))
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
#define atomic_sub_and_test(i,v) (atomic_sub_return((i), (v)) == 0)
#define atomic_dec_and_test(v) (atomic_sub_return(1, (v)) == 0)
#define atomic_inc(v) atomic_add(1, (v))
#define atomic_dec(v) atomic_sub(1, (v))
#define atomic_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic_xchg(v, new) (xchg(&((v)->counter), new))
#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n))) #define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
/**
* __atomic_add_unless - add unless the number is a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v.
*/
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c, old;
c = atomic_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c;
}
#endif /* CONFIG_CPU_J2 */ #endif /* CONFIG_CPU_J2 */
#endif /* __ASM_SH_ATOMIC_H */ #endif /* __ASM_SH_ATOMIC_H */
...@@ -8,7 +8,8 @@ ...@@ -8,7 +8,8 @@
* This work is licensed under the terms of the GNU GPL, version 2. See the * This work is licensed under the terms of the GNU GPL, version 2. See the
* file "COPYING" in the main directory of this archive for more details. * file "COPYING" in the main directory of this archive for more details.
*/ */
#include <linux/bitops.h> #include <linux/bits.h>
#include <linux/compiler.h>
#include <asm/byteorder.h> #include <asm/byteorder.h>
/* /*
......
...@@ -27,17 +27,17 @@ int atomic_fetch_or(int, atomic_t *); ...@@ -27,17 +27,17 @@ int atomic_fetch_or(int, atomic_t *);
int atomic_fetch_xor(int, atomic_t *); int atomic_fetch_xor(int, atomic_t *);
int atomic_cmpxchg(atomic_t *, int, int); int atomic_cmpxchg(atomic_t *, int, int);
int atomic_xchg(atomic_t *, int); int atomic_xchg(atomic_t *, int);
int __atomic_add_unless(atomic_t *, int, int); int atomic_fetch_add_unless(atomic_t *, int, int);
void atomic_set(atomic_t *, int); void atomic_set(atomic_t *, int);
#define atomic_fetch_add_unless atomic_fetch_add_unless
#define atomic_set_release(v, i) atomic_set((v), (i)) #define atomic_set_release(v, i) atomic_set((v), (i))
#define atomic_read(v) READ_ONCE((v)->counter) #define atomic_read(v) READ_ONCE((v)->counter)
#define atomic_add(i, v) ((void)atomic_add_return( (int)(i), (v))) #define atomic_add(i, v) ((void)atomic_add_return( (int)(i), (v)))
#define atomic_sub(i, v) ((void)atomic_add_return(-(int)(i), (v))) #define atomic_sub(i, v) ((void)atomic_add_return(-(int)(i), (v)))
#define atomic_inc(v) ((void)atomic_add_return( 1, (v)))
#define atomic_dec(v) ((void)atomic_add_return( -1, (v)))
#define atomic_and(i, v) ((void)atomic_fetch_and((i), (v))) #define atomic_and(i, v) ((void)atomic_fetch_and((i), (v)))
#define atomic_or(i, v) ((void)atomic_fetch_or((i), (v))) #define atomic_or(i, v) ((void)atomic_fetch_or((i), (v)))
...@@ -46,22 +46,4 @@ void atomic_set(atomic_t *, int); ...@@ -46,22 +46,4 @@ void atomic_set(atomic_t *, int);
#define atomic_sub_return(i, v) (atomic_add_return(-(int)(i), (v))) #define atomic_sub_return(i, v) (atomic_add_return(-(int)(i), (v)))
#define atomic_fetch_sub(i, v) (atomic_fetch_add (-(int)(i), (v))) #define atomic_fetch_sub(i, v) (atomic_fetch_add (-(int)(i), (v)))
#define atomic_inc_return(v) (atomic_add_return( 1, (v)))
#define atomic_dec_return(v) (atomic_add_return( -1, (v)))
#define atomic_add_negative(a, v) (atomic_add_return((a), (v)) < 0)
/*
* atomic_inc_and_test - increment and test
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
#endif /* !(__ARCH_SPARC_ATOMIC__) */ #endif /* !(__ARCH_SPARC_ATOMIC__) */
...@@ -50,38 +50,6 @@ ATOMIC_OPS(xor) ...@@ -50,38 +50,6 @@ ATOMIC_OPS(xor)
#undef ATOMIC_OP_RETURN #undef ATOMIC_OP_RETURN
#undef ATOMIC_OP #undef ATOMIC_OP
#define atomic_dec_return(v) atomic_sub_return(1, v)
#define atomic64_dec_return(v) atomic64_sub_return(1, v)
#define atomic_inc_return(v) atomic_add_return(1, v)
#define atomic64_inc_return(v) atomic64_add_return(1, v)
/*
* atomic_inc_and_test - increment and test
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
#define atomic64_sub_and_test(i, v) (atomic64_sub_return(i, v) == 0)
#define atomic_dec_and_test(v) (atomic_sub_return(1, v) == 0)
#define atomic64_dec_and_test(v) (atomic64_sub_return(1, v) == 0)
#define atomic_inc(v) atomic_add(1, v)
#define atomic64_inc(v) atomic64_add(1, v)
#define atomic_dec(v) atomic_sub(1, v)
#define atomic64_dec(v) atomic64_sub(1, v)
#define atomic_add_negative(i, v) (atomic_add_return(i, v) < 0)
#define atomic64_add_negative(i, v) (atomic64_add_return(i, v) < 0)
#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n))) #define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
static inline int atomic_xchg(atomic_t *v, int new) static inline int atomic_xchg(atomic_t *v, int new)
...@@ -89,42 +57,11 @@ static inline int atomic_xchg(atomic_t *v, int new) ...@@ -89,42 +57,11 @@ static inline int atomic_xchg(atomic_t *v, int new)
return xchg(&v->counter, new); return xchg(&v->counter, new);
} }
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c, old;
c = atomic_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c;
}
#define atomic64_cmpxchg(v, o, n) \ #define atomic64_cmpxchg(v, o, n) \
((__typeof__((v)->counter))cmpxchg(&((v)->counter), (o), (n))) ((__typeof__((v)->counter))cmpxchg(&((v)->counter), (o), (n)))
#define atomic64_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic64_xchg(v, new) (xchg(&((v)->counter), new))
static inline long atomic64_add_unless(atomic64_t *v, long a, long u)
{
long c, old;
c = atomic64_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic64_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c != (u);
}
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
long atomic64_dec_if_positive(atomic64_t *v); long atomic64_dec_if_positive(atomic64_t *v);
#define atomic64_dec_if_positive atomic64_dec_if_positive
#endif /* !(__ARCH_SPARC64_ATOMIC__) */ #endif /* !(__ARCH_SPARC64_ATOMIC__) */
...@@ -95,7 +95,7 @@ int atomic_cmpxchg(atomic_t *v, int old, int new) ...@@ -95,7 +95,7 @@ int atomic_cmpxchg(atomic_t *v, int old, int new)
} }
EXPORT_SYMBOL(atomic_cmpxchg); EXPORT_SYMBOL(atomic_cmpxchg);
int __atomic_add_unless(atomic_t *v, int a, int u) int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{ {
int ret; int ret;
unsigned long flags; unsigned long flags;
...@@ -107,7 +107,7 @@ int __atomic_add_unless(atomic_t *v, int a, int u) ...@@ -107,7 +107,7 @@ int __atomic_add_unless(atomic_t *v, int a, int u)
spin_unlock_irqrestore(ATOMIC_HASH(v), flags); spin_unlock_irqrestore(ATOMIC_HASH(v), flags);
return ret; return ret;
} }
EXPORT_SYMBOL(__atomic_add_unless); EXPORT_SYMBOL(atomic_fetch_add_unless);
/* Atomic operations are already serializing */ /* Atomic operations are already serializing */
void atomic_set(atomic_t *v, int i) void atomic_set(atomic_t *v, int i)
......
...@@ -80,6 +80,7 @@ static __always_inline void arch_atomic_sub(int i, atomic_t *v) ...@@ -80,6 +80,7 @@ static __always_inline void arch_atomic_sub(int i, atomic_t *v)
* true if the result is zero, or false for all * true if the result is zero, or false for all
* other cases. * other cases.
*/ */
#define arch_atomic_sub_and_test arch_atomic_sub_and_test
static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v) static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v)
{ {
GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e); GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e);
...@@ -91,6 +92,7 @@ static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v) ...@@ -91,6 +92,7 @@ static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v)
* *
* Atomically increments @v by 1. * Atomically increments @v by 1.
*/ */
#define arch_atomic_inc arch_atomic_inc
static __always_inline void arch_atomic_inc(atomic_t *v) static __always_inline void arch_atomic_inc(atomic_t *v)
{ {
asm volatile(LOCK_PREFIX "incl %0" asm volatile(LOCK_PREFIX "incl %0"
...@@ -103,6 +105,7 @@ static __always_inline void arch_atomic_inc(atomic_t *v) ...@@ -103,6 +105,7 @@ static __always_inline void arch_atomic_inc(atomic_t *v)
* *
* Atomically decrements @v by 1. * Atomically decrements @v by 1.
*/ */
#define arch_atomic_dec arch_atomic_dec
static __always_inline void arch_atomic_dec(atomic_t *v) static __always_inline void arch_atomic_dec(atomic_t *v)
{ {
asm volatile(LOCK_PREFIX "decl %0" asm volatile(LOCK_PREFIX "decl %0"
...@@ -117,6 +120,7 @@ static __always_inline void arch_atomic_dec(atomic_t *v) ...@@ -117,6 +120,7 @@ static __always_inline void arch_atomic_dec(atomic_t *v)
* returns true if the result is 0, or false for all other * returns true if the result is 0, or false for all other
* cases. * cases.
*/ */
#define arch_atomic_dec_and_test arch_atomic_dec_and_test
static __always_inline bool arch_atomic_dec_and_test(atomic_t *v) static __always_inline bool arch_atomic_dec_and_test(atomic_t *v)
{ {
GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e); GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e);
...@@ -130,6 +134,7 @@ static __always_inline bool arch_atomic_dec_and_test(atomic_t *v) ...@@ -130,6 +134,7 @@ static __always_inline bool arch_atomic_dec_and_test(atomic_t *v)
* and returns true if the result is zero, or false for all * and returns true if the result is zero, or false for all
* other cases. * other cases.
*/ */
#define arch_atomic_inc_and_test arch_atomic_inc_and_test
static __always_inline bool arch_atomic_inc_and_test(atomic_t *v) static __always_inline bool arch_atomic_inc_and_test(atomic_t *v)
{ {
GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e); GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e);
...@@ -144,6 +149,7 @@ static __always_inline bool arch_atomic_inc_and_test(atomic_t *v) ...@@ -144,6 +149,7 @@ static __always_inline bool arch_atomic_inc_and_test(atomic_t *v)
* if the result is negative, or false when * if the result is negative, or false when
* result is greater than or equal to zero. * result is greater than or equal to zero.
*/ */
#define arch_atomic_add_negative arch_atomic_add_negative
static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v) static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v)
{ {
GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s); GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s);
...@@ -173,9 +179,6 @@ static __always_inline int arch_atomic_sub_return(int i, atomic_t *v) ...@@ -173,9 +179,6 @@ static __always_inline int arch_atomic_sub_return(int i, atomic_t *v)
return arch_atomic_add_return(-i, v); return arch_atomic_add_return(-i, v);
} }
#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v))
#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v))
static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v) static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v)
{ {
return xadd(&v->counter, i); return xadd(&v->counter, i);
...@@ -199,7 +202,7 @@ static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int n ...@@ -199,7 +202,7 @@ static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int n
static inline int arch_atomic_xchg(atomic_t *v, int new) static inline int arch_atomic_xchg(atomic_t *v, int new)
{ {
return xchg(&v->counter, new); return arch_xchg(&v->counter, new);
} }
static inline void arch_atomic_and(int i, atomic_t *v) static inline void arch_atomic_and(int i, atomic_t *v)
...@@ -253,27 +256,6 @@ static inline int arch_atomic_fetch_xor(int i, atomic_t *v) ...@@ -253,27 +256,6 @@ static inline int arch_atomic_fetch_xor(int i, atomic_t *v)
return val; return val;
} }
/**
* __arch_atomic_add_unless - add unless the number is already a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, so long as @v was not already @u.
* Returns the old value of @v.
*/
static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u)
{
int c = arch_atomic_read(v);
do {
if (unlikely(c == u))
break;
} while (!arch_atomic_try_cmpxchg(v, &c, c + a));
return c;
}
#ifdef CONFIG_X86_32 #ifdef CONFIG_X86_32
# include <asm/atomic64_32.h> # include <asm/atomic64_32.h>
#else #else
......
...@@ -158,6 +158,7 @@ static inline long long arch_atomic64_inc_return(atomic64_t *v) ...@@ -158,6 +158,7 @@ static inline long long arch_atomic64_inc_return(atomic64_t *v)
"S" (v) : "memory", "ecx"); "S" (v) : "memory", "ecx");
return a; return a;
} }
#define arch_atomic64_inc_return arch_atomic64_inc_return
static inline long long arch_atomic64_dec_return(atomic64_t *v) static inline long long arch_atomic64_dec_return(atomic64_t *v)
{ {
...@@ -166,6 +167,7 @@ static inline long long arch_atomic64_dec_return(atomic64_t *v) ...@@ -166,6 +167,7 @@ static inline long long arch_atomic64_dec_return(atomic64_t *v)
"S" (v) : "memory", "ecx"); "S" (v) : "memory", "ecx");
return a; return a;
} }
#define arch_atomic64_dec_return arch_atomic64_dec_return
/** /**
* arch_atomic64_add - add integer to atomic64 variable * arch_atomic64_add - add integer to atomic64 variable
...@@ -197,26 +199,13 @@ static inline long long arch_atomic64_sub(long long i, atomic64_t *v) ...@@ -197,26 +199,13 @@ static inline long long arch_atomic64_sub(long long i, atomic64_t *v)
return i; return i;
} }
/**
* arch_atomic64_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
* Atomically subtracts @i from @v and returns
* true if the result is zero, or false for all
* other cases.
*/
static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v)
{
return arch_atomic64_sub_return(i, v) == 0;
}
/** /**
* arch_atomic64_inc - increment atomic64 variable * arch_atomic64_inc - increment atomic64 variable
* @v: pointer to type atomic64_t * @v: pointer to type atomic64_t
* *
* Atomically increments @v by 1. * Atomically increments @v by 1.
*/ */
#define arch_atomic64_inc arch_atomic64_inc
static inline void arch_atomic64_inc(atomic64_t *v) static inline void arch_atomic64_inc(atomic64_t *v)
{ {
__alternative_atomic64(inc, inc_return, /* no output */, __alternative_atomic64(inc, inc_return, /* no output */,
...@@ -229,52 +218,13 @@ static inline void arch_atomic64_inc(atomic64_t *v) ...@@ -229,52 +218,13 @@ static inline void arch_atomic64_inc(atomic64_t *v)
* *
* Atomically decrements @v by 1. * Atomically decrements @v by 1.
*/ */
#define arch_atomic64_dec arch_atomic64_dec
static inline void arch_atomic64_dec(atomic64_t *v) static inline void arch_atomic64_dec(atomic64_t *v)
{ {
__alternative_atomic64(dec, dec_return, /* no output */, __alternative_atomic64(dec, dec_return, /* no output */,
"S" (v) : "memory", "eax", "ecx", "edx"); "S" (v) : "memory", "eax", "ecx", "edx");
} }
/**
* arch_atomic64_dec_and_test - decrement and test
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
static inline int arch_atomic64_dec_and_test(atomic64_t *v)
{
return arch_atomic64_dec_return(v) == 0;
}
/**
* atomic64_inc_and_test - increment and test
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
static inline int arch_atomic64_inc_and_test(atomic64_t *v)
{
return arch_atomic64_inc_return(v) == 0;
}
/**
* arch_atomic64_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v and returns true
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
static inline int arch_atomic64_add_negative(long long i, atomic64_t *v)
{
return arch_atomic64_add_return(i, v) < 0;
}
/** /**
* arch_atomic64_add_unless - add unless the number is a given value * arch_atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t * @v: pointer of type atomic64_t
...@@ -295,7 +245,7 @@ static inline int arch_atomic64_add_unless(atomic64_t *v, long long a, ...@@ -295,7 +245,7 @@ static inline int arch_atomic64_add_unless(atomic64_t *v, long long a,
return (int)a; return (int)a;
} }
#define arch_atomic64_inc_not_zero arch_atomic64_inc_not_zero
static inline int arch_atomic64_inc_not_zero(atomic64_t *v) static inline int arch_atomic64_inc_not_zero(atomic64_t *v)
{ {
int r; int r;
...@@ -304,6 +254,7 @@ static inline int arch_atomic64_inc_not_zero(atomic64_t *v) ...@@ -304,6 +254,7 @@ static inline int arch_atomic64_inc_not_zero(atomic64_t *v)
return r; return r;
} }
#define arch_atomic64_dec_if_positive arch_atomic64_dec_if_positive
static inline long long arch_atomic64_dec_if_positive(atomic64_t *v) static inline long long arch_atomic64_dec_if_positive(atomic64_t *v)
{ {
long long r; long long r;
......
...@@ -71,6 +71,7 @@ static inline void arch_atomic64_sub(long i, atomic64_t *v) ...@@ -71,6 +71,7 @@ static inline void arch_atomic64_sub(long i, atomic64_t *v)
* true if the result is zero, or false for all * true if the result is zero, or false for all
* other cases. * other cases.
*/ */
#define arch_atomic64_sub_and_test arch_atomic64_sub_and_test
static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v) static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v)
{ {
GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e); GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e);
...@@ -82,6 +83,7 @@ static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v) ...@@ -82,6 +83,7 @@ static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v)
* *
* Atomically increments @v by 1. * Atomically increments @v by 1.
*/ */
#define arch_atomic64_inc arch_atomic64_inc
static __always_inline void arch_atomic64_inc(atomic64_t *v) static __always_inline void arch_atomic64_inc(atomic64_t *v)
{ {
asm volatile(LOCK_PREFIX "incq %0" asm volatile(LOCK_PREFIX "incq %0"
...@@ -95,6 +97,7 @@ static __always_inline void arch_atomic64_inc(atomic64_t *v) ...@@ -95,6 +97,7 @@ static __always_inline void arch_atomic64_inc(atomic64_t *v)
* *
* Atomically decrements @v by 1. * Atomically decrements @v by 1.
*/ */
#define arch_atomic64_dec arch_atomic64_dec
static __always_inline void arch_atomic64_dec(atomic64_t *v) static __always_inline void arch_atomic64_dec(atomic64_t *v)
{ {
asm volatile(LOCK_PREFIX "decq %0" asm volatile(LOCK_PREFIX "decq %0"
...@@ -110,6 +113,7 @@ static __always_inline void arch_atomic64_dec(atomic64_t *v) ...@@ -110,6 +113,7 @@ static __always_inline void arch_atomic64_dec(atomic64_t *v)
* returns true if the result is 0, or false for all other * returns true if the result is 0, or false for all other
* cases. * cases.
*/ */
#define arch_atomic64_dec_and_test arch_atomic64_dec_and_test
static inline bool arch_atomic64_dec_and_test(atomic64_t *v) static inline bool arch_atomic64_dec_and_test(atomic64_t *v)
{ {
GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e); GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e);
...@@ -123,6 +127,7 @@ static inline bool arch_atomic64_dec_and_test(atomic64_t *v) ...@@ -123,6 +127,7 @@ static inline bool arch_atomic64_dec_and_test(atomic64_t *v)
* and returns true if the result is zero, or false for all * and returns true if the result is zero, or false for all
* other cases. * other cases.
*/ */
#define arch_atomic64_inc_and_test arch_atomic64_inc_and_test
static inline bool arch_atomic64_inc_and_test(atomic64_t *v) static inline bool arch_atomic64_inc_and_test(atomic64_t *v)
{ {
GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e); GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e);
...@@ -137,6 +142,7 @@ static inline bool arch_atomic64_inc_and_test(atomic64_t *v) ...@@ -137,6 +142,7 @@ static inline bool arch_atomic64_inc_and_test(atomic64_t *v)
* if the result is negative, or false when * if the result is negative, or false when
* result is greater than or equal to zero. * result is greater than or equal to zero.
*/ */
#define arch_atomic64_add_negative arch_atomic64_add_negative
static inline bool arch_atomic64_add_negative(long i, atomic64_t *v) static inline bool arch_atomic64_add_negative(long i, atomic64_t *v)
{ {
GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s); GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s);
...@@ -169,9 +175,6 @@ static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v) ...@@ -169,9 +175,6 @@ static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v)
return xadd(&v->counter, -i); return xadd(&v->counter, -i);
} }
#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v)))
#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v)))
static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new) static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new)
{ {
return arch_cmpxchg(&v->counter, old, new); return arch_cmpxchg(&v->counter, old, new);
...@@ -185,46 +188,7 @@ static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, l ...@@ -185,46 +188,7 @@ static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, l
static inline long arch_atomic64_xchg(atomic64_t *v, long new) static inline long arch_atomic64_xchg(atomic64_t *v, long new)
{ {
return xchg(&v->counter, new); return arch_xchg(&v->counter, new);
}
/**
* arch_atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v.
*/
static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u)
{
s64 c = arch_atomic64_read(v);
do {
if (unlikely(c == u))
return false;
} while (!arch_atomic64_try_cmpxchg(v, &c, c + a));
return true;
}
#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0)
/*
* arch_atomic64_dec_if_positive - decrement by 1 if old value positive
* @v: pointer of type atomic_t
*
* The function returns the old value of *v minus 1, even if
* the atomic variable, v, was not decremented.
*/
static inline long arch_atomic64_dec_if_positive(atomic64_t *v)
{
s64 dec, c = arch_atomic64_read(v);
do {
dec = c - 1;
if (unlikely(dec < 0))
break;
} while (!arch_atomic64_try_cmpxchg(v, &c, dec));
return dec;
} }
static inline void arch_atomic64_and(long i, atomic64_t *v) static inline void arch_atomic64_and(long i, atomic64_t *v)
......
...@@ -75,7 +75,7 @@ extern void __add_wrong_size(void) ...@@ -75,7 +75,7 @@ extern void __add_wrong_size(void)
* use "asm volatile" and "memory" clobbers to prevent gcc from moving * use "asm volatile" and "memory" clobbers to prevent gcc from moving
* information around. * information around.
*/ */
#define xchg(ptr, v) __xchg_op((ptr), (v), xchg, "") #define arch_xchg(ptr, v) __xchg_op((ptr), (v), xchg, "")
/* /*
* Atomic compare and exchange. Compare OLD with MEM, if identical, * Atomic compare and exchange. Compare OLD with MEM, if identical,
......
...@@ -10,13 +10,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) ...@@ -10,13 +10,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val)
#define arch_cmpxchg64(ptr, o, n) \ #define arch_cmpxchg64(ptr, o, n) \
({ \ ({ \
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
cmpxchg((ptr), (o), (n)); \ arch_cmpxchg((ptr), (o), (n)); \
}) })
#define arch_cmpxchg64_local(ptr, o, n) \ #define arch_cmpxchg64_local(ptr, o, n) \
({ \ ({ \
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
cmpxchg_local((ptr), (o), (n)); \ arch_cmpxchg_local((ptr), (o), (n)); \
}) })
#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) #define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16)
......
...@@ -5,6 +5,7 @@ ...@@ -5,6 +5,7 @@
* PaX/grsecurity. * PaX/grsecurity.
*/ */
#include <linux/refcount.h> #include <linux/refcount.h>
#include <asm/bug.h>
/* /*
* This is the first portion of the refcount error handling, which lives in * This is the first portion of the refcount error handling, which lives in
......
...@@ -197,107 +197,9 @@ ATOMIC_OPS(xor) ...@@ -197,107 +197,9 @@ ATOMIC_OPS(xor)
#undef ATOMIC_OP_RETURN #undef ATOMIC_OP_RETURN
#undef ATOMIC_OP #undef ATOMIC_OP
/**
* atomic_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer of type atomic_t
*
* Atomically subtracts @i from @v and returns
* true if the result is zero, or false for all
* other cases.
*/
#define atomic_sub_and_test(i,v) (atomic_sub_return((i),(v)) == 0)
/**
* atomic_inc - increment atomic variable
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1.
*/
#define atomic_inc(v) atomic_add(1,(v))
/**
* atomic_inc - increment atomic variable
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1.
*/
#define atomic_inc_return(v) atomic_add_return(1,(v))
/**
* atomic_dec - decrement atomic variable
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1.
*/
#define atomic_dec(v) atomic_sub(1,(v))
/**
* atomic_dec_return - decrement atomic variable
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1.
*/
#define atomic_dec_return(v) atomic_sub_return(1,(v))
/**
* atomic_dec_and_test - decrement and test
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
#define atomic_dec_and_test(v) (atomic_sub_return(1,(v)) == 0)
/**
* atomic_inc_and_test - increment and test
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
#define atomic_inc_and_test(v) (atomic_add_return(1,(v)) == 0)
/**
* atomic_add_negative - add and test if negative
* @v: pointer of type atomic_t
* @i: integer value to add
*
* Atomically adds @i to @v and returns true
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
#define atomic_add_negative(i,v) (atomic_add_return((i),(v)) < 0)
#define atomic_cmpxchg(v, o, n) ((int)cmpxchg(&((v)->counter), (o), (n))) #define atomic_cmpxchg(v, o, n) ((int)cmpxchg(&((v)->counter), (o), (n)))
#define atomic_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic_xchg(v, new) (xchg(&((v)->counter), new))
/**
* __atomic_add_unless - add unless the number is a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v.
*/
static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c, old;
c = atomic_read(v);
for (;;) {
if (unlikely(c == (u)))
break;
old = atomic_cmpxchg((v), c, c + (a));
if (likely(old == c))
break;
c = old;
}
return c;
}
#endif /* __KERNEL__ */ #endif /* __KERNEL__ */
#endif /* _XTENSA_ATOMIC_H */ #endif /* _XTENSA_ATOMIC_H */
...@@ -61,7 +61,7 @@ static int atomic_inc_return_safe(atomic_t *v) ...@@ -61,7 +61,7 @@ static int atomic_inc_return_safe(atomic_t *v)
{ {
unsigned int counter; unsigned int counter;
counter = (unsigned int)__atomic_add_unless(v, 1, 0); counter = (unsigned int)atomic_fetch_add_unless(v, 1, 0);
if (counter <= (unsigned int)INT_MAX) if (counter <= (unsigned int)INT_MAX)
return (int)counter; return (int)counter;
......
...@@ -121,7 +121,7 @@ static int uverbs_try_lock_object(struct ib_uobject *uobj, bool exclusive) ...@@ -121,7 +121,7 @@ static int uverbs_try_lock_object(struct ib_uobject *uobj, bool exclusive)
* this lock. * this lock.
*/ */
if (!exclusive) if (!exclusive)
return __atomic_add_unless(&uobj->usecnt, 1, -1) == -1 ? return atomic_fetch_add_unless(&uobj->usecnt, 1, -1) == -1 ?
-EBUSY : 0; -EBUSY : 0;
/* lock is either WRITE or DESTROY - should be exclusive */ /* lock is either WRITE or DESTROY - should be exclusive */
......
...@@ -648,7 +648,7 @@ static void afs_wake_up_async_call(struct sock *sk, struct rxrpc_call *rxcall, ...@@ -648,7 +648,7 @@ static void afs_wake_up_async_call(struct sock *sk, struct rxrpc_call *rxcall,
trace_afs_notify_call(rxcall, call); trace_afs_notify_call(rxcall, call);
call->need_attention = true; call->need_attention = true;
u = __atomic_add_unless(&call->usage, 1, 0); u = atomic_fetch_add_unless(&call->usage, 1, 0);
if (u != 0) { if (u != 0) {
trace_afs_call(call, afs_call_trace_wake, u, trace_afs_call(call, afs_call_trace_wake, u,
atomic_read(&call->net->nr_outstanding_calls), atomic_read(&call->net->nr_outstanding_calls),
......
...@@ -84,42 +84,59 @@ static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 ne ...@@ -84,42 +84,59 @@ static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 ne
} }
#endif #endif
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) #ifdef arch_atomic_fetch_add_unless
#define atomic_fetch_add_unless atomic_fetch_add_unless
static __always_inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return __arch_atomic_add_unless(v, a, u); return arch_atomic_fetch_add_unless(v, a, u);
} }
#endif
#ifdef arch_atomic64_fetch_add_unless
static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u) #define atomic64_fetch_add_unless atomic64_fetch_add_unless
static __always_inline s64 atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_unless(v, a, u); return arch_atomic64_fetch_add_unless(v, a, u);
} }
#endif
#ifdef arch_atomic_inc
#define atomic_inc atomic_inc
static __always_inline void atomic_inc(atomic_t *v) static __always_inline void atomic_inc(atomic_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
arch_atomic_inc(v); arch_atomic_inc(v);
} }
#endif
#ifdef arch_atomic64_inc
#define atomic64_inc atomic64_inc
static __always_inline void atomic64_inc(atomic64_t *v) static __always_inline void atomic64_inc(atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
arch_atomic64_inc(v); arch_atomic64_inc(v);
} }
#endif
#ifdef arch_atomic_dec
#define atomic_dec atomic_dec
static __always_inline void atomic_dec(atomic_t *v) static __always_inline void atomic_dec(atomic_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
arch_atomic_dec(v); arch_atomic_dec(v);
} }
#endif
#ifdef atch_atomic64_dec
#define atomic64_dec
static __always_inline void atomic64_dec(atomic64_t *v) static __always_inline void atomic64_dec(atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
arch_atomic64_dec(v); arch_atomic64_dec(v);
} }
#endif
static __always_inline void atomic_add(int i, atomic_t *v) static __always_inline void atomic_add(int i, atomic_t *v)
{ {
...@@ -181,65 +198,95 @@ static __always_inline void atomic64_xor(s64 i, atomic64_t *v) ...@@ -181,65 +198,95 @@ static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
arch_atomic64_xor(i, v); arch_atomic64_xor(i, v);
} }
#ifdef arch_atomic_inc_return
#define atomic_inc_return atomic_inc_return
static __always_inline int atomic_inc_return(atomic_t *v) static __always_inline int atomic_inc_return(atomic_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic_inc_return(v); return arch_atomic_inc_return(v);
} }
#endif
#ifdef arch_atomic64_in_return
#define atomic64_inc_return atomic64_inc_return
static __always_inline s64 atomic64_inc_return(atomic64_t *v) static __always_inline s64 atomic64_inc_return(atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_return(v); return arch_atomic64_inc_return(v);
} }
#endif
#ifdef arch_atomic_dec_return
#define atomic_dec_return atomic_dec_return
static __always_inline int atomic_dec_return(atomic_t *v) static __always_inline int atomic_dec_return(atomic_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic_dec_return(v); return arch_atomic_dec_return(v);
} }
#endif
#ifdef arch_atomic64_dec_return
#define atomic64_dec_return atomic64_dec_return
static __always_inline s64 atomic64_dec_return(atomic64_t *v) static __always_inline s64 atomic64_dec_return(atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_return(v); return arch_atomic64_dec_return(v);
} }
#endif
static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v) #ifdef arch_atomic64_inc_not_zero
#define atomic64_inc_not_zero atomic64_inc_not_zero
static __always_inline bool atomic64_inc_not_zero(atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_not_zero(v); return arch_atomic64_inc_not_zero(v);
} }
#endif
#ifdef arch_atomic64_dec_if_positive
#define atomic64_dec_if_positive atomic64_dec_if_positive
static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v) static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_if_positive(v); return arch_atomic64_dec_if_positive(v);
} }
#endif
#ifdef arch_atomic_dec_and_test
#define atomic_dec_and_test atomic_dec_and_test
static __always_inline bool atomic_dec_and_test(atomic_t *v) static __always_inline bool atomic_dec_and_test(atomic_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic_dec_and_test(v); return arch_atomic_dec_and_test(v);
} }
#endif
#ifdef arch_atomic64_dec_and_test
#define atomic64_dec_and_test atomic64_dec_and_test
static __always_inline bool atomic64_dec_and_test(atomic64_t *v) static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_and_test(v); return arch_atomic64_dec_and_test(v);
} }
#endif
#ifdef arch_atomic_inc_and_test
#define atomic_inc_and_test atomic_inc_and_test
static __always_inline bool atomic_inc_and_test(atomic_t *v) static __always_inline bool atomic_inc_and_test(atomic_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic_inc_and_test(v); return arch_atomic_inc_and_test(v);
} }
#endif
#ifdef arch_atomic64_inc_and_test
#define atomic64_inc_and_test atomic64_inc_and_test
static __always_inline bool atomic64_inc_and_test(atomic64_t *v) static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_and_test(v); return arch_atomic64_inc_and_test(v);
} }
#endif
static __always_inline int atomic_add_return(int i, atomic_t *v) static __always_inline int atomic_add_return(int i, atomic_t *v)
{ {
...@@ -325,152 +372,96 @@ static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v) ...@@ -325,152 +372,96 @@ static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
return arch_atomic64_fetch_xor(i, v); return arch_atomic64_fetch_xor(i, v);
} }
#ifdef arch_atomic_sub_and_test
#define atomic_sub_and_test atomic_sub_and_test
static __always_inline bool atomic_sub_and_test(int i, atomic_t *v) static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic_sub_and_test(i, v); return arch_atomic_sub_and_test(i, v);
} }
#endif
#ifdef arch_atomic64_sub_and_test
#define atomic64_sub_and_test atomic64_sub_and_test
static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v) static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic64_sub_and_test(i, v); return arch_atomic64_sub_and_test(i, v);
} }
#endif
#ifdef arch_atomic_add_negative
#define atomic_add_negative atomic_add_negative
static __always_inline bool atomic_add_negative(int i, atomic_t *v) static __always_inline bool atomic_add_negative(int i, atomic_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic_add_negative(i, v); return arch_atomic_add_negative(i, v);
} }
#endif
#ifdef arch_atomic64_add_negative
#define atomic64_add_negative atomic64_add_negative
static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
{ {
kasan_check_write(v, sizeof(*v)); kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_negative(i, v); return arch_atomic64_add_negative(i, v);
} }
#endif
static __always_inline unsigned long #define xchg(ptr, new) \
cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new, int size) ({ \
{ typeof(ptr) __ai_ptr = (ptr); \
kasan_check_write(ptr, size); kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
switch (size) { arch_xchg(__ai_ptr, (new)); \
case 1: })
return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
case 2:
return arch_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
case 4:
return arch_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
case 8:
BUILD_BUG_ON(sizeof(unsigned long) != 8);
return arch_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
}
BUILD_BUG();
return 0;
}
#define cmpxchg(ptr, old, new) \ #define cmpxchg(ptr, old, new) \
({ \ ({ \
((__typeof__(*(ptr)))cmpxchg_size((ptr), (unsigned long)(old), \ typeof(ptr) __ai_ptr = (ptr); \
(unsigned long)(new), sizeof(*(ptr)))); \ kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg(__ai_ptr, (old), (new)); \
}) })
static __always_inline unsigned long
sync_cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new,
int size)
{
kasan_check_write(ptr, size);
switch (size) {
case 1:
return arch_sync_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
case 2:
return arch_sync_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
case 4:
return arch_sync_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
case 8:
BUILD_BUG_ON(sizeof(unsigned long) != 8);
return arch_sync_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
}
BUILD_BUG();
return 0;
}
#define sync_cmpxchg(ptr, old, new) \ #define sync_cmpxchg(ptr, old, new) \
({ \ ({ \
((__typeof__(*(ptr)))sync_cmpxchg_size((ptr), \ typeof(ptr) __ai_ptr = (ptr); \
(unsigned long)(old), (unsigned long)(new), \ kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
sizeof(*(ptr)))); \ arch_sync_cmpxchg(__ai_ptr, (old), (new)); \
}) })
static __always_inline unsigned long
cmpxchg_local_size(volatile void *ptr, unsigned long old, unsigned long new,
int size)
{
kasan_check_write(ptr, size);
switch (size) {
case 1:
return arch_cmpxchg_local((u8 *)ptr, (u8)old, (u8)new);
case 2:
return arch_cmpxchg_local((u16 *)ptr, (u16)old, (u16)new);
case 4:
return arch_cmpxchg_local((u32 *)ptr, (u32)old, (u32)new);
case 8:
BUILD_BUG_ON(sizeof(unsigned long) != 8);
return arch_cmpxchg_local((u64 *)ptr, (u64)old, (u64)new);
}
BUILD_BUG();
return 0;
}
#define cmpxchg_local(ptr, old, new) \ #define cmpxchg_local(ptr, old, new) \
({ \ ({ \
((__typeof__(*(ptr)))cmpxchg_local_size((ptr), \ typeof(ptr) __ai_ptr = (ptr); \
(unsigned long)(old), (unsigned long)(new), \ kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
sizeof(*(ptr)))); \ arch_cmpxchg_local(__ai_ptr, (old), (new)); \
}) })
static __always_inline u64
cmpxchg64_size(volatile u64 *ptr, u64 old, u64 new)
{
kasan_check_write(ptr, sizeof(*ptr));
return arch_cmpxchg64(ptr, old, new);
}
#define cmpxchg64(ptr, old, new) \ #define cmpxchg64(ptr, old, new) \
({ \ ({ \
((__typeof__(*(ptr)))cmpxchg64_size((ptr), (u64)(old), \ typeof(ptr) __ai_ptr = (ptr); \
(u64)(new))); \ kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg64(__ai_ptr, (old), (new)); \
}) })
static __always_inline u64
cmpxchg64_local_size(volatile u64 *ptr, u64 old, u64 new)
{
kasan_check_write(ptr, sizeof(*ptr));
return arch_cmpxchg64_local(ptr, old, new);
}
#define cmpxchg64_local(ptr, old, new) \ #define cmpxchg64_local(ptr, old, new) \
({ \ ({ \
((__typeof__(*(ptr)))cmpxchg64_local_size((ptr), (u64)(old), \ typeof(ptr) __ai_ptr = (ptr); \
(u64)(new))); \ kasan_check_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg64_local(__ai_ptr, (old), (new)); \
}) })
/*
* Originally we had the following code here:
* __typeof__(p1) ____p1 = (p1);
* kasan_check_write(____p1, 2 * sizeof(*____p1));
* arch_cmpxchg_double(____p1, (p2), (o1), (o2), (n1), (n2));
* But it leads to compilation failures (see gcc issue 72873).
* So for now it's left non-instrumented.
* There are few callers of cmpxchg_double(), so it's not critical.
*/
#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
({ \ ({ \
arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \ typeof(p1) __ai_p1 = (p1); \
kasan_check_write(__ai_p1, 2 * sizeof(*__ai_p1)); \
arch_cmpxchg_double(__ai_p1, (p2), (o1), (o2), (n1), (n2)); \
}) })
#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ #define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
({ \ ({ \
arch_cmpxchg_double_local((p1), (p2), (o1), (o2), (n1), (n2)); \ typeof(p1) __ai_p1 = (p1); \
kasan_check_write(__ai_p1, 2 * sizeof(*__ai_p1)); \
arch_cmpxchg_double_local(__ai_p1, (p2), (o1), (o2), (n1), (n2)); \
}) })
#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
...@@ -186,11 +186,6 @@ ATOMIC_OP(xor, ^) ...@@ -186,11 +186,6 @@ ATOMIC_OP(xor, ^)
#include <linux/irqflags.h> #include <linux/irqflags.h>
static inline int atomic_add_negative(int i, atomic_t *v)
{
return atomic_add_return(i, v) < 0;
}
static inline void atomic_add(int i, atomic_t *v) static inline void atomic_add(int i, atomic_t *v)
{ {
atomic_add_return(i, v); atomic_add_return(i, v);
...@@ -201,35 +196,7 @@ static inline void atomic_sub(int i, atomic_t *v) ...@@ -201,35 +196,7 @@ static inline void atomic_sub(int i, atomic_t *v)
atomic_sub_return(i, v); atomic_sub_return(i, v);
} }
static inline void atomic_inc(atomic_t *v)
{
atomic_add_return(1, v);
}
static inline void atomic_dec(atomic_t *v)
{
atomic_sub_return(1, v);
}
#define atomic_dec_return(v) atomic_sub_return(1, (v))
#define atomic_inc_return(v) atomic_add_return(1, (v))
#define atomic_sub_and_test(i, v) (atomic_sub_return((i), (v)) == 0)
#define atomic_dec_and_test(v) (atomic_dec_return(v) == 0)
#define atomic_inc_and_test(v) (atomic_inc_return(v) == 0)
#define atomic_xchg(ptr, v) (xchg(&(ptr)->counter, (v))) #define atomic_xchg(ptr, v) (xchg(&(ptr)->counter, (v)))
#define atomic_cmpxchg(v, old, new) (cmpxchg(&((v)->counter), (old), (new))) #define atomic_cmpxchg(v, old, new) (cmpxchg(&((v)->counter), (old), (new)))
#ifndef __atomic_add_unless
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c, old;
c = atomic_read(v);
while (c != u && (old = atomic_cmpxchg(v, c, c + a)) != c)
c = old;
return c;
}
#endif
#endif /* __ASM_GENERIC_ATOMIC_H */ #endif /* __ASM_GENERIC_ATOMIC_H */
...@@ -11,6 +11,7 @@ ...@@ -11,6 +11,7 @@
*/ */
#ifndef _ASM_GENERIC_ATOMIC64_H #ifndef _ASM_GENERIC_ATOMIC64_H
#define _ASM_GENERIC_ATOMIC64_H #define _ASM_GENERIC_ATOMIC64_H
#include <linux/types.h>
typedef struct { typedef struct {
long long counter; long long counter;
...@@ -50,18 +51,10 @@ ATOMIC64_OPS(xor) ...@@ -50,18 +51,10 @@ ATOMIC64_OPS(xor)
#undef ATOMIC64_OP #undef ATOMIC64_OP
extern long long atomic64_dec_if_positive(atomic64_t *v); extern long long atomic64_dec_if_positive(atomic64_t *v);
#define atomic64_dec_if_positive atomic64_dec_if_positive
extern long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n); extern long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n);
extern long long atomic64_xchg(atomic64_t *v, long long new); extern long long atomic64_xchg(atomic64_t *v, long long new);
extern int atomic64_add_unless(atomic64_t *v, long long a, long long u); extern long long atomic64_fetch_add_unless(atomic64_t *v, long long a, long long u);
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
#define atomic64_add_negative(a, v) (atomic64_add_return((a), (v)) < 0)
#define atomic64_inc(v) atomic64_add(1LL, (v))
#define atomic64_inc_return(v) atomic64_add_return(1LL, (v))
#define atomic64_inc_and_test(v) (atomic64_inc_return(v) == 0)
#define atomic64_sub_and_test(a, v) (atomic64_sub_return((a), (v)) == 0)
#define atomic64_dec(v) atomic64_sub(1LL, (v))
#define atomic64_dec_return(v) atomic64_sub_return(1LL, (v))
#define atomic64_dec_and_test(v) (atomic64_dec_return((v)) == 0)
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1LL, 0LL)
#endif /* _ASM_GENERIC_ATOMIC64_H */ #endif /* _ASM_GENERIC_ATOMIC64_H */
...@@ -2,189 +2,67 @@ ...@@ -2,189 +2,67 @@
#ifndef _ASM_GENERIC_BITOPS_ATOMIC_H_ #ifndef _ASM_GENERIC_BITOPS_ATOMIC_H_
#define _ASM_GENERIC_BITOPS_ATOMIC_H_ #define _ASM_GENERIC_BITOPS_ATOMIC_H_
#include <asm/types.h> #include <linux/atomic.h>
#include <linux/irqflags.h> #include <linux/compiler.h>
#include <asm/barrier.h>
#ifdef CONFIG_SMP
#include <asm/spinlock.h>
#include <asm/cache.h> /* we use L1_CACHE_BYTES */
/* Use an array of spinlocks for our atomic_ts.
* Hash function to index into a different SPINLOCK.
* Since "a" is usually an address, use one spinlock per cacheline.
*/
# define ATOMIC_HASH_SIZE 4
# define ATOMIC_HASH(a) (&(__atomic_hash[ (((unsigned long) a)/L1_CACHE_BYTES) & (ATOMIC_HASH_SIZE-1) ]))
extern arch_spinlock_t __atomic_hash[ATOMIC_HASH_SIZE] __lock_aligned;
/* Can't use raw_spin_lock_irq because of #include problems, so
* this is the substitute */
#define _atomic_spin_lock_irqsave(l,f) do { \
arch_spinlock_t *s = ATOMIC_HASH(l); \
local_irq_save(f); \
arch_spin_lock(s); \
} while(0)
#define _atomic_spin_unlock_irqrestore(l,f) do { \
arch_spinlock_t *s = ATOMIC_HASH(l); \
arch_spin_unlock(s); \
local_irq_restore(f); \
} while(0)
#else
# define _atomic_spin_lock_irqsave(l,f) do { local_irq_save(f); } while (0)
# define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore(f); } while (0)
#endif
/* /*
* NMI events can occur at any time, including when interrupts have been * Implementation of atomic bitops using atomic-fetch ops.
* disabled by *_irqsave(). So you can get NMI events occurring while a * See Documentation/atomic_bitops.txt for details.
* *_bit function is holding a spin lock. If the NMI handler also wants
* to do bit manipulation (and they do) then you can get a deadlock
* between the original caller of *_bit() and the NMI handler.
*
* by Keith Owens
*/ */
/** static inline void set_bit(unsigned int nr, volatile unsigned long *p)
* set_bit - Atomically set a bit in memory
* @nr: the bit to set
* @addr: the address to start counting from
*
* This function is atomic and may not be reordered. See __set_bit()
* if you do not require the atomic guarantees.
*
* Note: there are no guarantees that this function will not be reordered
* on non x86 architectures, so if you are writing portable code,
* make sure not to rely on its reordering guarantees.
*
* Note that @nr may be almost arbitrarily large; this function is not
* restricted to acting on a single-word quantity.
*/
static inline void set_bit(int nr, volatile unsigned long *addr)
{ {
unsigned long mask = BIT_MASK(nr); p += BIT_WORD(nr);
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); atomic_long_or(BIT_MASK(nr), (atomic_long_t *)p);
unsigned long flags;
_atomic_spin_lock_irqsave(p, flags);
*p |= mask;
_atomic_spin_unlock_irqrestore(p, flags);
} }
/** static inline void clear_bit(unsigned int nr, volatile unsigned long *p)
* clear_bit - Clears a bit in memory
* @nr: Bit to clear
* @addr: Address to start counting from
*
* clear_bit() is atomic and may not be reordered. However, it does
* not contain a memory barrier, so if it is used for locking purposes,
* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic()
* in order to ensure changes are visible on other processors.
*/
static inline void clear_bit(int nr, volatile unsigned long *addr)
{ {
unsigned long mask = BIT_MASK(nr); p += BIT_WORD(nr);
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); atomic_long_andnot(BIT_MASK(nr), (atomic_long_t *)p);
unsigned long flags;
_atomic_spin_lock_irqsave(p, flags);
*p &= ~mask;
_atomic_spin_unlock_irqrestore(p, flags);
} }
/** static inline void change_bit(unsigned int nr, volatile unsigned long *p)
* change_bit - Toggle a bit in memory
* @nr: Bit to change
* @addr: Address to start counting from
*
* change_bit() is atomic and may not be reordered. It may be
* reordered on other architectures than x86.
* Note that @nr may be almost arbitrarily large; this function is not
* restricted to acting on a single-word quantity.
*/
static inline void change_bit(int nr, volatile unsigned long *addr)
{ {
unsigned long mask = BIT_MASK(nr); p += BIT_WORD(nr);
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr); atomic_long_xor(BIT_MASK(nr), (atomic_long_t *)p);
unsigned long flags;
_atomic_spin_lock_irqsave(p, flags);
*p ^= mask;
_atomic_spin_unlock_irqrestore(p, flags);
} }
/** static inline int test_and_set_bit(unsigned int nr, volatile unsigned long *p)
* test_and_set_bit - Set a bit and return its old value
* @nr: Bit to set
* @addr: Address to count from
*
* This operation is atomic and cannot be reordered.
* It may be reordered on other architectures than x86.
* It also implies a memory barrier.
*/
static inline int test_and_set_bit(int nr, volatile unsigned long *addr)
{ {
long old;
unsigned long mask = BIT_MASK(nr); unsigned long mask = BIT_MASK(nr);
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr);
unsigned long old;
unsigned long flags;
_atomic_spin_lock_irqsave(p, flags); p += BIT_WORD(nr);
old = *p; if (READ_ONCE(*p) & mask)
*p = old | mask; return 1;
_atomic_spin_unlock_irqrestore(p, flags);
return (old & mask) != 0; old = atomic_long_fetch_or(mask, (atomic_long_t *)p);
return !!(old & mask);
} }
/** static inline int test_and_clear_bit(unsigned int nr, volatile unsigned long *p)
* test_and_clear_bit - Clear a bit and return its old value
* @nr: Bit to clear
* @addr: Address to count from
*
* This operation is atomic and cannot be reordered.
* It can be reorderdered on other architectures other than x86.
* It also implies a memory barrier.
*/
static inline int test_and_clear_bit(int nr, volatile unsigned long *addr)
{ {
long old;
unsigned long mask = BIT_MASK(nr); unsigned long mask = BIT_MASK(nr);
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr);
unsigned long old;
unsigned long flags;
_atomic_spin_lock_irqsave(p, flags); p += BIT_WORD(nr);
old = *p; if (!(READ_ONCE(*p) & mask))
*p = old & ~mask; return 0;
_atomic_spin_unlock_irqrestore(p, flags);
return (old & mask) != 0; old = atomic_long_fetch_andnot(mask, (atomic_long_t *)p);
return !!(old & mask);
} }
/** static inline int test_and_change_bit(unsigned int nr, volatile unsigned long *p)
* test_and_change_bit - Change a bit and return its old value
* @nr: Bit to change
* @addr: Address to count from
*
* This operation is atomic and cannot be reordered.
* It also implies a memory barrier.
*/
static inline int test_and_change_bit(int nr, volatile unsigned long *addr)
{ {
long old;
unsigned long mask = BIT_MASK(nr); unsigned long mask = BIT_MASK(nr);
unsigned long *p = ((unsigned long *)addr) + BIT_WORD(nr);
unsigned long old;
unsigned long flags;
_atomic_spin_lock_irqsave(p, flags);
old = *p;
*p = old ^ mask;
_atomic_spin_unlock_irqrestore(p, flags);
return (old & mask) != 0; p += BIT_WORD(nr);
old = atomic_long_fetch_xor(mask, (atomic_long_t *)p);
return !!(old & mask);
} }
#endif /* _ASM_GENERIC_BITOPS_ATOMIC_H */ #endif /* _ASM_GENERIC_BITOPS_ATOMIC_H */
...@@ -2,6 +2,10 @@ ...@@ -2,6 +2,10 @@
#ifndef _ASM_GENERIC_BITOPS_LOCK_H_ #ifndef _ASM_GENERIC_BITOPS_LOCK_H_
#define _ASM_GENERIC_BITOPS_LOCK_H_ #define _ASM_GENERIC_BITOPS_LOCK_H_
#include <linux/atomic.h>
#include <linux/compiler.h>
#include <asm/barrier.h>
/** /**
* test_and_set_bit_lock - Set a bit and return its old value, for lock * test_and_set_bit_lock - Set a bit and return its old value, for lock
* @nr: Bit to set * @nr: Bit to set
...@@ -11,7 +15,20 @@ ...@@ -11,7 +15,20 @@
* the returned value is 0. * the returned value is 0.
* It can be used to implement bit locks. * It can be used to implement bit locks.
*/ */
#define test_and_set_bit_lock(nr, addr) test_and_set_bit(nr, addr) static inline int test_and_set_bit_lock(unsigned int nr,
volatile unsigned long *p)
{
long old;
unsigned long mask = BIT_MASK(nr);
p += BIT_WORD(nr);
if (READ_ONCE(*p) & mask)
return 1;
old = atomic_long_fetch_or_acquire(mask, (atomic_long_t *)p);
return !!(old & mask);
}
/** /**
* clear_bit_unlock - Clear a bit in memory, for unlock * clear_bit_unlock - Clear a bit in memory, for unlock
...@@ -20,11 +37,11 @@ ...@@ -20,11 +37,11 @@
* *
* This operation is atomic and provides release barrier semantics. * This operation is atomic and provides release barrier semantics.
*/ */
#define clear_bit_unlock(nr, addr) \ static inline void clear_bit_unlock(unsigned int nr, volatile unsigned long *p)
do { \ {
smp_mb__before_atomic(); \ p += BIT_WORD(nr);
clear_bit(nr, addr); \ atomic_long_fetch_andnot_release(BIT_MASK(nr), (atomic_long_t *)p);
} while (0) }
/** /**
* __clear_bit_unlock - Clear a bit in memory, for unlock * __clear_bit_unlock - Clear a bit in memory, for unlock
...@@ -37,11 +54,38 @@ do { \ ...@@ -37,11 +54,38 @@ do { \
* *
* See for example x86's implementation. * See for example x86's implementation.
*/ */
#define __clear_bit_unlock(nr, addr) \ static inline void __clear_bit_unlock(unsigned int nr,
do { \ volatile unsigned long *p)
smp_mb__before_atomic(); \ {
clear_bit(nr, addr); \ unsigned long old;
} while (0)
#endif /* _ASM_GENERIC_BITOPS_LOCK_H_ */ p += BIT_WORD(nr);
old = READ_ONCE(*p);
old &= ~BIT_MASK(nr);
atomic_long_set_release((atomic_long_t *)p, old);
}
/**
* clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom
* byte is negative, for unlock.
* @nr: the bit to clear
* @addr: the address to start counting from
*
* This is a bit of a one-trick-pony for the filemap code, which clears
* PG_locked and tests PG_waiters,
*/
#ifndef clear_bit_unlock_is_negative_byte
static inline bool clear_bit_unlock_is_negative_byte(unsigned int nr,
volatile unsigned long *p)
{
long old;
unsigned long mask = BIT_MASK(nr);
p += BIT_WORD(nr);
old = atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p);
return !!(old & BIT(7));
}
#define clear_bit_unlock_is_negative_byte clear_bit_unlock_is_negative_byte
#endif
#endif /* _ASM_GENERIC_BITOPS_LOCK_H_ */
...@@ -2,6 +2,8 @@ ...@@ -2,6 +2,8 @@
/* Atomic operations usable in machine independent code */ /* Atomic operations usable in machine independent code */
#ifndef _LINUX_ATOMIC_H #ifndef _LINUX_ATOMIC_H
#define _LINUX_ATOMIC_H #define _LINUX_ATOMIC_H
#include <linux/types.h>
#include <asm/atomic.h> #include <asm/atomic.h>
#include <asm/barrier.h> #include <asm/barrier.h>
...@@ -36,40 +38,46 @@ ...@@ -36,40 +38,46 @@
* barriers on top of the relaxed variant. In the case where the relaxed * barriers on top of the relaxed variant. In the case where the relaxed
* variant is already fully ordered, no additional barriers are needed. * variant is already fully ordered, no additional barriers are needed.
* *
* Besides, if an arch has a special barrier for acquire/release, it could * If an architecture overrides __atomic_acquire_fence() it will probably
* implement its own __atomic_op_* and use the same framework for building * want to define smp_mb__after_spinlock().
* variants
*
* If an architecture overrides __atomic_op_acquire() it will probably want
* to define smp_mb__after_spinlock().
*/ */
#ifndef __atomic_op_acquire #ifndef __atomic_acquire_fence
#define __atomic_acquire_fence smp_mb__after_atomic
#endif
#ifndef __atomic_release_fence
#define __atomic_release_fence smp_mb__before_atomic
#endif
#ifndef __atomic_pre_full_fence
#define __atomic_pre_full_fence smp_mb__before_atomic
#endif
#ifndef __atomic_post_full_fence
#define __atomic_post_full_fence smp_mb__after_atomic
#endif
#define __atomic_op_acquire(op, args...) \ #define __atomic_op_acquire(op, args...) \
({ \ ({ \
typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \
smp_mb__after_atomic(); \ __atomic_acquire_fence(); \
__ret; \ __ret; \
}) })
#endif
#ifndef __atomic_op_release
#define __atomic_op_release(op, args...) \ #define __atomic_op_release(op, args...) \
({ \ ({ \
smp_mb__before_atomic(); \ __atomic_release_fence(); \
op##_relaxed(args); \ op##_relaxed(args); \
}) })
#endif
#ifndef __atomic_op_fence
#define __atomic_op_fence(op, args...) \ #define __atomic_op_fence(op, args...) \
({ \ ({ \
typeof(op##_relaxed(args)) __ret; \ typeof(op##_relaxed(args)) __ret; \
smp_mb__before_atomic(); \ __atomic_pre_full_fence(); \
__ret = op##_relaxed(args); \ __ret = op##_relaxed(args); \
smp_mb__after_atomic(); \ __atomic_post_full_fence(); \
__ret; \ __ret; \
}) })
#endif
/* atomic_add_return_relaxed */ /* atomic_add_return_relaxed */
#ifndef atomic_add_return_relaxed #ifndef atomic_add_return_relaxed
...@@ -95,11 +103,23 @@ ...@@ -95,11 +103,23 @@
#endif #endif
#endif /* atomic_add_return_relaxed */ #endif /* atomic_add_return_relaxed */
#ifndef atomic_inc
#define atomic_inc(v) atomic_add(1, (v))
#endif
/* atomic_inc_return_relaxed */ /* atomic_inc_return_relaxed */
#ifndef atomic_inc_return_relaxed #ifndef atomic_inc_return_relaxed
#ifndef atomic_inc_return
#define atomic_inc_return(v) atomic_add_return(1, (v))
#define atomic_inc_return_relaxed(v) atomic_add_return_relaxed(1, (v))
#define atomic_inc_return_acquire(v) atomic_add_return_acquire(1, (v))
#define atomic_inc_return_release(v) atomic_add_return_release(1, (v))
#else /* atomic_inc_return */
#define atomic_inc_return_relaxed atomic_inc_return #define atomic_inc_return_relaxed atomic_inc_return
#define atomic_inc_return_acquire atomic_inc_return #define atomic_inc_return_acquire atomic_inc_return
#define atomic_inc_return_release atomic_inc_return #define atomic_inc_return_release atomic_inc_return
#endif /* atomic_inc_return */
#else /* atomic_inc_return_relaxed */ #else /* atomic_inc_return_relaxed */
...@@ -143,11 +163,23 @@ ...@@ -143,11 +163,23 @@
#endif #endif
#endif /* atomic_sub_return_relaxed */ #endif /* atomic_sub_return_relaxed */
#ifndef atomic_dec
#define atomic_dec(v) atomic_sub(1, (v))
#endif
/* atomic_dec_return_relaxed */ /* atomic_dec_return_relaxed */
#ifndef atomic_dec_return_relaxed #ifndef atomic_dec_return_relaxed
#ifndef atomic_dec_return
#define atomic_dec_return(v) atomic_sub_return(1, (v))
#define atomic_dec_return_relaxed(v) atomic_sub_return_relaxed(1, (v))
#define atomic_dec_return_acquire(v) atomic_sub_return_acquire(1, (v))
#define atomic_dec_return_release(v) atomic_sub_return_release(1, (v))
#else /* atomic_dec_return */
#define atomic_dec_return_relaxed atomic_dec_return #define atomic_dec_return_relaxed atomic_dec_return
#define atomic_dec_return_acquire atomic_dec_return #define atomic_dec_return_acquire atomic_dec_return
#define atomic_dec_return_release atomic_dec_return #define atomic_dec_return_release atomic_dec_return
#endif /* atomic_dec_return */
#else /* atomic_dec_return_relaxed */ #else /* atomic_dec_return_relaxed */
...@@ -328,12 +360,22 @@ ...@@ -328,12 +360,22 @@
#endif #endif
#endif /* atomic_fetch_and_relaxed */ #endif /* atomic_fetch_and_relaxed */
#ifdef atomic_andnot #ifndef atomic_andnot
/* atomic_fetch_andnot_relaxed */ #define atomic_andnot(i, v) atomic_and(~(int)(i), (v))
#endif
#ifndef atomic_fetch_andnot_relaxed #ifndef atomic_fetch_andnot_relaxed
#ifndef atomic_fetch_andnot
#define atomic_fetch_andnot(i, v) atomic_fetch_and(~(int)(i), (v))
#define atomic_fetch_andnot_relaxed(i, v) atomic_fetch_and_relaxed(~(int)(i), (v))
#define atomic_fetch_andnot_acquire(i, v) atomic_fetch_and_acquire(~(int)(i), (v))
#define atomic_fetch_andnot_release(i, v) atomic_fetch_and_release(~(int)(i), (v))
#else /* atomic_fetch_andnot */
#define atomic_fetch_andnot_relaxed atomic_fetch_andnot #define atomic_fetch_andnot_relaxed atomic_fetch_andnot
#define atomic_fetch_andnot_acquire atomic_fetch_andnot #define atomic_fetch_andnot_acquire atomic_fetch_andnot
#define atomic_fetch_andnot_release atomic_fetch_andnot #define atomic_fetch_andnot_release atomic_fetch_andnot
#endif /* atomic_fetch_andnot */
#else /* atomic_fetch_andnot_relaxed */ #else /* atomic_fetch_andnot_relaxed */
...@@ -352,7 +394,6 @@ ...@@ -352,7 +394,6 @@
__atomic_op_fence(atomic_fetch_andnot, __VA_ARGS__) __atomic_op_fence(atomic_fetch_andnot, __VA_ARGS__)
#endif #endif
#endif /* atomic_fetch_andnot_relaxed */ #endif /* atomic_fetch_andnot_relaxed */
#endif /* atomic_andnot */
/* atomic_fetch_xor_relaxed */ /* atomic_fetch_xor_relaxed */
#ifndef atomic_fetch_xor_relaxed #ifndef atomic_fetch_xor_relaxed
...@@ -519,113 +560,141 @@ ...@@ -519,113 +560,141 @@
#endif #endif
#endif /* xchg_relaxed */ #endif /* xchg_relaxed */
/**
* atomic_fetch_add_unless - add unless the number is already a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, if @v was not already @u.
* Returns the original value of @v.
*/
#ifndef atomic_fetch_add_unless
static inline int atomic_fetch_add_unless(atomic_t *v, int a, int u)
{
int c = atomic_read(v);
do {
if (unlikely(c == u))
break;
} while (!atomic_try_cmpxchg(v, &c, c + a));
return c;
}
#endif
/** /**
* atomic_add_unless - add unless the number is already a given value * atomic_add_unless - add unless the number is already a given value
* @v: pointer of type atomic_t * @v: pointer of type atomic_t
* @a: the amount to add to v... * @a: the amount to add to v...
* @u: ...unless v is equal to u. * @u: ...unless v is equal to u.
* *
* Atomically adds @a to @v, so long as @v was not already @u. * Atomically adds @a to @v, if @v was not already @u.
* Returns non-zero if @v was not @u, and zero otherwise. * Returns true if the addition was done.
*/ */
static inline int atomic_add_unless(atomic_t *v, int a, int u) static inline bool atomic_add_unless(atomic_t *v, int a, int u)
{ {
return __atomic_add_unless(v, a, u) != u; return atomic_fetch_add_unless(v, a, u) != u;
} }
/** /**
* atomic_inc_not_zero - increment unless the number is zero * atomic_inc_not_zero - increment unless the number is zero
* @v: pointer of type atomic_t * @v: pointer of type atomic_t
* *
* Atomically increments @v by 1, so long as @v is non-zero. * Atomically increments @v by 1, if @v is non-zero.
* Returns non-zero if @v was non-zero, and zero otherwise. * Returns true if the increment was done.
*/ */
#ifndef atomic_inc_not_zero #ifndef atomic_inc_not_zero
#define atomic_inc_not_zero(v) atomic_add_unless((v), 1, 0) #define atomic_inc_not_zero(v) atomic_add_unless((v), 1, 0)
#endif #endif
#ifndef atomic_andnot /**
static inline void atomic_andnot(int i, atomic_t *v) * atomic_inc_and_test - increment and test
{ * @v: pointer of type atomic_t
atomic_and(~i, v); *
} * Atomically increments @v by 1
* and returns true if the result is zero, or false for all
static inline int atomic_fetch_andnot(int i, atomic_t *v) * other cases.
{ */
return atomic_fetch_and(~i, v); #ifndef atomic_inc_and_test
} static inline bool atomic_inc_and_test(atomic_t *v)
static inline int atomic_fetch_andnot_relaxed(int i, atomic_t *v)
{ {
return atomic_fetch_and_relaxed(~i, v); return atomic_inc_return(v) == 0;
} }
#endif
static inline int atomic_fetch_andnot_acquire(int i, atomic_t *v) /**
* atomic_dec_and_test - decrement and test
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
#ifndef atomic_dec_and_test
static inline bool atomic_dec_and_test(atomic_t *v)
{ {
return atomic_fetch_and_acquire(~i, v); return atomic_dec_return(v) == 0;
} }
#endif
static inline int atomic_fetch_andnot_release(int i, atomic_t *v) /**
* atomic_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer of type atomic_t
*
* Atomically subtracts @i from @v and returns
* true if the result is zero, or false for all
* other cases.
*/
#ifndef atomic_sub_and_test
static inline bool atomic_sub_and_test(int i, atomic_t *v)
{ {
return atomic_fetch_and_release(~i, v); return atomic_sub_return(i, v) == 0;
} }
#endif #endif
/** /**
* atomic_inc_not_zero_hint - increment if not null * atomic_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer of type atomic_t * @v: pointer of type atomic_t
* @hint: probable value of the atomic before the increment
*
* This version of atomic_inc_not_zero() gives a hint of probable
* value of the atomic. This helps processor to not read the memory
* before doing the atomic read/modify/write cycle, lowering
* number of bus transactions on some arches.
* *
* Returns: 0 if increment was not done, 1 otherwise. * Atomically adds @i to @v and returns true
* if the result is negative, or false when
* result is greater than or equal to zero.
*/ */
#ifndef atomic_inc_not_zero_hint #ifndef atomic_add_negative
static inline int atomic_inc_not_zero_hint(atomic_t *v, int hint) static inline bool atomic_add_negative(int i, atomic_t *v)
{ {
int val, c = hint; return atomic_add_return(i, v) < 0;
/* sanity test, should be removed by compiler if hint is a constant */
if (!hint)
return atomic_inc_not_zero(v);
do {
val = atomic_cmpxchg(v, c, c + 1);
if (val == c)
return 1;
c = val;
} while (c);
return 0;
} }
#endif #endif
#ifndef atomic_inc_unless_negative #ifndef atomic_inc_unless_negative
static inline int atomic_inc_unless_negative(atomic_t *p) static inline bool atomic_inc_unless_negative(atomic_t *v)
{ {
int v, v1; int c = atomic_read(v);
for (v = 0; v >= 0; v = v1) {
v1 = atomic_cmpxchg(p, v, v + 1); do {
if (likely(v1 == v)) if (unlikely(c < 0))
return 1; return false;
} } while (!atomic_try_cmpxchg(v, &c, c + 1));
return 0;
return true;
} }
#endif #endif
#ifndef atomic_dec_unless_positive #ifndef atomic_dec_unless_positive
static inline int atomic_dec_unless_positive(atomic_t *p) static inline bool atomic_dec_unless_positive(atomic_t *v)
{ {
int v, v1; int c = atomic_read(v);
for (v = 0; v <= 0; v = v1) {
v1 = atomic_cmpxchg(p, v, v - 1); do {
if (likely(v1 == v)) if (unlikely(c > 0))
return 1; return false;
} } while (!atomic_try_cmpxchg(v, &c, c - 1));
return 0;
return true;
} }
#endif #endif
...@@ -639,17 +708,14 @@ static inline int atomic_dec_unless_positive(atomic_t *p) ...@@ -639,17 +708,14 @@ static inline int atomic_dec_unless_positive(atomic_t *p)
#ifndef atomic_dec_if_positive #ifndef atomic_dec_if_positive
static inline int atomic_dec_if_positive(atomic_t *v) static inline int atomic_dec_if_positive(atomic_t *v)
{ {
int c, old, dec; int dec, c = atomic_read(v);
c = atomic_read(v);
for (;;) { do {
dec = c - 1; dec = c - 1;
if (unlikely(dec < 0)) if (unlikely(dec < 0))
break; break;
old = atomic_cmpxchg((v), c, dec); } while (!atomic_try_cmpxchg(v, &c, dec));
if (likely(old == c))
break;
c = old;
}
return dec; return dec;
} }
#endif #endif
...@@ -693,11 +759,23 @@ static inline int atomic_dec_if_positive(atomic_t *v) ...@@ -693,11 +759,23 @@ static inline int atomic_dec_if_positive(atomic_t *v)
#endif #endif
#endif /* atomic64_add_return_relaxed */ #endif /* atomic64_add_return_relaxed */
#ifndef atomic64_inc
#define atomic64_inc(v) atomic64_add(1, (v))
#endif
/* atomic64_inc_return_relaxed */ /* atomic64_inc_return_relaxed */
#ifndef atomic64_inc_return_relaxed #ifndef atomic64_inc_return_relaxed
#ifndef atomic64_inc_return
#define atomic64_inc_return(v) atomic64_add_return(1, (v))
#define atomic64_inc_return_relaxed(v) atomic64_add_return_relaxed(1, (v))
#define atomic64_inc_return_acquire(v) atomic64_add_return_acquire(1, (v))
#define atomic64_inc_return_release(v) atomic64_add_return_release(1, (v))
#else /* atomic64_inc_return */
#define atomic64_inc_return_relaxed atomic64_inc_return #define atomic64_inc_return_relaxed atomic64_inc_return
#define atomic64_inc_return_acquire atomic64_inc_return #define atomic64_inc_return_acquire atomic64_inc_return
#define atomic64_inc_return_release atomic64_inc_return #define atomic64_inc_return_release atomic64_inc_return
#endif /* atomic64_inc_return */
#else /* atomic64_inc_return_relaxed */ #else /* atomic64_inc_return_relaxed */
...@@ -742,11 +820,23 @@ static inline int atomic_dec_if_positive(atomic_t *v) ...@@ -742,11 +820,23 @@ static inline int atomic_dec_if_positive(atomic_t *v)
#endif #endif
#endif /* atomic64_sub_return_relaxed */ #endif /* atomic64_sub_return_relaxed */
#ifndef atomic64_dec
#define atomic64_dec(v) atomic64_sub(1, (v))
#endif
/* atomic64_dec_return_relaxed */ /* atomic64_dec_return_relaxed */
#ifndef atomic64_dec_return_relaxed #ifndef atomic64_dec_return_relaxed
#ifndef atomic64_dec_return
#define atomic64_dec_return(v) atomic64_sub_return(1, (v))
#define atomic64_dec_return_relaxed(v) atomic64_sub_return_relaxed(1, (v))
#define atomic64_dec_return_acquire(v) atomic64_sub_return_acquire(1, (v))
#define atomic64_dec_return_release(v) atomic64_sub_return_release(1, (v))
#else /* atomic64_dec_return */
#define atomic64_dec_return_relaxed atomic64_dec_return #define atomic64_dec_return_relaxed atomic64_dec_return
#define atomic64_dec_return_acquire atomic64_dec_return #define atomic64_dec_return_acquire atomic64_dec_return
#define atomic64_dec_return_release atomic64_dec_return #define atomic64_dec_return_release atomic64_dec_return
#endif /* atomic64_dec_return */
#else /* atomic64_dec_return_relaxed */ #else /* atomic64_dec_return_relaxed */
...@@ -927,12 +1017,22 @@ static inline int atomic_dec_if_positive(atomic_t *v) ...@@ -927,12 +1017,22 @@ static inline int atomic_dec_if_positive(atomic_t *v)
#endif #endif
#endif /* atomic64_fetch_and_relaxed */ #endif /* atomic64_fetch_and_relaxed */
#ifdef atomic64_andnot #ifndef atomic64_andnot
/* atomic64_fetch_andnot_relaxed */ #define atomic64_andnot(i, v) atomic64_and(~(long long)(i), (v))
#endif
#ifndef atomic64_fetch_andnot_relaxed #ifndef atomic64_fetch_andnot_relaxed
#ifndef atomic64_fetch_andnot
#define atomic64_fetch_andnot(i, v) atomic64_fetch_and(~(long long)(i), (v))
#define atomic64_fetch_andnot_relaxed(i, v) atomic64_fetch_and_relaxed(~(long long)(i), (v))
#define atomic64_fetch_andnot_acquire(i, v) atomic64_fetch_and_acquire(~(long long)(i), (v))
#define atomic64_fetch_andnot_release(i, v) atomic64_fetch_and_release(~(long long)(i), (v))
#else /* atomic64_fetch_andnot */
#define atomic64_fetch_andnot_relaxed atomic64_fetch_andnot #define atomic64_fetch_andnot_relaxed atomic64_fetch_andnot
#define atomic64_fetch_andnot_acquire atomic64_fetch_andnot #define atomic64_fetch_andnot_acquire atomic64_fetch_andnot
#define atomic64_fetch_andnot_release atomic64_fetch_andnot #define atomic64_fetch_andnot_release atomic64_fetch_andnot
#endif /* atomic64_fetch_andnot */
#else /* atomic64_fetch_andnot_relaxed */ #else /* atomic64_fetch_andnot_relaxed */
...@@ -951,7 +1051,6 @@ static inline int atomic_dec_if_positive(atomic_t *v) ...@@ -951,7 +1051,6 @@ static inline int atomic_dec_if_positive(atomic_t *v)
__atomic_op_fence(atomic64_fetch_andnot, __VA_ARGS__) __atomic_op_fence(atomic64_fetch_andnot, __VA_ARGS__)
#endif #endif
#endif /* atomic64_fetch_andnot_relaxed */ #endif /* atomic64_fetch_andnot_relaxed */
#endif /* atomic64_andnot */
/* atomic64_fetch_xor_relaxed */ /* atomic64_fetch_xor_relaxed */
#ifndef atomic64_fetch_xor_relaxed #ifndef atomic64_fetch_xor_relaxed
...@@ -1049,30 +1148,164 @@ static inline int atomic_dec_if_positive(atomic_t *v) ...@@ -1049,30 +1148,164 @@ static inline int atomic_dec_if_positive(atomic_t *v)
#define atomic64_try_cmpxchg_release atomic64_try_cmpxchg #define atomic64_try_cmpxchg_release atomic64_try_cmpxchg
#endif /* atomic64_try_cmpxchg */ #endif /* atomic64_try_cmpxchg */
#ifndef atomic64_andnot /**
static inline void atomic64_andnot(long long i, atomic64_t *v) * atomic64_fetch_add_unless - add unless the number is already a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, if @v was not already @u.
* Returns the original value of @v.
*/
#ifndef atomic64_fetch_add_unless
static inline long long atomic64_fetch_add_unless(atomic64_t *v, long long a,
long long u)
{ {
atomic64_and(~i, v); long long c = atomic64_read(v);
do {
if (unlikely(c == u))
break;
} while (!atomic64_try_cmpxchg(v, &c, c + a));
return c;
} }
#endif
static inline long long atomic64_fetch_andnot(long long i, atomic64_t *v) /**
* atomic64_add_unless - add unless the number is already a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
*
* Atomically adds @a to @v, if @v was not already @u.
* Returns true if the addition was done.
*/
static inline bool atomic64_add_unless(atomic64_t *v, long long a, long long u)
{ {
return atomic64_fetch_and(~i, v); return atomic64_fetch_add_unless(v, a, u) != u;
} }
static inline long long atomic64_fetch_andnot_relaxed(long long i, atomic64_t *v) /**
* atomic64_inc_not_zero - increment unless the number is zero
* @v: pointer of type atomic64_t
*
* Atomically increments @v by 1, if @v is non-zero.
* Returns true if the increment was done.
*/
#ifndef atomic64_inc_not_zero
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
#endif
/**
* atomic64_inc_and_test - increment and test
* @v: pointer of type atomic64_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
#ifndef atomic64_inc_and_test
static inline bool atomic64_inc_and_test(atomic64_t *v)
{ {
return atomic64_fetch_and_relaxed(~i, v); return atomic64_inc_return(v) == 0;
} }
#endif
static inline long long atomic64_fetch_andnot_acquire(long long i, atomic64_t *v) /**
* atomic64_dec_and_test - decrement and test
* @v: pointer of type atomic64_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
#ifndef atomic64_dec_and_test
static inline bool atomic64_dec_and_test(atomic64_t *v)
{ {
return atomic64_fetch_and_acquire(~i, v); return atomic64_dec_return(v) == 0;
} }
#endif
static inline long long atomic64_fetch_andnot_release(long long i, atomic64_t *v) /**
* atomic64_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer of type atomic64_t
*
* Atomically subtracts @i from @v and returns
* true if the result is zero, or false for all
* other cases.
*/
#ifndef atomic64_sub_and_test
static inline bool atomic64_sub_and_test(long long i, atomic64_t *v)
{
return atomic64_sub_return(i, v) == 0;
}
#endif
/**
* atomic64_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer of type atomic64_t
*
* Atomically adds @i to @v and returns true
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
#ifndef atomic64_add_negative
static inline bool atomic64_add_negative(long long i, atomic64_t *v)
{ {
return atomic64_fetch_and_release(~i, v); return atomic64_add_return(i, v) < 0;
}
#endif
#ifndef atomic64_inc_unless_negative
static inline bool atomic64_inc_unless_negative(atomic64_t *v)
{
long long c = atomic64_read(v);
do {
if (unlikely(c < 0))
return false;
} while (!atomic64_try_cmpxchg(v, &c, c + 1));
return true;
}
#endif
#ifndef atomic64_dec_unless_positive
static inline bool atomic64_dec_unless_positive(atomic64_t *v)
{
long long c = atomic64_read(v);
do {
if (unlikely(c > 0))
return false;
} while (!atomic64_try_cmpxchg(v, &c, c - 1));
return true;
}
#endif
/*
* atomic64_dec_if_positive - decrement by 1 if old value positive
* @v: pointer of type atomic64_t
*
* The function returns the old value of *v minus 1, even if
* the atomic64 variable, v, was not decremented.
*/
#ifndef atomic64_dec_if_positive
static inline long long atomic64_dec_if_positive(atomic64_t *v)
{
long long dec, c = atomic64_read(v);
do {
dec = c - 1;
if (unlikely(dec < 0))
break;
} while (!atomic64_try_cmpxchg(v, &c, dec));
return dec;
} }
#endif #endif
......
...@@ -2,29 +2,9 @@ ...@@ -2,29 +2,9 @@
#ifndef _LINUX_BITOPS_H #ifndef _LINUX_BITOPS_H
#define _LINUX_BITOPS_H #define _LINUX_BITOPS_H
#include <asm/types.h> #include <asm/types.h>
#include <linux/bits.h>
#ifdef __KERNEL__
#define BIT(nr) (1UL << (nr))
#define BIT_ULL(nr) (1ULL << (nr))
#define BIT_MASK(nr) (1UL << ((nr) % BITS_PER_LONG))
#define BIT_WORD(nr) ((nr) / BITS_PER_LONG)
#define BIT_ULL_MASK(nr) (1ULL << ((nr) % BITS_PER_LONG_LONG))
#define BIT_ULL_WORD(nr) ((nr) / BITS_PER_LONG_LONG)
#define BITS_PER_BYTE 8
#define BITS_TO_LONGS(nr) DIV_ROUND_UP(nr, BITS_PER_BYTE * sizeof(long)) #define BITS_TO_LONGS(nr) DIV_ROUND_UP(nr, BITS_PER_BYTE * sizeof(long))
#endif
/*
* Create a contiguous bitmask starting at bit position @l and ending at
* position @h. For example
* GENMASK_ULL(39, 21) gives us the 64bit vector 0x000000ffffe00000.
*/
#define GENMASK(h, l) \
(((~0UL) - (1UL << (l)) + 1) & (~0UL >> (BITS_PER_LONG - 1 - (h))))
#define GENMASK_ULL(h, l) \
(((~0ULL) - (1ULL << (l)) + 1) & \
(~0ULL >> (BITS_PER_LONG_LONG - 1 - (h))))
extern unsigned int __sw_hweight8(unsigned int w); extern unsigned int __sw_hweight8(unsigned int w);
extern unsigned int __sw_hweight16(unsigned int w); extern unsigned int __sw_hweight16(unsigned int w);
......
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef __LINUX_BITS_H
#define __LINUX_BITS_H
#include <asm/bitsperlong.h>
#define BIT(nr) (1UL << (nr))
#define BIT_ULL(nr) (1ULL << (nr))
#define BIT_MASK(nr) (1UL << ((nr) % BITS_PER_LONG))
#define BIT_WORD(nr) ((nr) / BITS_PER_LONG)
#define BIT_ULL_MASK(nr) (1ULL << ((nr) % BITS_PER_LONG_LONG))
#define BIT_ULL_WORD(nr) ((nr) / BITS_PER_LONG_LONG)
#define BITS_PER_BYTE 8
/*
* Create a contiguous bitmask starting at bit position @l and ending at
* position @h. For example
* GENMASK_ULL(39, 21) gives us the 64bit vector 0x000000ffffe00000.
*/
#define GENMASK(h, l) \
(((~0UL) - (1UL << (l)) + 1) & (~0UL >> (BITS_PER_LONG - 1 - (h))))
#define GENMASK_ULL(h, l) \
(((~0ULL) - (1ULL << (l)) + 1) & \
(~0ULL >> (BITS_PER_LONG_LONG - 1 - (h))))
#endif /* __LINUX_BITS_H */
...@@ -3,9 +3,10 @@ ...@@ -3,9 +3,10 @@
#define _LINUX_REFCOUNT_H #define _LINUX_REFCOUNT_H
#include <linux/atomic.h> #include <linux/atomic.h>
#include <linux/mutex.h> #include <linux/compiler.h>
#include <linux/spinlock.h> #include <linux/spinlock_types.h>
#include <linux/kernel.h>
struct mutex;
/** /**
* struct refcount_t - variant of atomic_t specialized for reference counts * struct refcount_t - variant of atomic_t specialized for reference counts
...@@ -42,17 +43,30 @@ static inline unsigned int refcount_read(const refcount_t *r) ...@@ -42,17 +43,30 @@ static inline unsigned int refcount_read(const refcount_t *r)
return atomic_read(&r->refs); return atomic_read(&r->refs);
} }
extern __must_check bool refcount_add_not_zero_checked(unsigned int i, refcount_t *r);
extern void refcount_add_checked(unsigned int i, refcount_t *r);
extern __must_check bool refcount_inc_not_zero_checked(refcount_t *r);
extern void refcount_inc_checked(refcount_t *r);
extern __must_check bool refcount_sub_and_test_checked(unsigned int i, refcount_t *r);
extern __must_check bool refcount_dec_and_test_checked(refcount_t *r);
extern void refcount_dec_checked(refcount_t *r);
#ifdef CONFIG_REFCOUNT_FULL #ifdef CONFIG_REFCOUNT_FULL
extern __must_check bool refcount_add_not_zero(unsigned int i, refcount_t *r);
extern void refcount_add(unsigned int i, refcount_t *r);
extern __must_check bool refcount_inc_not_zero(refcount_t *r); #define refcount_add_not_zero refcount_add_not_zero_checked
extern void refcount_inc(refcount_t *r); #define refcount_add refcount_add_checked
#define refcount_inc_not_zero refcount_inc_not_zero_checked
#define refcount_inc refcount_inc_checked
#define refcount_sub_and_test refcount_sub_and_test_checked
extern __must_check bool refcount_sub_and_test(unsigned int i, refcount_t *r); #define refcount_dec_and_test refcount_dec_and_test_checked
#define refcount_dec refcount_dec_checked
extern __must_check bool refcount_dec_and_test(refcount_t *r);
extern void refcount_dec(refcount_t *r);
#else #else
# ifdef CONFIG_ARCH_HAS_REFCOUNT # ifdef CONFIG_ARCH_HAS_REFCOUNT
# include <asm/refcount.h> # include <asm/refcount.h>
......
...@@ -167,8 +167,8 @@ struct task_group; ...@@ -167,8 +167,8 @@ struct task_group;
* need_sleep = false; * need_sleep = false;
* wake_up_state(p, TASK_UNINTERRUPTIBLE); * wake_up_state(p, TASK_UNINTERRUPTIBLE);
* *
* Where wake_up_state() (and all other wakeup primitives) imply enough * where wake_up_state() executes a full memory barrier before accessing the
* barriers to order the store of the variable against wakeup. * task state.
* *
* Wakeup will do: if (@state & p->state) p->state = TASK_RUNNING, that is, * Wakeup will do: if (@state & p->state) p->state = TASK_RUNNING, that is,
* once it observes the TASK_UNINTERRUPTIBLE store the waking CPU can issue a * once it observes the TASK_UNINTERRUPTIBLE store the waking CPU can issue a
......
...@@ -114,29 +114,48 @@ do { \ ...@@ -114,29 +114,48 @@ do { \
#endif /*arch_spin_is_contended*/ #endif /*arch_spin_is_contended*/
/* /*
* This barrier must provide two things: * smp_mb__after_spinlock() provides the equivalent of a full memory barrier
* between program-order earlier lock acquisitions and program-order later
* memory accesses.
* *
* - it must guarantee a STORE before the spin_lock() is ordered against a * This guarantees that the following two properties hold:
* LOAD after it, see the comments at its two usage sites.
* *
* - it must ensure the critical section is RCsc. * 1) Given the snippet:
* *
* The latter is important for cases where we observe values written by other * { X = 0; Y = 0; }
* CPUs in spin-loops, without barriers, while being subject to scheduling. *
* CPU0 CPU1
*
* WRITE_ONCE(X, 1); WRITE_ONCE(Y, 1);
* spin_lock(S); smp_mb();
* smp_mb__after_spinlock(); r1 = READ_ONCE(X);
* r0 = READ_ONCE(Y);
* spin_unlock(S);
*
* it is forbidden that CPU0 does not observe CPU1's store to Y (r0 = 0)
* and CPU1 does not observe CPU0's store to X (r1 = 0); see the comments
* preceding the call to smp_mb__after_spinlock() in __schedule() and in
* try_to_wake_up().
*
* 2) Given the snippet:
*
* { X = 0; Y = 0; }
* *
* CPU0 CPU1 CPU2 * CPU0 CPU1 CPU2
* *
* for (;;) { * spin_lock(S); spin_lock(S); r1 = READ_ONCE(Y);
* if (READ_ONCE(X)) * WRITE_ONCE(X, 1); smp_mb__after_spinlock(); smp_rmb();
* break; * spin_unlock(S); r0 = READ_ONCE(X); r2 = READ_ONCE(X);
* } * WRITE_ONCE(Y, 1);
* X=1 * spin_unlock(S);
* <sched-out> *
* <sched-in> * it is forbidden that CPU0's critical section executes before CPU1's
* r = X; * critical section (r0 = 1), CPU2 observes CPU1's store to Y (r1 = 1)
* * and CPU2 does not observe CPU0's store to X (r2 = 0); see the comments
* without transitivity it could be that CPU1 observes X!=0 breaks the loop, * preceding the calls to smp_rmb() in try_to_wake_up() for similar
* we get migrated and CPU2 sees X==0. * snippets but "projected" onto two CPUs.
*
* Property (2) upgrades the lock to an RCsc lock.
* *
* Since most load-store architectures implement ACQUIRE with an smp_mb() after * Since most load-store architectures implement ACQUIRE with an smp_mb() after
* the LL/SC loop, they need no further barriers. Similarly all our TSO * the LL/SC loop, they need no further barriers. Similarly all our TSO
......
...@@ -575,7 +575,7 @@ static struct bpf_map *bpf_map_inc_not_zero(struct bpf_map *map, ...@@ -575,7 +575,7 @@ static struct bpf_map *bpf_map_inc_not_zero(struct bpf_map *map,
{ {
int refold; int refold;
refold = __atomic_add_unless(&map->refcnt, 1, 0); refold = atomic_fetch_add_unless(&map->refcnt, 1, 0);
if (refold >= BPF_MAX_REFCNT) { if (refold >= BPF_MAX_REFCNT) {
__bpf_map_put(map, false); __bpf_map_put(map, false);
...@@ -1144,7 +1144,7 @@ struct bpf_prog *bpf_prog_inc_not_zero(struct bpf_prog *prog) ...@@ -1144,7 +1144,7 @@ struct bpf_prog *bpf_prog_inc_not_zero(struct bpf_prog *prog)
{ {
int refold; int refold;
refold = __atomic_add_unless(&prog->aux->refcnt, 1, 0); refold = atomic_fetch_add_unless(&prog->aux->refcnt, 1, 0);
if (refold >= BPF_MAX_REFCNT) { if (refold >= BPF_MAX_REFCNT) {
__bpf_prog_put(prog, false); __bpf_prog_put(prog, false);
......
...@@ -22,8 +22,8 @@ ...@@ -22,8 +22,8 @@
* *
* See also complete_all(), wait_for_completion() and related routines. * See also complete_all(), wait_for_completion() and related routines.
* *
* It may be assumed that this function implies a write memory barrier before * If this function wakes up a task, it executes a full memory barrier before
* changing the task state if and only if any tasks are woken up. * accessing the task state.
*/ */
void complete(struct completion *x) void complete(struct completion *x)
{ {
...@@ -44,8 +44,8 @@ EXPORT_SYMBOL(complete); ...@@ -44,8 +44,8 @@ EXPORT_SYMBOL(complete);
* *
* This will wake up all threads waiting on this particular completion event. * This will wake up all threads waiting on this particular completion event.
* *
* It may be assumed that this function implies a write memory barrier before * If this function wakes up a task, it executes a full memory barrier before
* changing the task state if and only if any tasks are woken up. * accessing the task state.
* *
* Since complete_all() sets the completion of @x permanently to done * Since complete_all() sets the completion of @x permanently to done
* to allow multiple waiters to finish, a call to reinit_completion() * to allow multiple waiters to finish, a call to reinit_completion()
......
...@@ -406,8 +406,8 @@ void wake_q_add(struct wake_q_head *head, struct task_struct *task) ...@@ -406,8 +406,8 @@ void wake_q_add(struct wake_q_head *head, struct task_struct *task)
* its already queued (either by us or someone else) and will get the * its already queued (either by us or someone else) and will get the
* wakeup due to that. * wakeup due to that.
* *
* This cmpxchg() implies a full barrier, which pairs with the write * This cmpxchg() executes a full barrier, which pairs with the full
* barrier implied by the wakeup in wake_up_q(). * barrier executed by the wakeup in wake_up_q().
*/ */
if (cmpxchg(&node->next, NULL, WAKE_Q_TAIL)) if (cmpxchg(&node->next, NULL, WAKE_Q_TAIL))
return; return;
...@@ -435,8 +435,8 @@ void wake_up_q(struct wake_q_head *head) ...@@ -435,8 +435,8 @@ void wake_up_q(struct wake_q_head *head)
task->wake_q.next = NULL; task->wake_q.next = NULL;
/* /*
* wake_up_process() implies a wmb() to pair with the queueing * wake_up_process() executes a full barrier, which pairs with
* in wake_q_add() so as not to miss wakeups. * the queueing in wake_q_add() so as not to miss wakeups.
*/ */
wake_up_process(task); wake_up_process(task);
put_task_struct(task); put_task_struct(task);
...@@ -1859,8 +1859,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) ...@@ -1859,8 +1859,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
* rq(c1)->lock (if not at the same time, then in that order). * rq(c1)->lock (if not at the same time, then in that order).
* C) LOCK of the rq(c1)->lock scheduling in task * C) LOCK of the rq(c1)->lock scheduling in task
* *
* Transitivity guarantees that B happens after A and C after B. * Release/acquire chaining guarantees that B happens after A and C after B.
* Note: we only require RCpc transitivity.
* Note: the CPU doing B need not be c0 or c1 * Note: the CPU doing B need not be c0 or c1
* *
* Example: * Example:
...@@ -1922,16 +1921,9 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) ...@@ -1922,16 +1921,9 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
* UNLOCK rq(0)->lock * UNLOCK rq(0)->lock
* *
* *
* However; for wakeups there is a second guarantee we must provide, namely we * However, for wakeups there is a second guarantee we must provide, namely we
* must observe the state that lead to our wakeup. That is, not only must our * must ensure that CONDITION=1 done by the caller can not be reordered with
* task observe its own prior state, it must also observe the stores prior to * accesses to the task state; see try_to_wake_up() and set_current_state().
* its wakeup.
*
* This means that any means of doing remote wakeups must order the CPU doing
* the wakeup against the CPU the task is going to end up running on. This,
* however, is already required for the regular Program-Order guarantee above,
* since the waking CPU is the one issueing the ACQUIRE (smp_cond_load_acquire).
*
*/ */
/** /**
...@@ -1947,6 +1939,9 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) ...@@ -1947,6 +1939,9 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
* Atomic against schedule() which would dequeue a task, also see * Atomic against schedule() which would dequeue a task, also see
* set_current_state(). * set_current_state().
* *
* This function executes a full memory barrier before accessing the task
* state; see set_current_state().
*
* Return: %true if @p->state changes (an actual wakeup was done), * Return: %true if @p->state changes (an actual wakeup was done),
* %false otherwise. * %false otherwise.
*/ */
...@@ -1979,20 +1974,19 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) ...@@ -1979,20 +1974,19 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
* in smp_cond_load_acquire() below. * in smp_cond_load_acquire() below.
* *
* sched_ttwu_pending() try_to_wake_up() * sched_ttwu_pending() try_to_wake_up()
* [S] p->on_rq = 1; [L] P->state * STORE p->on_rq = 1 LOAD p->state
* UNLOCK rq->lock -----. * UNLOCK rq->lock
* \ *
* +--- RMB * __schedule() (switch to task 'p')
* schedule() / * LOCK rq->lock smp_rmb();
* LOCK rq->lock -----' * smp_mb__after_spinlock();
* UNLOCK rq->lock * UNLOCK rq->lock
* *
* [task p] * [task p]
* [S] p->state = UNINTERRUPTIBLE [L] p->on_rq * STORE p->state = UNINTERRUPTIBLE LOAD p->on_rq
* *
* Pairs with the UNLOCK+LOCK on rq->lock from the * Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in
* last wakeup of our task and the schedule that got our task * __schedule(). See the comment for smp_mb__after_spinlock().
* current.
*/ */
smp_rmb(); smp_rmb();
if (p->on_rq && ttwu_remote(p, wake_flags)) if (p->on_rq && ttwu_remote(p, wake_flags))
...@@ -2006,15 +2000,17 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) ...@@ -2006,15 +2000,17 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
* One must be running (->on_cpu == 1) in order to remove oneself * One must be running (->on_cpu == 1) in order to remove oneself
* from the runqueue. * from the runqueue.
* *
* [S] ->on_cpu = 1; [L] ->on_rq * __schedule() (switch to task 'p') try_to_wake_up()
* STORE p->on_cpu = 1 LOAD p->on_rq
* UNLOCK rq->lock * UNLOCK rq->lock
* RMB
* LOCK rq->lock
* [S] ->on_rq = 0; [L] ->on_cpu
* *
* Pairs with the full barrier implied in the UNLOCK+LOCK on rq->lock * __schedule() (put 'p' to sleep)
* from the consecutive calls to schedule(); the first switching to our * LOCK rq->lock smp_rmb();
* task, the second putting it to sleep. * smp_mb__after_spinlock();
* STORE p->on_rq = 0 LOAD p->on_cpu
*
* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in
* __schedule(). See the comment for smp_mb__after_spinlock().
*/ */
smp_rmb(); smp_rmb();
...@@ -2120,8 +2116,7 @@ static void try_to_wake_up_local(struct task_struct *p, struct rq_flags *rf) ...@@ -2120,8 +2116,7 @@ static void try_to_wake_up_local(struct task_struct *p, struct rq_flags *rf)
* *
* Return: 1 if the process was woken up, 0 if it was already running. * Return: 1 if the process was woken up, 0 if it was already running.
* *
* It may be assumed that this function implies a write memory barrier before * This function executes a full memory barrier before accessing the task state.
* changing the task state if and only if any tasks are woken up.
*/ */
int wake_up_process(struct task_struct *p) int wake_up_process(struct task_struct *p)
{ {
......
...@@ -134,8 +134,8 @@ static void __wake_up_common_lock(struct wait_queue_head *wq_head, unsigned int ...@@ -134,8 +134,8 @@ static void __wake_up_common_lock(struct wait_queue_head *wq_head, unsigned int
* @nr_exclusive: how many wake-one or wake-many threads to wake up * @nr_exclusive: how many wake-one or wake-many threads to wake up
* @key: is directly passed to the wakeup function * @key: is directly passed to the wakeup function
* *
* It may be assumed that this function implies a write memory barrier before * If this function wakes up a task, it executes a full memory barrier before
* changing the task state if and only if any tasks are woken up. * accessing the task state.
*/ */
void __wake_up(struct wait_queue_head *wq_head, unsigned int mode, void __wake_up(struct wait_queue_head *wq_head, unsigned int mode,
int nr_exclusive, void *key) int nr_exclusive, void *key)
...@@ -180,8 +180,8 @@ EXPORT_SYMBOL_GPL(__wake_up_locked_key_bookmark); ...@@ -180,8 +180,8 @@ EXPORT_SYMBOL_GPL(__wake_up_locked_key_bookmark);
* *
* On UP it can prevent extra preemption. * On UP it can prevent extra preemption.
* *
* It may be assumed that this function implies a write memory barrier before * If this function wakes up a task, it executes a full memory barrier before
* changing the task state if and only if any tasks are woken up. * accessing the task state.
*/ */
void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode,
int nr_exclusive, void *key) int nr_exclusive, void *key)
...@@ -392,35 +392,36 @@ static inline bool is_kthread_should_stop(void) ...@@ -392,35 +392,36 @@ static inline bool is_kthread_should_stop(void)
* if (condition) * if (condition)
* break; * break;
* *
* p->state = mode; condition = true; * // in wait_woken() // in woken_wake_function()
* smp_mb(); // A smp_wmb(); // C
* if (!wq_entry->flags & WQ_FLAG_WOKEN) wq_entry->flags |= WQ_FLAG_WOKEN;
* schedule() try_to_wake_up();
* p->state = TASK_RUNNING; ~~~~~~~~~~~~~~~~~~
* wq_entry->flags &= ~WQ_FLAG_WOKEN; condition = true;
* smp_mb() // B smp_wmb(); // C
* wq_entry->flags |= WQ_FLAG_WOKEN;
* }
* remove_wait_queue(&wq_head, &wait);
* *
* p->state = mode; wq_entry->flags |= WQ_FLAG_WOKEN;
* smp_mb(); // A try_to_wake_up():
* if (!(wq_entry->flags & WQ_FLAG_WOKEN)) <full barrier>
* schedule() if (p->state & mode)
* p->state = TASK_RUNNING; p->state = TASK_RUNNING;
* wq_entry->flags &= ~WQ_FLAG_WOKEN; ~~~~~~~~~~~~~~~~~~
* smp_mb(); // B condition = true;
* } smp_mb(); // C
* remove_wait_queue(&wq_head, &wait); wq_entry->flags |= WQ_FLAG_WOKEN;
*/ */
long wait_woken(struct wait_queue_entry *wq_entry, unsigned mode, long timeout) long wait_woken(struct wait_queue_entry *wq_entry, unsigned mode, long timeout)
{ {
set_current_state(mode); /* A */
/* /*
* The above implies an smp_mb(), which matches with the smp_wmb() from * The below executes an smp_mb(), which matches with the full barrier
* woken_wake_function() such that if we observe WQ_FLAG_WOKEN we must * executed by the try_to_wake_up() in woken_wake_function() such that
* also observe all state before the wakeup. * either we see the store to wq_entry->flags in woken_wake_function()
* or woken_wake_function() sees our store to current->state.
*/ */
set_current_state(mode); /* A */
if (!(wq_entry->flags & WQ_FLAG_WOKEN) && !is_kthread_should_stop()) if (!(wq_entry->flags & WQ_FLAG_WOKEN) && !is_kthread_should_stop())
timeout = schedule_timeout(timeout); timeout = schedule_timeout(timeout);
__set_current_state(TASK_RUNNING); __set_current_state(TASK_RUNNING);
/* /*
* The below implies an smp_mb(), it too pairs with the smp_wmb() from * The below executes an smp_mb(), which matches with the smp_mb() (C)
* woken_wake_function() such that we must either observe the wait * in woken_wake_function() such that either we see the wait condition
* condition being true _OR_ WQ_FLAG_WOKEN such that we will not miss * being true or the store to wq_entry->flags in woken_wake_function()
* an event. * follows ours in the coherence order.
*/ */
smp_store_mb(wq_entry->flags, wq_entry->flags & ~WQ_FLAG_WOKEN); /* B */ smp_store_mb(wq_entry->flags, wq_entry->flags & ~WQ_FLAG_WOKEN); /* B */
...@@ -430,14 +431,8 @@ EXPORT_SYMBOL(wait_woken); ...@@ -430,14 +431,8 @@ EXPORT_SYMBOL(wait_woken);
int woken_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key) int woken_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key)
{ {
/* /* Pairs with the smp_store_mb() in wait_woken(). */
* Although this function is called under waitqueue lock, LOCK smp_mb(); /* C */
* doesn't imply write barrier and the users expects write
* barrier semantics on wakeup functions. The following
* smp_wmb() is equivalent to smp_wmb() in try_to_wake_up()
* and is paired with smp_store_mb() in wait_woken().
*/
smp_wmb(); /* C */
wq_entry->flags |= WQ_FLAG_WOKEN; wq_entry->flags |= WQ_FLAG_WOKEN;
return default_wake_function(wq_entry, mode, sync, key); return default_wake_function(wq_entry, mode, sync, key);
......
...@@ -178,18 +178,18 @@ long long atomic64_xchg(atomic64_t *v, long long new) ...@@ -178,18 +178,18 @@ long long atomic64_xchg(atomic64_t *v, long long new)
} }
EXPORT_SYMBOL(atomic64_xchg); EXPORT_SYMBOL(atomic64_xchg);
int atomic64_add_unless(atomic64_t *v, long long a, long long u) long long atomic64_fetch_add_unless(atomic64_t *v, long long a, long long u)
{ {
unsigned long flags; unsigned long flags;
raw_spinlock_t *lock = lock_addr(v); raw_spinlock_t *lock = lock_addr(v);
int ret = 0; long long val;
raw_spin_lock_irqsave(lock, flags); raw_spin_lock_irqsave(lock, flags);
if (v->counter != u) { val = v->counter;
if (val != u)
v->counter += a; v->counter += a;
ret = 1;
}
raw_spin_unlock_irqrestore(lock, flags); raw_spin_unlock_irqrestore(lock, flags);
return ret;
return val;
} }
EXPORT_SYMBOL(atomic64_add_unless); EXPORT_SYMBOL(atomic64_fetch_add_unless);
...@@ -35,13 +35,13 @@ ...@@ -35,13 +35,13 @@
* *
*/ */
#include <linux/mutex.h>
#include <linux/refcount.h> #include <linux/refcount.h>
#include <linux/spinlock.h>
#include <linux/bug.h> #include <linux/bug.h>
#ifdef CONFIG_REFCOUNT_FULL
/** /**
* refcount_add_not_zero - add a value to a refcount unless it is 0 * refcount_add_not_zero_checked - add a value to a refcount unless it is 0
* @i: the value to add to the refcount * @i: the value to add to the refcount
* @r: the refcount * @r: the refcount
* *
...@@ -58,7 +58,7 @@ ...@@ -58,7 +58,7 @@
* *
* Return: false if the passed refcount is 0, true otherwise * Return: false if the passed refcount is 0, true otherwise
*/ */
bool refcount_add_not_zero(unsigned int i, refcount_t *r) bool refcount_add_not_zero_checked(unsigned int i, refcount_t *r)
{ {
unsigned int new, val = atomic_read(&r->refs); unsigned int new, val = atomic_read(&r->refs);
...@@ -79,10 +79,10 @@ bool refcount_add_not_zero(unsigned int i, refcount_t *r) ...@@ -79,10 +79,10 @@ bool refcount_add_not_zero(unsigned int i, refcount_t *r)
return true; return true;
} }
EXPORT_SYMBOL(refcount_add_not_zero); EXPORT_SYMBOL(refcount_add_not_zero_checked);
/** /**
* refcount_add - add a value to a refcount * refcount_add_checked - add a value to a refcount
* @i: the value to add to the refcount * @i: the value to add to the refcount
* @r: the refcount * @r: the refcount
* *
...@@ -97,14 +97,14 @@ EXPORT_SYMBOL(refcount_add_not_zero); ...@@ -97,14 +97,14 @@ EXPORT_SYMBOL(refcount_add_not_zero);
* cases, refcount_inc(), or one of its variants, should instead be used to * cases, refcount_inc(), or one of its variants, should instead be used to
* increment a reference count. * increment a reference count.
*/ */
void refcount_add(unsigned int i, refcount_t *r) void refcount_add_checked(unsigned int i, refcount_t *r)
{ {
WARN_ONCE(!refcount_add_not_zero(i, r), "refcount_t: addition on 0; use-after-free.\n"); WARN_ONCE(!refcount_add_not_zero_checked(i, r), "refcount_t: addition on 0; use-after-free.\n");
} }
EXPORT_SYMBOL(refcount_add); EXPORT_SYMBOL(refcount_add_checked);
/** /**
* refcount_inc_not_zero - increment a refcount unless it is 0 * refcount_inc_not_zero_checked - increment a refcount unless it is 0
* @r: the refcount to increment * @r: the refcount to increment
* *
* Similar to atomic_inc_not_zero(), but will saturate at UINT_MAX and WARN. * Similar to atomic_inc_not_zero(), but will saturate at UINT_MAX and WARN.
...@@ -115,7 +115,7 @@ EXPORT_SYMBOL(refcount_add); ...@@ -115,7 +115,7 @@ EXPORT_SYMBOL(refcount_add);
* *
* Return: true if the increment was successful, false otherwise * Return: true if the increment was successful, false otherwise
*/ */
bool refcount_inc_not_zero(refcount_t *r) bool refcount_inc_not_zero_checked(refcount_t *r)
{ {
unsigned int new, val = atomic_read(&r->refs); unsigned int new, val = atomic_read(&r->refs);
...@@ -134,10 +134,10 @@ bool refcount_inc_not_zero(refcount_t *r) ...@@ -134,10 +134,10 @@ bool refcount_inc_not_zero(refcount_t *r)
return true; return true;
} }
EXPORT_SYMBOL(refcount_inc_not_zero); EXPORT_SYMBOL(refcount_inc_not_zero_checked);
/** /**
* refcount_inc - increment a refcount * refcount_inc_checked - increment a refcount
* @r: the refcount to increment * @r: the refcount to increment
* *
* Similar to atomic_inc(), but will saturate at UINT_MAX and WARN. * Similar to atomic_inc(), but will saturate at UINT_MAX and WARN.
...@@ -148,14 +148,14 @@ EXPORT_SYMBOL(refcount_inc_not_zero); ...@@ -148,14 +148,14 @@ EXPORT_SYMBOL(refcount_inc_not_zero);
* Will WARN if the refcount is 0, as this represents a possible use-after-free * Will WARN if the refcount is 0, as this represents a possible use-after-free
* condition. * condition.
*/ */
void refcount_inc(refcount_t *r) void refcount_inc_checked(refcount_t *r)
{ {
WARN_ONCE(!refcount_inc_not_zero(r), "refcount_t: increment on 0; use-after-free.\n"); WARN_ONCE(!refcount_inc_not_zero_checked(r), "refcount_t: increment on 0; use-after-free.\n");
} }
EXPORT_SYMBOL(refcount_inc); EXPORT_SYMBOL(refcount_inc_checked);
/** /**
* refcount_sub_and_test - subtract from a refcount and test if it is 0 * refcount_sub_and_test_checked - subtract from a refcount and test if it is 0
* @i: amount to subtract from the refcount * @i: amount to subtract from the refcount
* @r: the refcount * @r: the refcount
* *
...@@ -174,7 +174,7 @@ EXPORT_SYMBOL(refcount_inc); ...@@ -174,7 +174,7 @@ EXPORT_SYMBOL(refcount_inc);
* *
* Return: true if the resulting refcount is 0, false otherwise * Return: true if the resulting refcount is 0, false otherwise
*/ */
bool refcount_sub_and_test(unsigned int i, refcount_t *r) bool refcount_sub_and_test_checked(unsigned int i, refcount_t *r)
{ {
unsigned int new, val = atomic_read(&r->refs); unsigned int new, val = atomic_read(&r->refs);
...@@ -192,10 +192,10 @@ bool refcount_sub_and_test(unsigned int i, refcount_t *r) ...@@ -192,10 +192,10 @@ bool refcount_sub_and_test(unsigned int i, refcount_t *r)
return !new; return !new;
} }
EXPORT_SYMBOL(refcount_sub_and_test); EXPORT_SYMBOL(refcount_sub_and_test_checked);
/** /**
* refcount_dec_and_test - decrement a refcount and test if it is 0 * refcount_dec_and_test_checked - decrement a refcount and test if it is 0
* @r: the refcount * @r: the refcount
* *
* Similar to atomic_dec_and_test(), it will WARN on underflow and fail to * Similar to atomic_dec_and_test(), it will WARN on underflow and fail to
...@@ -207,14 +207,14 @@ EXPORT_SYMBOL(refcount_sub_and_test); ...@@ -207,14 +207,14 @@ EXPORT_SYMBOL(refcount_sub_and_test);
* *
* Return: true if the resulting refcount is 0, false otherwise * Return: true if the resulting refcount is 0, false otherwise
*/ */
bool refcount_dec_and_test(refcount_t *r) bool refcount_dec_and_test_checked(refcount_t *r)
{ {
return refcount_sub_and_test(1, r); return refcount_sub_and_test_checked(1, r);
} }
EXPORT_SYMBOL(refcount_dec_and_test); EXPORT_SYMBOL(refcount_dec_and_test_checked);
/** /**
* refcount_dec - decrement a refcount * refcount_dec_checked - decrement a refcount
* @r: the refcount * @r: the refcount
* *
* Similar to atomic_dec(), it will WARN on underflow and fail to decrement * Similar to atomic_dec(), it will WARN on underflow and fail to decrement
...@@ -223,12 +223,11 @@ EXPORT_SYMBOL(refcount_dec_and_test); ...@@ -223,12 +223,11 @@ EXPORT_SYMBOL(refcount_dec_and_test);
* Provides release memory ordering, such that prior loads and stores are done * Provides release memory ordering, such that prior loads and stores are done
* before. * before.
*/ */
void refcount_dec(refcount_t *r) void refcount_dec_checked(refcount_t *r)
{ {
WARN_ONCE(refcount_dec_and_test(r), "refcount_t: decrement hit 0; leaking memory.\n"); WARN_ONCE(refcount_dec_and_test_checked(r), "refcount_t: decrement hit 0; leaking memory.\n");
} }
EXPORT_SYMBOL(refcount_dec); EXPORT_SYMBOL(refcount_dec_checked);
#endif /* CONFIG_REFCOUNT_FULL */
/** /**
* refcount_dec_if_one - decrement a refcount if it is 1 * refcount_dec_if_one - decrement a refcount if it is 1
......
...@@ -244,7 +244,7 @@ static int pppoatm_may_send(struct pppoatm_vcc *pvcc, int size) ...@@ -244,7 +244,7 @@ static int pppoatm_may_send(struct pppoatm_vcc *pvcc, int size)
* the packet count limit, so... * the packet count limit, so...
*/ */
if (atm_may_send(pvcc->atmvcc, size) && if (atm_may_send(pvcc->atmvcc, size) &&
atomic_inc_not_zero_hint(&pvcc->inflight, NONE_INFLIGHT)) atomic_inc_not_zero(&pvcc->inflight))
return 1; return 1;
/* /*
......
...@@ -415,7 +415,7 @@ void rxrpc_incoming_call(struct rxrpc_sock *rx, ...@@ -415,7 +415,7 @@ void rxrpc_incoming_call(struct rxrpc_sock *rx,
bool rxrpc_queue_call(struct rxrpc_call *call) bool rxrpc_queue_call(struct rxrpc_call *call)
{ {
const void *here = __builtin_return_address(0); const void *here = __builtin_return_address(0);
int n = __atomic_add_unless(&call->usage, 1, 0); int n = atomic_fetch_add_unless(&call->usage, 1, 0);
if (n == 0) if (n == 0)
return false; return false;
if (rxrpc_queue_work(&call->processor)) if (rxrpc_queue_work(&call->processor))
......
...@@ -266,7 +266,7 @@ void rxrpc_kill_connection(struct rxrpc_connection *conn) ...@@ -266,7 +266,7 @@ void rxrpc_kill_connection(struct rxrpc_connection *conn)
bool rxrpc_queue_conn(struct rxrpc_connection *conn) bool rxrpc_queue_conn(struct rxrpc_connection *conn)
{ {
const void *here = __builtin_return_address(0); const void *here = __builtin_return_address(0);
int n = __atomic_add_unless(&conn->usage, 1, 0); int n = atomic_fetch_add_unless(&conn->usage, 1, 0);
if (n == 0) if (n == 0)
return false; return false;
if (rxrpc_queue_work(&conn->processor)) if (rxrpc_queue_work(&conn->processor))
...@@ -309,7 +309,7 @@ rxrpc_get_connection_maybe(struct rxrpc_connection *conn) ...@@ -309,7 +309,7 @@ rxrpc_get_connection_maybe(struct rxrpc_connection *conn)
const void *here = __builtin_return_address(0); const void *here = __builtin_return_address(0);
if (conn) { if (conn) {
int n = __atomic_add_unless(&conn->usage, 1, 0); int n = atomic_fetch_add_unless(&conn->usage, 1, 0);
if (n > 0) if (n > 0)
trace_rxrpc_conn(conn, rxrpc_conn_got, n + 1, here); trace_rxrpc_conn(conn, rxrpc_conn_got, n + 1, here);
else else
......
...@@ -305,7 +305,7 @@ struct rxrpc_local *rxrpc_get_local_maybe(struct rxrpc_local *local) ...@@ -305,7 +305,7 @@ struct rxrpc_local *rxrpc_get_local_maybe(struct rxrpc_local *local)
const void *here = __builtin_return_address(0); const void *here = __builtin_return_address(0);
if (local) { if (local) {
int n = __atomic_add_unless(&local->usage, 1, 0); int n = atomic_fetch_add_unless(&local->usage, 1, 0);
if (n > 0) if (n > 0)
trace_rxrpc_local(local, rxrpc_local_got, n + 1, here); trace_rxrpc_local(local, rxrpc_local_got, n + 1, here);
else else
......
...@@ -406,7 +406,7 @@ struct rxrpc_peer *rxrpc_get_peer_maybe(struct rxrpc_peer *peer) ...@@ -406,7 +406,7 @@ struct rxrpc_peer *rxrpc_get_peer_maybe(struct rxrpc_peer *peer)
const void *here = __builtin_return_address(0); const void *here = __builtin_return_address(0);
if (peer) { if (peer) {
int n = __atomic_add_unless(&peer->usage, 1, 0); int n = atomic_fetch_add_unless(&peer->usage, 1, 0);
if (n > 0) if (n > 0)
trace_rxrpc_peer(peer, rxrpc_peer_got, n + 1, here); trace_rxrpc_peer(peer, rxrpc_peer_got, n + 1, here);
else else
......
...@@ -804,7 +804,7 @@ type of fence: ...@@ -804,7 +804,7 @@ type of fence:
Second, some types of fence affect the way the memory subsystem Second, some types of fence affect the way the memory subsystem
propagates stores. When a fence instruction is executed on CPU C: propagates stores. When a fence instruction is executed on CPU C:
For each other CPU C', smb_wmb() forces all po-earlier stores For each other CPU C', smp_wmb() forces all po-earlier stores
on C to propagate to C' before any po-later stores do. on C to propagate to C' before any po-later stores do.
For each other CPU C', any store which propagates to C before For each other CPU C', any store which propagates to C before
......
...@@ -126,7 +126,7 @@ However, it is not necessarily the case that accesses ordered by ...@@ -126,7 +126,7 @@ However, it is not necessarily the case that accesses ordered by
locking will be seen as ordered by CPUs not holding that lock. locking will be seen as ordered by CPUs not holding that lock.
Consider this example: Consider this example:
/* See Z6.0+pooncelock+pooncelock+pombonce.litmus. */ /* See Z6.0+pooncerelease+poacquirerelease+fencembonceonce.litmus. */
void CPU0(void) void CPU0(void)
{ {
spin_lock(&mylock); spin_lock(&mylock);
...@@ -292,7 +292,7 @@ and to use smp_load_acquire() instead of smp_rmb(). However, the older ...@@ -292,7 +292,7 @@ and to use smp_load_acquire() instead of smp_rmb(). However, the older
smp_wmb() and smp_rmb() APIs are still heavily used, so it is important smp_wmb() and smp_rmb() APIs are still heavily used, so it is important
to understand their use cases. The general approach is shown below: to understand their use cases. The general approach is shown below:
/* See MP+wmbonceonce+rmbonceonce.litmus. */ /* See MP+fencewmbonceonce+fencermbonceonce.litmus. */
void CPU0(void) void CPU0(void)
{ {
WRITE_ONCE(x, 1); WRITE_ONCE(x, 1);
...@@ -322,9 +322,9 @@ the following write-side code fragment: ...@@ -322,9 +322,9 @@ the following write-side code fragment:
And the xlog_valid_lsn() function in fs/xfs/xfs_log_priv.h contains And the xlog_valid_lsn() function in fs/xfs/xfs_log_priv.h contains
the corresponding read-side code fragment: the corresponding read-side code fragment:
cur_cycle = ACCESS_ONCE(log->l_curr_cycle); cur_cycle = READ_ONCE(log->l_curr_cycle);
smp_rmb(); smp_rmb();
cur_block = ACCESS_ONCE(log->l_curr_block); cur_block = READ_ONCE(log->l_curr_block);
Alternatively, consider the following comment in function Alternatively, consider the following comment in function
perf_output_put_handle() in kernel/events/ring_buffer.c: perf_output_put_handle() in kernel/events/ring_buffer.c:
...@@ -360,7 +360,7 @@ can be seen in the LB+poonceonces.litmus litmus test. ...@@ -360,7 +360,7 @@ can be seen in the LB+poonceonces.litmus litmus test.
One way of avoiding the counter-intuitive outcome is through the use of a One way of avoiding the counter-intuitive outcome is through the use of a
control dependency paired with a full memory barrier: control dependency paired with a full memory barrier:
/* See LB+ctrlonceonce+mbonceonce.litmus. */ /* See LB+fencembonceonce+ctrlonceonce.litmus. */
void CPU0(void) void CPU0(void)
{ {
r0 = READ_ONCE(x); r0 = READ_ONCE(x);
...@@ -476,7 +476,7 @@ that one CPU first stores to one variable and then loads from a second, ...@@ -476,7 +476,7 @@ that one CPU first stores to one variable and then loads from a second,
while another CPU stores to the second variable and then loads from the while another CPU stores to the second variable and then loads from the
first. Preserving order requires nothing less than full barriers: first. Preserving order requires nothing less than full barriers:
/* See SB+mbonceonces.litmus. */ /* See SB+fencembonceonces.litmus. */
void CPU0(void) void CPU0(void)
{ {
WRITE_ONCE(x, 1); WRITE_ONCE(x, 1);
......
...@@ -35,13 +35,13 @@ BASIC USAGE: HERD7 ...@@ -35,13 +35,13 @@ BASIC USAGE: HERD7
The memory model is used, in conjunction with "herd7", to exhaustively The memory model is used, in conjunction with "herd7", to exhaustively
explore the state space of small litmus tests. explore the state space of small litmus tests.
For example, to run SB+mbonceonces.litmus against the memory model: For example, to run SB+fencembonceonces.litmus against the memory model:
$ herd7 -conf linux-kernel.cfg litmus-tests/SB+mbonceonces.litmus $ herd7 -conf linux-kernel.cfg litmus-tests/SB+fencembonceonces.litmus
Here is the corresponding output: Here is the corresponding output:
Test SB+mbonceonces Allowed Test SB+fencembonceonces Allowed
States 3 States 3
0:r0=0; 1:r0=1; 0:r0=0; 1:r0=1;
0:r0=1; 1:r0=0; 0:r0=1; 1:r0=0;
...@@ -50,8 +50,8 @@ Here is the corresponding output: ...@@ -50,8 +50,8 @@ Here is the corresponding output:
Witnesses Witnesses
Positive: 0 Negative: 3 Positive: 0 Negative: 3
Condition exists (0:r0=0 /\ 1:r0=0) Condition exists (0:r0=0 /\ 1:r0=0)
Observation SB+mbonceonces Never 0 3 Observation SB+fencembonceonces Never 0 3
Time SB+mbonceonces 0.01 Time SB+fencembonceonces 0.01
Hash=d66d99523e2cac6b06e66f4c995ebb48 Hash=d66d99523e2cac6b06e66f4c995ebb48
The "Positive: 0 Negative: 3" and the "Never 0 3" each indicate that The "Positive: 0 Negative: 3" and the "Never 0 3" each indicate that
...@@ -67,16 +67,16 @@ BASIC USAGE: KLITMUS7 ...@@ -67,16 +67,16 @@ BASIC USAGE: KLITMUS7
The "klitmus7" tool converts a litmus test into a Linux kernel module, The "klitmus7" tool converts a litmus test into a Linux kernel module,
which may then be loaded and run. which may then be loaded and run.
For example, to run SB+mbonceonces.litmus against hardware: For example, to run SB+fencembonceonces.litmus against hardware:
$ mkdir mymodules $ mkdir mymodules
$ klitmus7 -o mymodules litmus-tests/SB+mbonceonces.litmus $ klitmus7 -o mymodules litmus-tests/SB+fencembonceonces.litmus
$ cd mymodules ; make $ cd mymodules ; make
$ sudo sh run.sh $ sudo sh run.sh
The corresponding output includes: The corresponding output includes:
Test SB+mbonceonces Allowed Test SB+fencembonceonces Allowed
Histogram (3 states) Histogram (3 states)
644580 :>0:r0=1; 1:r0=0; 644580 :>0:r0=1; 1:r0=0;
644328 :>0:r0=0; 1:r0=1; 644328 :>0:r0=0; 1:r0=1;
...@@ -86,8 +86,8 @@ The corresponding output includes: ...@@ -86,8 +86,8 @@ The corresponding output includes:
Positive: 0, Negative: 2000000 Positive: 0, Negative: 2000000
Condition exists (0:r0=0 /\ 1:r0=0) is NOT validated Condition exists (0:r0=0 /\ 1:r0=0) is NOT validated
Hash=d66d99523e2cac6b06e66f4c995ebb48 Hash=d66d99523e2cac6b06e66f4c995ebb48
Observation SB+mbonceonces Never 0 2000000 Observation SB+fencembonceonces Never 0 2000000
Time SB+mbonceonces 0.16 Time SB+fencembonceonces 0.16
The "Positive: 0 Negative: 2000000" and the "Never 0 2000000" indicate The "Positive: 0 Negative: 2000000" and the "Never 0 2000000" indicate
that during two million trials, the state specified in this litmus that during two million trials, the state specified in this litmus
......
...@@ -13,7 +13,7 @@ ...@@ -13,7 +13,7 @@
"Linux-kernel memory consistency model" "Linux-kernel memory consistency model"
enum Accesses = 'once (*READ_ONCE,WRITE_ONCE,ACCESS_ONCE*) || enum Accesses = 'once (*READ_ONCE,WRITE_ONCE*) ||
'release (*smp_store_release*) || 'release (*smp_store_release*) ||
'acquire (*smp_load_acquire*) || 'acquire (*smp_load_acquire*) ||
'noreturn (* R of non-return RMW *) 'noreturn (* R of non-return RMW *)
......
C ISA2+pooncelock+pooncelock+pombonce.litmus C ISA2+pooncelock+pooncelock+pombonce
(* (*
* Result: Sometimes * Result: Sometimes
......
C MP+wmbonceonce+rmbonceonce C MP+fencewmbonceonce+fencermbonceonce
(* (*
* Result: Never * Result: Never
......
...@@ -18,7 +18,7 @@ CoWW+poonceonce.litmus ...@@ -18,7 +18,7 @@ CoWW+poonceonce.litmus
Test of write-write coherence, that is, whether or not two Test of write-write coherence, that is, whether or not two
successive writes to the same variable are ordered. successive writes to the same variable are ordered.
IRIW+mbonceonces+OnceOnce.litmus IRIW+fencembonceonces+OnceOnce.litmus
Test of independent reads from independent writes with smp_mb() Test of independent reads from independent writes with smp_mb()
between each pairs of reads. In other words, is smp_mb() between each pairs of reads. In other words, is smp_mb()
sufficient to cause two different reading processes to agree on sufficient to cause two different reading processes to agree on
...@@ -47,7 +47,7 @@ ISA2+pooncerelease+poacquirerelease+poacquireonce.litmus ...@@ -47,7 +47,7 @@ ISA2+pooncerelease+poacquirerelease+poacquireonce.litmus
Can a release-acquire chain order a prior store against Can a release-acquire chain order a prior store against
a later load? a later load?
LB+ctrlonceonce+mbonceonce.litmus LB+fencembonceonce+ctrlonceonce.litmus
Does a control dependency and an smp_mb() suffice for the Does a control dependency and an smp_mb() suffice for the
load-buffering litmus test, where each process reads from one load-buffering litmus test, where each process reads from one
of two variables then writes to the other? of two variables then writes to the other?
...@@ -88,14 +88,14 @@ MP+porevlocks.litmus ...@@ -88,14 +88,14 @@ MP+porevlocks.litmus
As below, but with the first access of the writer process As below, but with the first access of the writer process
and the second access of reader process protected by a lock. and the second access of reader process protected by a lock.
MP+wmbonceonce+rmbonceonce.litmus MP+fencewmbonceonce+fencermbonceonce.litmus
Does a smp_wmb() (between the stores) and an smp_rmb() (between Does a smp_wmb() (between the stores) and an smp_rmb() (between
the loads) suffice for the message-passing litmus test, where one the loads) suffice for the message-passing litmus test, where one
process writes data and then a flag, and the other process reads process writes data and then a flag, and the other process reads
the flag and then the data. (This is similar to the ISA2 tests, the flag and then the data. (This is similar to the ISA2 tests,
but with two processes instead of three.) but with two processes instead of three.)
R+mbonceonces.litmus R+fencembonceonces.litmus
This is the fully ordered (via smp_mb()) version of one of This is the fully ordered (via smp_mb()) version of one of
the classic counterintuitive litmus tests that illustrates the the classic counterintuitive litmus tests that illustrates the
effects of store propagation delays. effects of store propagation delays.
...@@ -103,7 +103,7 @@ R+mbonceonces.litmus ...@@ -103,7 +103,7 @@ R+mbonceonces.litmus
R+poonceonces.litmus R+poonceonces.litmus
As above, but without the smp_mb() invocations. As above, but without the smp_mb() invocations.
SB+mbonceonces.litmus SB+fencembonceonces.litmus
This is the fully ordered (again, via smp_mb() version of store This is the fully ordered (again, via smp_mb() version of store
buffering, which forms the core of Dekker's mutual-exclusion buffering, which forms the core of Dekker's mutual-exclusion
algorithm. algorithm.
...@@ -111,15 +111,24 @@ SB+mbonceonces.litmus ...@@ -111,15 +111,24 @@ SB+mbonceonces.litmus
SB+poonceonces.litmus SB+poonceonces.litmus
As above, but without the smp_mb() invocations. As above, but without the smp_mb() invocations.
SB+rfionceonce-poonceonces.litmus
This litmus test demonstrates that LKMM is not fully multicopy
atomic. (Neither is it other multicopy atomic.) This litmus test
also demonstrates the "locations" debugging aid, which designates
additional registers and locations to be printed out in the dump
of final states in the herd7 output. Without the "locations"
statement, only those registers and locations mentioned in the
"exists" clause will be printed.
S+poonceonces.litmus S+poonceonces.litmus
As below, but without the smp_wmb() and acquire load. As below, but without the smp_wmb() and acquire load.
S+wmbonceonce+poacquireonce.litmus S+fencewmbonceonce+poacquireonce.litmus
Can a smp_wmb(), instead of a release, and an acquire order Can a smp_wmb(), instead of a release, and an acquire order
a prior store against a subsequent store? a prior store against a subsequent store?
WRC+poonceonces+Once.litmus WRC+poonceonces+Once.litmus
WRC+pooncerelease+rmbonceonce+Once.litmus WRC+pooncerelease+fencermbonceonce+Once.litmus
These two are members of an extension of the MP litmus-test These two are members of an extension of the MP litmus-test
class in which the first write is moved to a separate process. class in which the first write is moved to a separate process.
The second is forbidden because smp_store_release() is The second is forbidden because smp_store_release() is
...@@ -134,7 +143,7 @@ Z6.0+pooncelock+poonceLock+pombonce.litmus ...@@ -134,7 +143,7 @@ Z6.0+pooncelock+poonceLock+pombonce.litmus
As above, but with smp_mb__after_spinlock() immediately As above, but with smp_mb__after_spinlock() immediately
following the spin_lock(). following the spin_lock().
Z6.0+pooncerelease+poacquirerelease+mbonceonce.litmus Z6.0+pooncerelease+poacquirerelease+fencembonceonce.litmus
Is the ordering provided by a release-acquire chain sufficient Is the ordering provided by a release-acquire chain sufficient
to make ordering apparent to accesses by a process that does to make ordering apparent to accesses by a process that does
not participate in that release-acquire chain? not participate in that release-acquire chain?
......
C S+wmbonceonce+poacquireonce C S+fencewmbonceonce+poacquireonce
(* (*
* Result: Never * Result: Never
......
C SB+rfionceonce-poonceonces
(*
* Result: Sometimes
*
* This litmus test demonstrates that LKMM is not fully multicopy atomic.
*)
{}
P0(int *x, int *y)
{
int r1;
int r2;
WRITE_ONCE(*x, 1);
r1 = READ_ONCE(*x);
r2 = READ_ONCE(*y);
}
P1(int *x, int *y)
{
int r3;
int r4;
WRITE_ONCE(*y, 1);
r3 = READ_ONCE(*y);
r4 = READ_ONCE(*x);
}
locations [0:r1; 1:r3; x; y] (* Debug aid: Print things not in "exists". *)
exists (0:r2=0 /\ 1:r4=0)
C WRC+pooncerelease+rmbonceonce+Once C WRC+pooncerelease+fencermbonceonce+Once
(* (*
* Result: Never * Result: Never
......
C Z6.0+pooncerelease+poacquirerelease+mbonceonce C Z6.0+pooncerelease+poacquirerelease+fencembonceonce
(* (*
* Result: Sometimes * Result: Sometimes
......
...@@ -9,7 +9,7 @@ ...@@ -9,7 +9,7 @@
# appended. # appended.
# #
# Usage: # Usage:
# sh checkalllitmus.sh [ directory ] # checkalllitmus.sh [ directory ]
# #
# The LINUX_HERD_OPTIONS environment variable may be used to specify # The LINUX_HERD_OPTIONS environment variable may be used to specify
# arguments to herd, whose default is defined by the checklitmus.sh script. # arguments to herd, whose default is defined by the checklitmus.sh script.
......
...@@ -8,7 +8,7 @@ ...@@ -8,7 +8,7 @@
# with ".out" appended. # with ".out" appended.
# #
# Usage: # Usage:
# sh checklitmus.sh file.litmus # checklitmus.sh file.litmus
# #
# The LINUX_HERD_OPTIONS environment variable may be used to specify # The LINUX_HERD_OPTIONS environment variable may be used to specify
# arguments to herd, which default to "-conf linux-kernel.cfg". Thus, # arguments to herd, which default to "-conf linux-kernel.cfg". Thus,
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment