- 15 Sep, 2015 1 commit
-
-
Jonathan Corbet authored
Fix a few small mistakes in the static key documentation and delete an unneeded sentence. Suggested-by:
Jason Baron <jbaron@akamai.com> Signed-off-by:
Jonathan Corbet <corbet@lwn.net> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: http://lkml.kernel.org/r/20150914171105.511e1e21@lwn.netSigned-off-by:
Ingo Molnar <mingo@kernel.org>
-
- 11 Sep, 2015 2 commits
-
-
Peter Zijlstra authored
Only emit the test-and-set fallback for Hypervisors lacking PARAVIRT_SPINLOCKS support when building for guests. Suggested-by:
Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by:
Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: linux-kernel@vger.kernel.org Cc: stable@vger.kernel.org # 4.2 Signed-off-by:
-
Peter Zijlstra authored
Dave ran into horrible performance on a VM without PARAVIRT_SPINLOCKS set and Linus noted that the test-and-set implementation was retarded. One should spin on the variable with a load, not a RMW. While there, remove 'queued' from the name, as the lock isn't queued at all, but a simple test-and-set. Suggested-by:
Dave Chinner <david@fromorbit.com> Tested-by:
-
- 08 Sep, 2015 1 commit
-
-
Jonathan Corbet authored
Commit: 412758cb ("jump label, locking/static_keys: Update docs") introduced a typo that might as well get fixed. Signed-off-by:
-
- 13 Aug, 2015 1 commit
-
-
Peter Zijlstra authored
Boris reported that gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) fails to build linux-next kernels that have this fresh commit via the locking tree: 11276d53 ("locking/static_keys: Add a new static_key interface") The problem appears to be that even though @key and @branch are compile time constants, it doesn't see the following expression as an immediate value: &((char *)key)[branch] More recent GCCs don't appear to have this problem. In particular, Red Hat backported the 'asm goto' feature into 4.4, 'normal' 4.4 compilers will not have this feature and thus not run into this asm. The workaround is to supply both values to the asm as immediates and do the addition in asm. Suggested-by:
H. Peter Anvin <hpa@zytor.com> Reported-by:
Boris Ostrovsky <boris.ostrovsky@oracle.com> Tested-by:
-
- 12 Aug, 2015 10 commits
-
-
Will Deacon authored
By defining our SMP atomics in terms of relaxed operations, we gain a small reduction in code size and have acquire/release/fence variants generated automatically by the core code. Signed-off-by:
Will Deacon <will.deacon@arm.com> Signed-off-by:
-
Will Deacon authored
Including an asm/ header directly is best avoided, so use linux/atomic.h instead of asm/cmpxchg.h in linux/llist.h. Signed-off-by:
-
Will Deacon authored
The qrwlock implementation is slightly heavy in its use of memory barriers, mainly through the use of _cmpxchg() and _return() atomics, which imply full barrier semantics. This patch modifies the qrwlock code to use the more relaxed atomic routines so that we can reduce the unnecessary barrier overhead on weakly-ordered architectures. Signed-off-by:
-
Will Deacon authored
Since the following commit: 536fa402 ("compiler: Allow 1- and 2-byte smp_load_acquire() and smp_store_release()") smp_store_release() supports byte accesses, so use that in writer unlock and remove the conditional macro override. Signed-off-by:
Waiman Long <Waiman.Long@hp.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: paulmck@linux.vnet.ibm.com Link: http://lkml.kernel.org/r/1438880084-18856-6-git-send-email-will.deacon@arm.comSigned-off-by:
-
Will Deacon authored
cmpxchg64_relaxed() is now defined by linux/atomic.h, so we can remove our local definition from the lockref code. Signed-off-by:
-
Will Deacon authored
This patch adds 'atomic_long_t' wrappers for the new relaxed atomic operations. Signed-off-by:
-
Will Deacon authored
We can use some (admittedly ugly) macros to generate the 32-bit and 64-bit based atomic_long implementations from the same code. Signed-off-by:
-
Will Deacon authored
Whilst porting the generic qrwlock code over to arm64, it became apparent that any portable locking code needs finer-grained control of the memory-ordering guarantees provided by our atomic routines. In particular: xchg, cmpxchg, {add,sub}_return are often used in situations where full barrier semantics (currently the only option available) are not required. For example, when a reader increments a reader count to obtain a lock, checking the old value to see if a writer was present, only acquire semantics are strictly needed. This patch introduces three new ordering semantics for these operations: - *_relaxed: No ordering guarantees. This is similar to what we have already for the non-return atomics (e.g. atomic_add). - *_acquire: ACQUIRE semantics, similar to smp_load_acquire. - *_release: RELEASE semantics, similar to smp_store_release. In memory-ordering speak, this means that the acquire/release semantics are RCpc as opposed to RCsc. Consequently a RELEASE followed by an ACQUIRE does not imply a full barrier, as already documented in memory-barriers.txt. Currently, all the new macros are conditionally mapped to the full-mb variants, however if the *_relaxed version is provided by the architecture, then the acquire/release variants are constructed by supplementing the relaxed routine with an explicit barrier. Signed-off-by: -
Christian Borntraeger authored
The kernel build bot showed a new warning triggered by commit: 76695af2 ("locking, arch: use WRITE_ONCE()/READ_ONCE() in smp_store_release()/smp_load_acquire()") because Sparse does not like WRITE_ONCE() accessing elements from the (sparse) RCU address space: fs/afs/inode.c:448:9: sparse: incorrect type in initializer (different address spaces) fs/afs/inode.c:448:9: expected struct afs_permits *__val fs/afs/inode.c:448:9: got void [noderef] <asn:4>*<noident> Solution is to force cast away the sparse attributes for the initializer of the union in WRITE_ONCE(). (And as this now gets too long, also split the macro into multiple lines.) Signed-off-by:
Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by:
-
Ingo Molnar authored
Signed-off-by:
-
- 05 Aug, 2015 1 commit
-
-
kbuild test robot authored
Signed-off-by:
Fengguang Wu <fengguang.wu@intel.com> Cc: Jason Baron <jbaron@akamai.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: http://lkml.kernel.org/r/20150803184748.GA80634@lkp-ib04Signed-off-by:
-
- 03 Aug, 2015 22 commits
-
-
Jason Baron authored
Signed-off-by:
-
Ingo Molnar authored
The 'jump label' self-test is in reality testing static keys - rename things accordingly. Also prettify the code in various places while at it. Acked-by:
-
Jason Baron authored
Signed-off-by:
-
Heiko Carstens authored
Use the new static_branch_likely() primitive to make sure that the most likely case is executed without taking an unconditional branch. This wasn't possible with the old jump label primitives. Signed-off-by:
Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by:
-
Peter Zijlstra authored
Because of the static_key restrictions we had to take an unconditional jump for the most likely case, causing $I bloat. Rewrite to use the new primitives. Signed-off-by:
-
Peter Zijlstra authored
Add a little selftest that validates all combinations. Signed-off-by:
-
Peter Zijlstra authored
There are various problems and short-comings with the current static_key interface: - static_key_{true,false}() read like a branch depending on the key value, instead of the actual likely/unlikely branch depending on init value. - static_key_{true,false}() are, as stated above, tied to the static_key init values STATIC_KEY_INIT_{TRUE,FALSE}. - we're limited to the 2 (out of 4) possible options that compile to a default NOP because that's what our arch_static_branch() assembly emits. So provide a new static_key interface: DEFINE_STATIC_KEY_TRUE(name); DEFINE_STATIC_KEY_FALSE(name); Which define a key of different types with an initial true/false value. Then allow: static_branch_likely() static_branch_unlikely() to take a key of either type and emit the right instruction for the case. This means adding a second arch_static_branch_jump() assembly helper which emits a JMP per default. In order to determine the right instruction for the right state, encode the branch type in the LSB of jump_entry::key. This is the final step in removing the naming confusion that has led to a stream of avoidable bugs such as: a833581e ("x86, perf: Fix static_key bug in load_mm_cr4()") ... but it also allows new static key combinations that will give us performance enhancements in the subsequent patches. Tested-by: Rabin Vincent <rabin@rab.in> # arm Signed-off-by: -
Peter Zijlstra authored
Instead of spreading the branch_default logic all over the place, concentrate it into the one jump_label_type() function. This does mean we need to actually increment/decrement the enabled count _before_ calling the update path, otherwise jump_label_type() will not see the right state. Signed-off-by:
-
Peter Zijlstra authored
Add two helpers to make it easier to treat the refcount as boolean. Suggested-by:
Jason Baron <jasonbaron0@gmail.com> Signed-off-by:
-
Peter Zijlstra authored
Avoid some casting with a helper, also prepares the way for overloading the LSB of jump_entry::key. Signed-off-by:
-
Peter Zijlstra authored
jump_label, locking/static_keys: Rename JUMP_LABEL_TYPE_* and related helpers to the static_key* pattern Rename the JUMP_LABEL_TYPE_* macros to be JUMP_TYPE_* and move the inline helpers into kernel/jump_label.c, since that's the only place they're ever used. Also rename the helpers where it's all about static keys. This is the second step in removing the naming confusion that has led to a stream of avoidable bugs such as: a833581e ("x86, perf: Fix static_key bug in load_mm_cr4()") Signed-off-by:
-
Peter Zijlstra authored
Since we've already stepped away from ENABLE is a JMP and DISABLE is a NOP with the branch_default bits, and are going to make it even worse, rename it to make it all clearer. This way we don't mix multiple levels of logic attributes, but have a plain 'physical' name for what the current instruction patching status of a jump label is. This is a first step in removing the naming confusion that has led to a stream of avoidable bugs such as: a833581e ("x86, perf: Fix static_key bug in load_mm_cr4()") Signed-off-by:
-
Ingo Molnar authored
Upcoming changes to static keys is interacting/conflicting with the following pending TSC commits in tip:x86/asm: 4ea1636b x86/asm/tsc: Rename native_read_tsc() to rdtsc() ... So merge it into the locking tree to have a smoother resolution. Signed-off-by:
-
Andrey Konovalov authored
Replace ACCESS_ONCE() macro in smp_store_release() and smp_load_acquire() with WRITE_ONCE() and READ_ONCE() on x86, arm, arm64, ia64, metag, mips, powerpc, s390, sparc and asm-generic since ACCESS_ONCE() does not work reliably on non-scalar types. WRITE_ONCE() and READ_ONCE() were introduced in the following commits: 230fa253 ("kernel: Provide READ_ONCE and ASSIGN_ONCE") 43239cbe ("kernel: Change ASSIGN_ONCE(val, x) to WRITE_ONCE(x, val)") Signed-off-by:
Andrey Konovalov <andreyknvl@google.com> Signed-off-by:
Davidlohr Bueso <dbueso@suse.de> Acked-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc) Acked-by:
Ralf Baechle <ralf@linux-mips.org> Cc: Alexander Duyck <alexander.h.duyck@redhat.com> Cc: Andre Przywara <andre.przywara@arm.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Borislav Petkov <bp@suse.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christian Borntraeger <borntraeger@de.ibm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Fenghua Yu <fenghua.yu@intel.com> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: James Hogan <james.hogan@imgtec.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Russell King <linux@arm.linux.org.uk> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Tony Luck <tony.luck@intel.com> Cc: Will Deacon <will.deacon@arm.com> Cc: linux-arch@vger.kernel.org Link: http://lkml.kernel.org/r/1438528264-714-1-git-send-email-andreyknvl@google.comSigned-off-by:
-
Waiman Long authored
For an over-committed guest with more vCPUs than physical CPUs available, it is possible that a vCPU may be kicked twice before getting the lock - once before it becomes queue head and once again before it gets the lock. All these CPU kicking and halting (VMEXIT) can be expensive and slow down system performance. This patch adds a new vCPU state (vcpu_hashed) which enables the code to delay CPU kicking until at unlock time. Once this state is set, the new lock holder will set _Q_SLOW_VAL and fill in the hash table on behalf of the halted queue head vCPU. The original vcpu_halted state will be used by pv_wait_node() only to differentiate other queue nodes from the qeue head. Signed-off-by:
-
Waiman Long authored
Currently, a reader will check first to make sure that the writer mode byte is cleared before incrementing the reader count. That waiting is not really necessary. It increases the latency in the reader/writer to reader transition and reduces readers performance. This patch eliminates that waiting. It also has the side effect of reducing the chance of writer lock stealing and improving the fairness of the lock. Using a locking microbenchmark, a 10-threads 5M locking loop of mostly readers (RW ratio = 10,000:1) has the following performance numbers in a Haswell-EX box: Kernel Locking Rate (Kops/s) ------ --------------------- 4.1.1 15,063,081 4.1.1+patch 17,241,552 (+14.4%) Signed-off-by: -
Will Deacon authored
When we unlock in __pv_queued_spin_unlock(), a failed cmpxchg() on the lock value indicates that we need to take the slow-path and unhash the corresponding node blocked on the lock. Since a failed cmpxchg() does not provide any memory-ordering guarantees, it is possible that the node data could be read before the cmpxchg() on weakly-ordered architectures and therefore return a stale value, leading to hash corruption and/or a BUG(). This patch adds an smb_rmb() following the failed cmpxchg operation, so that the unhashing is ordered after the lock has been checked. Reported-by:
-
Will Deacon authored
A failed cmpxchg does not provide any memory ordering guarantees, a property that is used to optimise the cmpxchg implementations on Alpha, PowerPC and arm64. This patch updates atomic_ops.txt and memory-barriers.txt to reflect this. Signed-off-by:
-
Peter Zijlstra authored
- Rename the on-stack variable to match the datastructure variable, - place the cmpxchg back under the comment that explains it, - clean up the WARN() statement to avoid superfluous conditionals and line-breaks. Signed-off-by:
-
Ingo Molnar authored
Merge branch 'locking/urgent', tag 'v4.2-rc5' into locking/core, to pick up fixes before applying new changes Signed-off-by: -
Linus Torvalds authored
-
git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linuxLinus Torvalds authored
Pull powerpc fixes from Michael Ellerman: - TCE table memory calculation fix from Alexey - Build fix for ans-lcd from Luis - Unbalanced IRQ warning fix from Alistair * tag 'powerpc-4.2-3' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux: powerpc/eeh-powernv: Fix unbalanced IRQ warning macintosh/ans-lcd: fix build failure after module_init/exit relocation powerpc/powernv/ioda2: Fix calculation for memory allocated for TCE table
-
- 02 Aug, 2015 2 commits
-
-
Linus Torvalds authored
Ted Ts'o reports that his Lenovo T540p ThinkPad crashes at boot if attached to the docking station. This is a regression that he was able to bisect to commit 8c7b5ccb: "drm/i915: Use atomic helpers for computing changed flags:" The reason seems to be the new call to drm_atomic_helper_check_modeset() added to intel_modeset_compute_config(), which in turn calls update_connector_routing(), and somehow ends up picking a NULL crtc for the connector state, causing the subsequent drm_crtc_index() to OOPS. Daniel Vetter says that the fundamental issue seems to be confusion in the encoder selection, and this isn't the right fix, but while he chases down the proper fix, this at least avoids the NULL pointer dereference and makes Ted's docking station work again. Reported-bisected-and-tested-by:
Theodore Ts'o <tytso@mit.edu> Cc: Daniel Vetter <daniel.vetter@intel.com> Cc: Mani Nikula <jani.nikula@linux.intel.com> Cc: Dave Airlie <airlied@gmail.com> Signed-off-by:
-
git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsiLinus Torvalds authored
Pull SCSI fixes from James Bottomley: "A set of three fixes for the ipr driver and one fairly major one for memory leaks in the mq path of SCSI" * tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi: scsi: fix memory leak with scsi-mq ipr: Fix invalid array indexing for HRRQ ipr: Fix incorrect trace indexing ipr: Fix locking for unit attention handling
-
