- 17 Aug, 2021 40 commits
-
-
Thomas Gleixner authored
Move the futex key match out of handle_early_requeue_pi_wakeup() which allows to simplify that function. The upcoming state machine for requeue_pi() will make that go away. Signed-off-by:
Thomas Gleixner <tglx@linutronix.de> Signed-off-by:
Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by:
Ingo Molnar <mingo@kernel.org> Link: https://lore.kernel.org/r/20210815211305.638938670@linutronix.de
-
Thomas Gleixner authored
No point in allocating memory when the input parameters are bogus. Validate all parameters before proceeding. Suggested-by:
Davidlohr Bueso <dave@stgolabs.net> Signed-off-by:
-
Thomas Gleixner authored
The comment about the restriction of the number of waiters to wake for the REQUEUE_PI case is confusing at best. Rewrite it. Signed-off-by:
-
Thomas Gleixner authored
No point in taking two more 'requeue_pi' conditionals just to get to the requeue. Same for the requeue_pi case just the other way round. No functional change. Signed-off-by:
-
Thomas Gleixner authored
The accounting is wrong when either the PI sanity check or the requeue PI operation fails. Adjust it in the failure path. Will be simplified in the next step. Signed-off-by:
-
Thomas Gleixner authored
For requeue PI it's required to establish PI state for the PI futex to which waiters are requeued. This either acquires the user space futex on behalf of the top most waiter on the inner 'waitqueue' futex, or attaches to the PI state of an existing waiter, or creates on attached to the owner of the futex. This code can retry in case of failure, but retry can never happen when the pi state was successfully created. The condition to run this code is: (task_count - nr_wake) < nr_requeue which is always true because: task_count = 0 nr_wake = 1 nr_requeue >= 0 Remove it completely. Signed-off-by:
-
Thomas Gleixner authored
When requeuing to a PI futex, then the requeue code tries to trylock the PI futex on behalf of the topmost waiter on the inner 'waitqueue' futex. If that succeeds, then PI state has to be allocated in order to requeue further waiters to the PI futex. The comment and the code are confusing, as the PI state allocation uses lookup_pi_state(), which either attaches to an existing waiter or to the owner. As the PI futex was just acquired, there cannot be a waiter on the PI futex because the hash bucket lock is held. Clarify the comment and use attach_to_pi_owner() directly. As the task on which behalf the PI futex has been acquired is guaranteed to be alive and not exiting, this call must succeed. Add a WARN_ON() in case that fails. Signed-off-by:
-
Thomas Gleixner authored
The futex key reference mechanism is long gone. Clean up the stale comments which still mention it. Signed-off-by:
-
Thomas Gleixner authored
The loop in futex_requeue() has a sanity check for the waiter, which is missing in futex_proxy_trylock_atomic(). In theory the key2 check is sufficient, but futexes are cursed so add it for completeness and paranoia sake. Signed-off-by:
-
Sebastian Andrzej Siewior authored
The inner parts of certain locks (mutex, rwlocks) changed due to a rework for RT and non RT code. Most users remain unaffected, but those who fiddle around in the inner parts need to be updated. Match the struct names to the new layout. Signed-off-by:
Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by:
-
Thomas Gleixner authored
Add the necessary defines, helpers and API functions for replacing struct mutex on a PREEMPT_RT enabled kernel with an rtmutex based variant. No functional change when CONFIG_PREEMPT_RT=n Signed-off-by:
-
Peter Zijlstra authored
Add the actual ww_mutex API functions which replace the mutex based variant on RT enabled kernels. Signed-off-by:
-
Peter Zijlstra authored
Add a ww acquire context pointer to the waiter and various functions and add the ww_mutex related invocations to the proper spots in the locking code, similar to the mutex based variant. Signed-off-by:
-
Peter Zijlstra authored
Provide the defines for RT mutex based ww_mutexes and fix up the debug logic so it's either enabled by DEBUG_MUTEXES or DEBUG_RT_MUTEXES on RT kernels. Signed-off-by:
-
Peter Zijlstra authored
RT mutex based ww_mutexes cannot order based on timestamps. They have to order based on priority. Add the necessary decision logic. Signed-off-by:
-
Peter Zijlstra authored
Provide the type defines and the helper inlines for rtmutex based ww_mutexes. Signed-off-by:
-
Thomas Gleixner authored
Accessing the internal wait_lock of mutex and rtmutex is slightly different. Provide helper functions for that. Signed-off-by:
-
Peter Zijlstra authored
Some ww_mutex helper functions use pointers for the underlying mutex and mutex_waiter. The upcoming rtmutex based implementation needs to share these functions. Add and use defines for the types and replace the direct types in the affected functions. Signed-off-by:
-
Peter Zijlstra authored
Move the mutex related access from various ww_mutex functions into helper functions so they can be substituted for rtmutex based ww_mutex later. Signed-off-by:
-
Peter Zijlstra authored
The upcoming rtmutex based ww_mutex needs a different handling for enqueueing a waiter. Split it out into a helper function. Signed-off-by:
-
Peter Zijlstra authored
Split out the waiter iteration functions so they can be substituted for a rtmutex based ww_mutex later. Signed-off-by:
-
Peter Zijlstra authored
None of these functions will be on the stack when blocking in schedule(), hence __sched is not needed. Signed-off-by:
-
Peter Zijlstra (Intel) authored
Split the W/W mutex helper functions out into a separate header file, so they can be shared with a rtmutex based variant later. Signed-off-by:
-
Peter Zijlstra (Intel) authored
Split the ww related part out into a helper function so it can be reused for a rtmutex based ww_mutex implementation. [ mingo: Fixed bisection failure. ] Signed-off-by:
-
Peter Zijlstra authored
Signed-off-by:
-
Peter Zijlstra authored
No functional change. Signed-off-by:
-
Thomas Gleixner authored
The wait_lock of mutex is really a low level lock. Convert it to a raw_spinlock like the wait_lock of rtmutex. [ mingo: backmerged the test_lockup.c build fix by bigeasy. ] Co-developed-by:
-
Thomas Gleixner authored
Move the ww_mutex definitions into the ww_mutex specific header where they belong. Preparatory change to allow compiling ww_mutexes standalone. Signed-off-by:
-
Thomas Gleixner authored
locking/mutex: Move the 'struct mutex_waiter' definition from <linux/mutex.h> to the internal header Move the mutex waiter declaration from the public <linux/mutex.h> header to the internal kernel/locking/mutex.h header. There is no reason to expose it outside of the core code. Signed-off-by:
-
Thomas Gleixner authored
Having two header files which contain just the non-debug and debug variants is mostly waste of disc space and has no real value. Stick the debug variants into the common mutex.h file as counterpart to the stubs for the non-debug case. Signed-off-by:
-
Peter Zijlstra authored
Ensure all !RT tasks have the same prio such that they end up in FIFO order and aren't split up according to nice level. The reason why nice levels were taken into account so far is historical. In the early days of the rtmutex code it was done to give the PI boosting and deboosting a larger coverage. Signed-off-by:
-
Thomas Gleixner authored
Similar to rw_semaphores, on RT the rwlock substitution is not writer fair, because it's not feasible to have a writer inherit its priority to multiple readers. Readers blocked on a writer follow the normal rules of priority inheritance. Like RT spinlocks, RT rwlocks are state preserving across the slow lock operations (contended case). Signed-off-by:
-
Thomas Gleixner authored
Provide the actual locking functions which make use of the general and spinlock specific rtmutex code. Signed-off-by:
-
Thomas Gleixner authored
A simplified version of the rtmutex slowlock function, which neither handles signals nor timeouts, and is careful about preserving the state of the blocked task across the lock operation. Signed-off-by:
-
Thomas Gleixner authored
Provide the necessary wrappers around the actual rtmutex based spinlock implementation. Signed-off-by:
-
Thomas Gleixner authored
RT replaces spinlocks with a simple wrapper around an rtmutex, which turns spinlocks on RT into 'sleeping' spinlocks. The actual implementation of the spinlock API differs from a regular rtmutex, as it does neither handle timeouts nor signals and it is state preserving across the lock operation. Signed-off-by:
-
Sebastian Andrzej Siewior authored
We have the following header dependency problem on RT: - <linux/rtmutex.h> needs the definition of 'struct rb_root_cached'. - <linux/rbtree.h> includes <linux/kernel.h>, which includes <linux/spinlock.h> That works nicely for non-RT enabled kernels, but on RT enabled kernels spinlocks are based on rtmutexes, which creates another circular header dependency as <linux/spinlocks.h> will require <linux/rtmutex.h>. Include <linux/rbtree_types.h> instead. Signed-off-by:
-
Sebastian Andrzej Siewior authored
So we have this header dependency problem on RT: - <linux/rtmutex.h> needs the definition of 'struct rb_root_cached'. - <linux/rbtree.h> includes <linux/kernel.h>, which includes <linux/spinlock.h>. That works nicely for non-RT enabled kernels, but on RT enabled kernels spinlocks are based on rtmutexes, which creates another circular header dependency, as <linux/spinlocks.h> will require <linux/rtmutex.h>. Split out the type definitions and move them into their own header file so the rtmutex header can include just those. Signed-off-by:
-
Sebastian Andrzej Siewior authored
The inclusion of printk.h leads to a circular dependency if spinlock_t is based on rtmutexes on RT enabled kernels. Include only atomic.h (xchg()) and cache.h (__read_mostly) which is all what debug_locks.h requires. Signed-off-by:
-
Sebastian Andrzej Siewior authored
rtmutex only needs raw_spinlock_t, but it includes spinlock_types.h, which is not a problem on an non RT enabled kernel. RT kernels substitute regular spinlocks with 'sleeping' spinlocks, which are based on rtmutexes, and therefore must be able to include rtmutex.h. Include <linux/spinlock_types_raw.h> instead. Signed-off-by:
-
