UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge branch 'futexes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip 

* 'futexes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip:
  futex: fix restart in wait_requeue_pi
  futex: fix restart for early wakeup in futex_wait_requeue_pi()
  futex: cleanup error exit
  futex: remove the wait queue
  futex: add requeue-pi documentation
  futex: remove FUTEX_REQUEUE_PI (non CMP)
  futex: fix futex_wait_setup key handling
  sparc64: extend TI_RESTART_BLOCK space by 8 bytes
  futex: fixup unlocked requeue pi case
  futex: add requeue_pi functionality
  futex: split out futex value validation code
  futex: distangle futex_requeue()
  futex: add FUTEX_HAS_TIMEOUT flag to restart.futex.flags
  rt_mutex: add proxy lock routines
  futex: split out fixup owner logic from futex_lock_pi()
  futex: split out atomic logic from futex_lock_pi()
  futex: add helper to find the top prio waiter of a futex
  futex: separate futex_wait_queue_me() logic from futex_wait()
This commit is contained in:
Linus Torvalds 2009-06-10 16:16:48 -07:00
parent be15f9d63b 2070887fde
commit 75063600fd
7 changed files with 1254 additions and 384 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

131
Documentation/futex-requeue-pi.txt Normal file
View File

@ -0,0 +1,131 @@
Futex Requeue PI
----------------
Requeueing of tasks from a non-PI futex to a PI futex requires
special handling in order to ensure the underlying rt_mutex is never
left without an owner if it has waiters; doing so would break the PI
boosting logic [see rt-mutex-desgin.txt] For the purposes of
brevity, this action will be referred to as "requeue_pi" throughout
this document. Priority inheritance is abbreviated throughout as
"PI".
Motivation
----------
Without requeue_pi, the glibc implementation of
pthread_cond_broadcast() must resort to waking all the tasks waiting
on a pthread_condvar and letting them try to sort out which task
gets to run first in classic thundering-herd formation. An ideal
implementation would wake the highest-priority waiter, and leave the
rest to the natural wakeup inherent in unlocking the mutex
associated with the condvar.
Consider the simplified glibc calls:
/* caller must lock mutex */
pthread_cond_wait(cond, mutex)
{
lock(cond->__data.__lock);
unlock(mutex);
do {
unlock(cond->__data.__lock);
futex_wait(cond->__data.__futex);
lock(cond->__data.__lock);
} while(...)
unlock(cond->__data.__lock);
lock(mutex);
}
pthread_cond_broadcast(cond)
{
lock(cond->__data.__lock);
unlock(cond->__data.__lock);
futex_requeue(cond->data.__futex, cond->mutex);
}
Once pthread_cond_broadcast() requeues the tasks, the cond->mutex
has waiters. Note that pthread_cond_wait() attempts to lock the
mutex only after it has returned to user space. This will leave the
underlying rt_mutex with waiters, and no owner, breaking the
previously mentioned PI-boosting algorithms.
In order to support PI-aware pthread_condvar's, the kernel needs to
be able to requeue tasks to PI futexes. This support implies that
upon a successful futex_wait system call, the caller would return to
user space already holding the PI futex. The glibc implementation
would be modified as follows:
/* caller must lock mutex */
pthread_cond_wait_pi(cond, mutex)
{
lock(cond->__data.__lock);
unlock(mutex);
do {
unlock(cond->__data.__lock);
futex_wait_requeue_pi(cond->__data.__futex);
lock(cond->__data.__lock);
} while(...)
unlock(cond->__data.__lock);
/* the kernel acquired the the mutex for us */
}
pthread_cond_broadcast_pi(cond)
{
lock(cond->__data.__lock);
unlock(cond->__data.__lock);
futex_requeue_pi(cond->data.__futex, cond->mutex);
}
The actual glibc implementation will likely test for PI and make the
necessary changes inside the existing calls rather than creating new
calls for the PI cases. Similar changes are needed for
pthread_cond_timedwait() and pthread_cond_signal().
Implementation
--------------
In order to ensure the rt_mutex has an owner if it has waiters, it
is necessary for both the requeue code, as well as the waiting code,
to be able to acquire the rt_mutex before returning to user space.
The requeue code cannot simply wake the waiter and leave it to
acquire the rt_mutex as it would open a race window between the
requeue call returning to user space and the waiter waking and
starting to run. This is especially true in the uncontended case.
The solution involves two new rt_mutex helper routines,
rt_mutex_start_proxy_lock() and rt_mutex_finish_proxy_lock(), which
allow the requeue code to acquire an uncontended rt_mutex on behalf
of the waiter and to enqueue the waiter on a contended rt_mutex.
Two new system calls provide the kernel<->user interface to
requeue_pi: FUTEX_WAIT_REQUEUE_PI and FUTEX_REQUEUE_CMP_PI.
FUTEX_WAIT_REQUEUE_PI is called by the waiter (pthread_cond_wait()
and pthread_cond_timedwait()) to block on the initial futex and wait
to be requeued to a PI-aware futex. The implementation is the
result of a high-speed collision between futex_wait() and
futex_lock_pi(), with some extra logic to check for the additional
wake-up scenarios.
FUTEX_REQUEUE_CMP_PI is called by the waker
(pthread_cond_broadcast() and pthread_cond_signal()) to requeue and
possibly wake the waiting tasks. Internally, this system call is
still handled by futex_requeue (by passing requeue_pi=1). Before
requeueing, futex_requeue() attempts to acquire the requeue target
PI futex on behalf of the top waiter. If it can, this waiter is
woken. futex_requeue() then proceeds to requeue the remaining
nr_wake+nr_requeue tasks to the PI futex, calling
rt_mutex_start_proxy_lock() prior to each requeue to prepare the
task as a waiter on the underlying rt_mutex. It is possible that
the lock can be acquired at this stage as well, if so, the next
waiter is woken to finish the acquisition of the lock.
FUTEX_REQUEUE_PI accepts nr_wake and nr_requeue as arguments, but
their sum is all that really matters. futex_requeue() will wake or
requeue up to nr_wake + nr_requeue tasks. It will wake only as many
tasks as it can acquire the lock for, which in the majority of cases
should be 0 as good programming practice dictates that the caller of
either pthread_cond_broadcast() or pthread_cond_signal() acquire the
mutex prior to making the call. FUTEX_REQUEUE_PI requires that
nr_wake=1. nr_requeue should be INT_MAX for broadcast and 0 for
signal.

4
arch/sparc/include/asm/thread_info_64.h
View File

@ -102,8 +102,8 @@ struct thread_info {
#define TI_KERN_CNTD1 0x00000488 #define TI_KERN_CNTD1 0x00000488
#define TI_PCR 0x00000490 #define TI_PCR 0x00000490
#define TI_RESTART_BLOCK 0x00000498 #define TI_RESTART_BLOCK 0x00000498
#define TI_KUNA_REGS 0x000004c0 #define TI_KUNA_REGS 0x000004c8
#define TI_KUNA_INSN 0x000004c8 #define TI_KUNA_INSN 0x000004d0
#define TI_FPREGS 0x00000500 #define TI_FPREGS 0x00000500
/* We embed this in the uppermost byte of thread_info->flags */ /* We embed this in the uppermost byte of thread_info->flags */

6
include/linux/futex.h
View File

@ -23,6 +23,8 @@ union ktime;
#define FUTEX_TRYLOCK_PI 8 #define FUTEX_TRYLOCK_PI 8
#define FUTEX_WAIT_BITSET 9 #define FUTEX_WAIT_BITSET 9
#define FUTEX_WAKE_BITSET 10 #define FUTEX_WAKE_BITSET 10
#define FUTEX_WAIT_REQUEUE_PI 11
#define FUTEX_CMP_REQUEUE_PI 12
#define FUTEX_PRIVATE_FLAG 128 #define FUTEX_PRIVATE_FLAG 128
#define FUTEX_CLOCK_REALTIME 256 #define FUTEX_CLOCK_REALTIME 256
@ -38,6 +40,10 @@ union ktime;
#define FUTEX_TRYLOCK_PI_PRIVATE (FUTEX_TRYLOCK_PI | FUTEX_PRIVATE_FLAG) #define FUTEX_TRYLOCK_PI_PRIVATE (FUTEX_TRYLOCK_PI | FUTEX_PRIVATE_FLAG)
#define FUTEX_WAIT_BITSET_PRIVATE (FUTEX_WAIT_BITS | FUTEX_PRIVATE_FLAG) #define FUTEX_WAIT_BITSET_PRIVATE (FUTEX_WAIT_BITS | FUTEX_PRIVATE_FLAG)
#define FUTEX_WAKE_BITSET_PRIVATE (FUTEX_WAKE_BITS | FUTEX_PRIVATE_FLAG) #define FUTEX_WAKE_BITSET_PRIVATE (FUTEX_WAKE_BITS | FUTEX_PRIVATE_FLAG)
#define FUTEX_WAIT_REQUEUE_PI_PRIVATE (FUTEX_WAIT_REQUEUE_PI | \
FUTEX_PRIVATE_FLAG)
#define FUTEX_CMP_REQUEUE_PI_PRIVATE (FUTEX_CMP_REQUEUE_PI | \
FUTEX_PRIVATE_FLAG)
/* /*
* Support for robust futexes: the kernel cleans up held futexes at * Support for robust futexes: the kernel cleans up held futexes at

3
include/linux/thread_info.h
View File

@ -21,13 +21,14 @@ struct restart_block {
struct { struct {
unsigned long arg0, arg1, arg2, arg3; unsigned long arg0, arg1, arg2, arg3;
}; };
/* For futex_wait */ /* For futex_wait and futex_wait_requeue_pi */
struct { struct {
u32 *uaddr; u32 *uaddr;
u32 val; u32 val;
u32 flags; u32 flags;
u32 bitset; u32 bitset;
u64 time; u64 time;
u32 *uaddr2;
} futex; } futex;
/* For nanosleep */ /* For nanosleep */
struct { struct {

1198
kernel/futex.c
View File

File diff suppressed because it is too large Load Diff

288
kernel/rtmutex.c
View File

@ -300,7 +300,8 @@ static int rt_mutex_adjust_prio_chain(struct task_struct *task,
* assigned pending owner [which might not have taken the * assigned pending owner [which might not have taken the
* lock yet]: * lock yet]:
*/ */
static inline int try_to_steal_lock(struct rt_mutex *lock) static inline int try_to_steal_lock(struct rt_mutex *lock,
struct task_struct *task)
{ {
struct task_struct *pendowner = rt_mutex_owner(lock); struct task_struct *pendowner = rt_mutex_owner(lock);
struct rt_mutex_waiter *next; struct rt_mutex_waiter *next;
@ -309,11 +310,11 @@ static inline int try_to_steal_lock(struct rt_mutex *lock)
if (!rt_mutex_owner_pending(lock)) if (!rt_mutex_owner_pending(lock))
return 0; return 0;
if (pendowner == current) if (pendowner == task)
return 1; return 1;
spin_lock_irqsave(&pendowner->pi_lock, flags); spin_lock_irqsave(&pendowner->pi_lock, flags);
if (current->prio >= pendowner->prio) { if (task->prio >= pendowner->prio) {
spin_unlock_irqrestore(&pendowner->pi_lock, flags); spin_unlock_irqrestore(&pendowner->pi_lock, flags);
return 0; return 0;
} }
@ -338,21 +339,21 @@ static inline int try_to_steal_lock(struct rt_mutex *lock)
* We are going to steal the lock and a waiter was * We are going to steal the lock and a waiter was
* enqueued on the pending owners pi_waiters queue. So * enqueued on the pending owners pi_waiters queue. So
* we have to enqueue this waiter into * we have to enqueue this waiter into
* current->pi_waiters list. This covers the case, * task->pi_waiters list. This covers the case,
* where current is boosted because it holds another * where task is boosted because it holds another
* lock and gets unboosted because the booster is * lock and gets unboosted because the booster is
* interrupted, so we would delay a waiter with higher * interrupted, so we would delay a waiter with higher
* priority as current->normal_prio. * priority as task->normal_prio.
* *
* Note: in the rare case of a SCHED_OTHER task changing * Note: in the rare case of a SCHED_OTHER task changing
* its priority and thus stealing the lock, next->task * its priority and thus stealing the lock, next->task
* might be current: * might be task:
*/ */
if (likely(next->task != current)) { if (likely(next->task != task)) {
spin_lock_irqsave(&current->pi_lock, flags); spin_lock_irqsave(&task->pi_lock, flags);
plist_add(&next->pi_list_entry, &current->pi_waiters); plist_add(&next->pi_list_entry, &task->pi_waiters);
__rt_mutex_adjust_prio(current); __rt_mutex_adjust_prio(task);
spin_unlock_irqrestore(&current->pi_lock, flags); spin_unlock_irqrestore(&task->pi_lock, flags);
} }
return 1; return 1;
} }
@ -389,7 +390,7 @@ static int try_to_take_rt_mutex(struct rt_mutex *lock)
*/ */
mark_rt_mutex_waiters(lock); mark_rt_mutex_waiters(lock);
if (rt_mutex_owner(lock) && !try_to_steal_lock(lock)) if (rt_mutex_owner(lock) && !try_to_steal_lock(lock, current))
return 0; return 0;
/* We got the lock. */ /* We got the lock. */
@ -411,6 +412,7 @@ static int try_to_take_rt_mutex(struct rt_mutex *lock)
*/ */
static int task_blocks_on_rt_mutex(struct rt_mutex *lock, static int task_blocks_on_rt_mutex(struct rt_mutex *lock,
struct rt_mutex_waiter *waiter, struct rt_mutex_waiter *waiter,
struct task_struct *task,
int detect_deadlock) int detect_deadlock)
{ {
struct task_struct *owner = rt_mutex_owner(lock); struct task_struct *owner = rt_mutex_owner(lock);
@ -418,21 +420,21 @@ static int task_blocks_on_rt_mutex(struct rt_mutex *lock,
unsigned long flags; unsigned long flags;
int chain_walk = 0, res; int chain_walk = 0, res;
spin_lock_irqsave(&current->pi_lock, flags); spin_lock_irqsave(&task->pi_lock, flags);
__rt_mutex_adjust_prio(current); __rt_mutex_adjust_prio(task);
waiter->task = current; waiter->task = task;
waiter->lock = lock; waiter->lock = lock;
plist_node_init(&waiter->list_entry, current->prio); plist_node_init(&waiter->list_entry, task->prio);
plist_node_init(&waiter->pi_list_entry, current->prio); plist_node_init(&waiter->pi_list_entry, task->prio);
/* Get the top priority waiter on the lock */ /* Get the top priority waiter on the lock */
if (rt_mutex_has_waiters(lock)) if (rt_mutex_has_waiters(lock))
top_waiter = rt_mutex_top_waiter(lock); top_waiter = rt_mutex_top_waiter(lock);
plist_add(&waiter->list_entry, &lock->wait_list); plist_add(&waiter->list_entry, &lock->wait_list);
current->pi_blocked_on = waiter; task->pi_blocked_on = waiter;
spin_unlock_irqrestore(&current->pi_lock, flags); spin_unlock_irqrestore(&task->pi_lock, flags);
if (waiter == rt_mutex_top_waiter(lock)) { if (waiter == rt_mutex_top_waiter(lock)) {
spin_lock_irqsave(&owner->pi_lock, flags); spin_lock_irqsave(&owner->pi_lock, flags);
@ -460,7 +462,7 @@ static int task_blocks_on_rt_mutex(struct rt_mutex *lock,
spin_unlock(&lock->wait_lock); spin_unlock(&lock->wait_lock);
res = rt_mutex_adjust_prio_chain(owner, detect_deadlock, lock, waiter, res = rt_mutex_adjust_prio_chain(owner, detect_deadlock, lock, waiter,
current); task);
spin_lock(&lock->wait_lock); spin_lock(&lock->wait_lock);
@ -605,6 +607,85 @@ void rt_mutex_adjust_pi(struct task_struct *task)
rt_mutex_adjust_prio_chain(task, 0, NULL, NULL, task); rt_mutex_adjust_prio_chain(task, 0, NULL, NULL, task);
} }
/**
* __rt_mutex_slowlock() - Perform the wait-wake-try-to-take loop
* @lock: the rt_mutex to take
* @state: the state the task should block in (TASK_INTERRUPTIBLE
* or TASK_UNINTERRUPTIBLE)
* @timeout: the pre-initialized and started timer, or NULL for none
* @waiter: the pre-initialized rt_mutex_waiter
* @detect_deadlock: passed to task_blocks_on_rt_mutex
*
* lock->wait_lock must be held by the caller.
*/
static int __sched
__rt_mutex_slowlock(struct rt_mutex *lock, int state,
struct hrtimer_sleeper *timeout,
struct rt_mutex_waiter *waiter,
int detect_deadlock)
{
int ret = 0;
for (;;) {
/* Try to acquire the lock: */
if (try_to_take_rt_mutex(lock))
break;
/*
* TASK_INTERRUPTIBLE checks for signals and
* timeout. Ignored otherwise.
*/
if (unlikely(state == TASK_INTERRUPTIBLE)) {
/* Signal pending? */
if (signal_pending(current))
ret = -EINTR;
if (timeout && !timeout->task)
ret = -ETIMEDOUT;
if (ret)
break;
}
/*
* waiter->task is NULL the first time we come here and
* when we have been woken up by the previous owner
* but the lock got stolen by a higher prio task.
*/
if (!waiter->task) {
ret = task_blocks_on_rt_mutex(lock, waiter, current,
detect_deadlock);
/*
* If we got woken up by the owner then start loop
* all over without going into schedule to try
* to get the lock now:
*/
if (unlikely(!waiter->task)) {
/*
* Reset the return value. We might
* have returned with -EDEADLK and the
* owner released the lock while we
* were walking the pi chain.
*/
ret = 0;
continue;
}
if (unlikely(ret))
break;
}
spin_unlock(&lock->wait_lock);
debug_rt_mutex_print_deadlock(waiter);
if (waiter->task)
schedule_rt_mutex(lock);
spin_lock(&lock->wait_lock);
set_current_state(state);
}
return ret;
}
/* /*
* Slow path lock function: * Slow path lock function:
*/ */
@ -636,62 +717,8 @@ rt_mutex_slowlock(struct rt_mutex *lock, int state,
timeout->task = NULL; timeout->task = NULL;
} }
for (;;) { ret = __rt_mutex_slowlock(lock, state, timeout, &waiter,
/* Try to acquire the lock: */ detect_deadlock);
if (try_to_take_rt_mutex(lock))
break;
/*
* TASK_INTERRUPTIBLE checks for signals and
* timeout. Ignored otherwise.
*/
if (unlikely(state == TASK_INTERRUPTIBLE)) {
/* Signal pending? */
if (signal_pending(current))
ret = -EINTR;
if (timeout && !timeout->task)
ret = -ETIMEDOUT;
if (ret)
break;
}
/*
* waiter.task is NULL the first time we come here and
* when we have been woken up by the previous owner
* but the lock got stolen by a higher prio task.
*/
if (!waiter.task) {
ret = task_blocks_on_rt_mutex(lock, &waiter,
detect_deadlock);
/*
* If we got woken up by the owner then start loop
* all over without going into schedule to try
* to get the lock now:
*/
if (unlikely(!waiter.task)) {
/*
* Reset the return value. We might
* have returned with -EDEADLK and the
* owner released the lock while we
* were walking the pi chain.
*/
ret = 0;
continue;
}
if (unlikely(ret))
break;
}
spin_unlock(&lock->wait_lock);
debug_rt_mutex_print_deadlock(&waiter);
if (waiter.task)
schedule_rt_mutex(lock);
spin_lock(&lock->wait_lock);
set_current_state(state);
}
set_current_state(TASK_RUNNING); set_current_state(TASK_RUNNING);
@ -985,6 +1012,59 @@ void rt_mutex_proxy_unlock(struct rt_mutex *lock,
rt_mutex_deadlock_account_unlock(proxy_owner); rt_mutex_deadlock_account_unlock(proxy_owner);
} }
/**
* rt_mutex_start_proxy_lock() - Start lock acquisition for another task
* @lock: the rt_mutex to take
* @waiter: the pre-initialized rt_mutex_waiter
* @task: the task to prepare
* @detect_deadlock: perform deadlock detection (1) or not (0)
*
* Returns:
* 0 - task blocked on lock
* 1 - acquired the lock for task, caller should wake it up
* <0 - error
*
* Special API call for FUTEX_REQUEUE_PI support.
*/
int rt_mutex_start_proxy_lock(struct rt_mutex *lock,
struct rt_mutex_waiter *waiter,
struct task_struct *task, int detect_deadlock)
{
int ret;
spin_lock(&lock->wait_lock);
mark_rt_mutex_waiters(lock);
if (!rt_mutex_owner(lock) || try_to_steal_lock(lock, task)) {
/* We got the lock for task. */
debug_rt_mutex_lock(lock);
rt_mutex_set_owner(lock, task, 0);
rt_mutex_deadlock_account_lock(lock, task);
return 1;
}
ret = task_blocks_on_rt_mutex(lock, waiter, task, detect_deadlock);
if (ret && !waiter->task) {
/*
* Reset the return value. We might have
* returned with -EDEADLK and the owner
* released the lock while we were walking the
* pi chain. Let the waiter sort it out.
*/
ret = 0;
}
spin_unlock(&lock->wait_lock);
debug_rt_mutex_print_deadlock(waiter);
return ret;
}
/** /**
* rt_mutex_next_owner - return the next owner of the lock * rt_mutex_next_owner - return the next owner of the lock
* *
@ -1004,3 +1084,57 @@ struct task_struct *rt_mutex_next_owner(struct rt_mutex *lock)
return rt_mutex_top_waiter(lock)->task; return rt_mutex_top_waiter(lock)->task;
} }
/**
* rt_mutex_finish_proxy_lock() - Complete lock acquisition
* @lock: the rt_mutex we were woken on
* @to: the timeout, null if none. hrtimer should already have
* been started.
* @waiter: the pre-initialized rt_mutex_waiter
* @detect_deadlock: perform deadlock detection (1) or not (0)
*
* Complete the lock acquisition started our behalf by another thread.
*
* Returns:
* 0 - success
* <0 - error, one of -EINTR, -ETIMEDOUT, or -EDEADLK
*
* Special API call for PI-futex requeue support
*/
int rt_mutex_finish_proxy_lock(struct rt_mutex *lock,
struct hrtimer_sleeper *to,
struct rt_mutex_waiter *waiter,
int detect_deadlock)
{
int ret;
spin_lock(&lock->wait_lock);
set_current_state(TASK_INTERRUPTIBLE);
ret = __rt_mutex_slowlock(lock, TASK_INTERRUPTIBLE, to, waiter,
detect_deadlock);
set_current_state(TASK_RUNNING);
if (unlikely(waiter->task))
remove_waiter(lock, waiter);
/*
* try_to_take_rt_mutex() sets the waiter bit unconditionally. We might
* have to fix that up.
*/
fixup_rt_mutex_waiters(lock);
spin_unlock(&lock->wait_lock);
/*
* Readjust priority, when we did not get the lock. We might have been
* the pending owner and boosted. Since we did not take the lock, the
* PI boost has to go.
*/
if (unlikely(ret))
rt_mutex_adjust_prio(current);
return ret;
}

8
kernel/rtmutex_common.h
View File

@ -120,6 +120,14 @@ extern void rt_mutex_init_proxy_locked(struct rt_mutex *lock,
struct task_struct *proxy_owner); struct task_struct *proxy_owner);
extern void rt_mutex_proxy_unlock(struct rt_mutex *lock, extern void rt_mutex_proxy_unlock(struct rt_mutex *lock,
struct task_struct *proxy_owner); struct task_struct *proxy_owner);
extern int rt_mutex_start_proxy_lock(struct rt_mutex *lock,
struct rt_mutex_waiter *waiter,
struct task_struct *task,
int detect_deadlock);
extern int rt_mutex_finish_proxy_lock(struct rt_mutex *lock,
struct hrtimer_sleeper *to,
struct rt_mutex_waiter *waiter,
int detect_deadlock);
#ifdef CONFIG_DEBUG_RT_MUTEXES #ifdef CONFIG_DEBUG_RT_MUTEXES
# include "rtmutex-debug.h" # include "rtmutex-debug.h"