From: Mingming Cao <cmm@us.ibm.com>

Basically, freeary() is called with the spinlock for that semaphore set
hold.  But after the semaphore set is removed from the ID array by
calling sem_rmid(), there is no lock to protect the waiting queue for
that semaphore set.  So, if a waiter is woken up by a signal (not by the
wakeup from freeary()), it will check the q->status and q->prev fields.
At that moment, freeary() may not have a chance to update those fields
yet.

static void freeary (int id)
{
	.......
        sma = sem_rmid(id);

	......
        /* Wake up all pending processes and let them fail with EIDRM.*/
        for (q = sma->sem_pending; q; q = q->next) {
                q->status = -EIDRM;
                q->prev = NULL;
                wake_up_process(q->sleeper); /* doesn't sleep */
        }
        sem_unlock(sma);
	......
}

So I propose move sem_rmid() after the loop of waking up every waiters.
That could gurantee that when the waiters are woke up, the updates for
q->status and q->prev have already done.  Similar thing in message queue
case.  The patch is attached below. Comments are very welcomed.

I have tested this patch on 2.5.68 kernel with LTP tests, seems fine to
me. Paul, could you test this on DOTS test again? Thanks!