[PATCH] mempool wakeup fix

When the mempool is empty, tasks wait on the waitqueue in "exclusive
mode".  So one task is woken for each returned element.

But if the number of tasks which are waiting exceeds the mempool's
specified size (min_nr), mempool_free() ends up deciding that as the
pool is fully replenished, there cannot possibly be anyone waiting for
more elements.

But with 16384 threads running tiobench, it happens.

We could fix this with a waitqueue_active() test in mempool_free().
But rather than adding that test to this fastpath I changed the wait to
be non-exclusive, and used the prepare_to_wait/finish_wait API, which
will be quite beneficial in this case.

Also, convert the schedule() in mempool_alloc() to an io_schedule(), so
this sleep time is accounted as "IO wait".  Which is a bit approximate
- we don't _know_ that the caller is really waiting for IO completion.
But for most current users of mempools, io_schedule() is more accurate
than schedule() here.

mm/mempool.c

@@ -12,6 +12,7 @@
 #include <linux/slab.h>
 #include <linux/module.h>
 #include <linux/mempool.h>
+#include <linux/blkdev.h>
 #include <linux/writeback.h>
 
 static void add_element(mempool_t *pool, void *element)
@@ -184,8 +185,7 @@ void * mempool_alloc(mempool_t *pool, int gfp_mask)
 {
 	void *element;
 	unsigned long flags;
-	int curr_nr;
-	DECLARE_WAITQUEUE(wait, current);
+	DEFINE_WAIT(wait);
 	int gfp_nowait = gfp_mask & ~(__GFP_WAIT | __GFP_IO);
 	int pf_flags = current->flags;
 
@@ -226,18 +226,10 @@ void * mempool_alloc(mempool_t *pool, int gfp_mask)
 
 	blk_run_queues();
 
-	add_wait_queue_exclusive(&pool->wait, &wait);
-	set_task_state(current, TASK_UNINTERRUPTIBLE);
-
-	spin_lock_irqsave(&pool->lock, flags);
-	curr_nr = pool->curr_nr;
-	spin_unlock_irqrestore(&pool->lock, flags);
-
-	if (!curr_nr)
-		schedule();
-
-	current->state = TASK_RUNNING;
-	remove_wait_queue(&pool->wait, &wait);
+	prepare_to_wait(&pool->wait, &wait, TASK_UNINTERRUPTIBLE);
+	if (!pool->curr_nr)
+		io_schedule();
+	finish_wait(&pool->wait, &wait);
 
 	goto repeat_alloc;
 }