-
Andrew Morton authored
When the mempool is empty, tasks wait on the waitqueue in "exclusive mode". So one task is woken for each returned element. But if the number of tasks which are waiting exceeds the mempool's specified size (min_nr), mempool_free() ends up deciding that as the pool is fully replenished, there cannot possibly be anyone waiting for more elements. But with 16384 threads running tiobench, it happens. We could fix this with a waitqueue_active() test in mempool_free(). But rather than adding that test to this fastpath I changed the wait to be non-exclusive, and used the prepare_to_wait/finish_wait API, which will be quite beneficial in this case. Also, convert the schedule() in mempool_alloc() to an io_schedule(), so this sleep time is accounted as "IO wait". Which is a bit approximate - we don't _know_ that the caller is really waiting for IO completion. But for most current users of mempools, io_schedule() is more accurate than schedule() here.
fe66ad33