• Vlastimil Babka's avatar
    mm, vmscan: prevent kswapd livelock due to pfmemalloc-throttled process being killed · 9e5e3661
    Vlastimil Babka authored
    Charles Shirron and Paul Cassella from Cray Inc have reported kswapd
    stuck in a busy loop with nothing left to balance, but
    kswapd_try_to_sleep() failing to sleep.  Their analysis found the cause
    to be a combination of several factors:
    
    1. A process is waiting in throttle_direct_reclaim() on pgdat->pfmemalloc_wait
    
    2. The process has been killed (by OOM in this case), but has not yet been
       scheduled to remove itself from the waitqueue and die.
    
    3. kswapd checks for throttled processes in prepare_kswapd_sleep():
    
            if (waitqueue_active(&pgdat->pfmemalloc_wait)) {
                    wake_up(&pgdat->pfmemalloc_wait);
    		return false; // kswapd will not go to sleep
    	}
    
       However, for a process that was already killed, wake_up() does not remove
       the process from the waitqueue, since try_to_wake_up() checks its state
       first and returns false when the process is no longer waiting.
    
    4. kswapd is running on the same CPU as the only CPU that the process is
       allowed to run on (through cpus_allowed, or possibly single-cpu system).
    
    5. CONFIG_PREEMPT_NONE=y kernel is used. If there's nothing to balance, kswapd
       encounters no voluntary preemption points and repeatedly fails
       prepare_kswapd_sleep(), blocking the process from running and removing
       itself from the waitqueue, which would let kswapd sleep.
    
    So, the source of the problem is that we prevent kswapd from going to
    sleep until there are processes waiting on the pfmemalloc_wait queue,
    and a process waiting on a queue is guaranteed to be removed from the
    queue only when it gets scheduled.  This was done to make sure that no
    process is left sleeping on pfmemalloc_wait when kswapd itself goes to
    sleep.
    
    However, it isn't necessary to postpone kswapd sleep until the
    pfmemalloc_wait queue actually empties.  To prevent processes from being
    left sleeping, it's actually enough to guarantee that all processes
    waiting on pfmemalloc_wait queue have been woken up by the time we put
    kswapd to sleep.
    
    This patch therefore fixes this issue by substituting 'wake_up' with
    'wake_up_all' and removing 'return false' in the code snippet from
    prepare_kswapd_sleep() above.  Note that if any process puts itself in
    the queue after this waitqueue_active() check, or after the wake up
    itself, it means that the process will also wake up kswapd - and since
    we are under prepare_to_wait(), the wake up won't be missed.  Also we
    update the comment prepare_kswapd_sleep() to hopefully more clearly
    describe the races it is preventing.
    
    Fixes: 5515061d ("mm: throttle direct reclaimers if PF_MEMALLOC reserves are low and swap is backed by network storage")
    Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Signed-off-by: default avatarVladimir Davydov <vdavydov@parallels.com>
    Cc: Mel Gorman <mgorman@suse.de>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Acked-by: default avatarMichal Hocko <mhocko@suse.cz>
    Acked-by: default avatarRik van Riel <riel@redhat.com>
    Cc: <stable@vger.kernel.org>	[3.6+]
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    9e5e3661
vmscan.c 108 KB