• Joonsoo Kim's avatar
    mm/page_alloc: add freepage on isolate pageblock to correct buddy list · 51bb1a40
    Joonsoo Kim authored
    In free_pcppages_bulk(), we use cached migratetype of freepage to
    determine type of buddy list where freepage will be added.  This
    information is stored when freepage is added to pcp list, so if
    isolation of pageblock of this freepage begins after storing, this
    cached information could be stale.  In other words, it has original
    migratetype rather than MIGRATE_ISOLATE.
    
    There are two problems caused by this stale information.
    
    One is that we can't keep these freepages from being allocated.
    Although this pageblock is isolated, freepage will be added to normal
    buddy list so that it could be allocated without any restriction.  And
    the other problem is incorrect freepage accounting.  Freepages on
    isolate pageblock should not be counted for number of freepage.
    
    Following is the code snippet in free_pcppages_bulk().
    
        /* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */
        __free_one_page(page, page_to_pfn(page), zone, 0, mt);
        trace_mm_page_pcpu_drain(page, 0, mt);
        if (likely(!is_migrate_isolate_page(page))) {
            __mod_zone_page_state(zone, NR_FREE_PAGES, 1);
            if (is_migrate_cma(mt))
                __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, 1);
        }
    
    As you can see above snippet, current code already handle second
    problem, incorrect freepage accounting, by re-fetching pageblock
    migratetype through is_migrate_isolate_page(page).
    
    But, because this re-fetched information isn't used for
    __free_one_page(), first problem would not be solved.  This patch try to
    solve this situation to re-fetch pageblock migratetype before
    __free_one_page() and to use it for __free_one_page().
    
    In addition to move up position of this re-fetch, this patch use
    optimization technique, re-fetching migratetype only if there is isolate
    pageblock.  Pageblock isolation is rare event, so we can avoid
    re-fetching in common case with this optimization.
    
    This patch also correct migratetype of the tracepoint output.
    Signed-off-by: default avatarJoonsoo Kim <iamjoonsoo.kim@lge.com>
    Acked-by: default avatarMinchan Kim <minchan@kernel.org>
    Acked-by: default avatarMichal Nazarewicz <mina86@mina86.com>
    Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Cc: "Kirill A. Shutemov" <kirill@shutemov.name>
    Cc: Mel Gorman <mgorman@suse.de>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
    Cc: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
    Cc: Tang Chen <tangchen@cn.fujitsu.com>
    Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
    Cc: Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
    Cc: Wen Congyang <wency@cn.fujitsu.com>
    Cc: Marek Szyprowski <m.szyprowski@samsung.com>
    Cc: Laura Abbott <lauraa@codeaurora.org>
    Cc: Heesub Shin <heesub.shin@samsung.com>
    Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com>
    Cc: Ritesh Harjani <ritesh.list@gmail.com>
    Cc: Gioh Kim <gioh.kim@lge.com>
    Cc: <stable@vger.kernel.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    51bb1a40
page_alloc.c 182 KB