Commit dff03381 authored by Muchun Song's avatar Muchun Song Committed by Andrew Morton

mm: hugetlb_vmemmap: introduce the name HVO

It it inconvenient to mention the feature of optimizing vmemmap pages
associated with HugeTLB pages when communicating with others since there
is no specific or abbreviated name for it when it is first introduced. 
Let us give it a name HVO (HugeTLB Vmemmap Optimization) from now.

This commit also updates the document about "hugetlb_free_vmemmap" by the
way discussed in thread [1].

Link: https://lore.kernel.org/all/21aae898-d54d-cc4b-a11f-1bb7fddcfffa@redhat.com/ [1]
Link: https://lkml.kernel.org/r/20220628092235.91270-4-songmuchun@bytedance.comSigned-off-by: default avatarMuchun Song <songmuchun@bytedance.com>
Reviewed-by: default avatarOscar Salvador <osalvador@suse.de>
Reviewed-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Will Deacon <will@kernel.org>
Cc: Xiongchun Duan <duanxiongchun@bytedance.com>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
parent cf5472e5
...@@ -1731,12 +1731,13 @@ ...@@ -1731,12 +1731,13 @@
hugetlb_free_vmemmap= hugetlb_free_vmemmap=
[KNL] Reguires CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP [KNL] Reguires CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
enabled. enabled.
Control if HugeTLB Vmemmap Optimization (HVO) is enabled.
Allows heavy hugetlb users to free up some more Allows heavy hugetlb users to free up some more
memory (7 * PAGE_SIZE for each 2MB hugetlb page). memory (7 * PAGE_SIZE for each 2MB hugetlb page).
Format: { [oO][Nn]/Y/y/1 | [oO][Ff]/N/n/0 (default) } Format: { on | off (default) }
[oO][Nn]/Y/y/1: enable the feature on: enable HVO
[oO][Ff]/N/n/0: disable the feature off: disable HVO
Built with CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=y, Built with CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=y,
the default is on. the default is on.
......
...@@ -164,8 +164,8 @@ default_hugepagesz ...@@ -164,8 +164,8 @@ default_hugepagesz
will all result in 256 2M huge pages being allocated. Valid default will all result in 256 2M huge pages being allocated. Valid default
huge page size is architecture dependent. huge page size is architecture dependent.
hugetlb_free_vmemmap hugetlb_free_vmemmap
When CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP is set, this enables optimizing When CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP is set, this enables HugeTLB
unused vmemmap pages associated with each HugeTLB page. Vmemmap Optimization (HVO).
When multiple huge page sizes are supported, ``/proc/sys/vm/nr_hugepages`` When multiple huge page sizes are supported, ``/proc/sys/vm/nr_hugepages``
indicates the current number of pre-allocated huge pages of the default size. indicates the current number of pre-allocated huge pages of the default size.
......
...@@ -653,8 +653,8 @@ block might fail: ...@@ -653,8 +653,8 @@ block might fail:
- Concurrent activity that operates on the same physical memory area, such as - Concurrent activity that operates on the same physical memory area, such as
allocating gigantic pages, can result in temporary offlining failures. allocating gigantic pages, can result in temporary offlining failures.
- Out of memory when dissolving huge pages, especially when freeing unused - Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap
vmemmap pages associated with each hugetlb page is enabled. Optimization (HVO) is enabled.
Offlining code may be able to migrate huge page contents, but may not be able Offlining code may be able to migrate huge page contents, but may not be able
to dissolve the source huge page because it fails allocating (unmovable) pages to dissolve the source huge page because it fails allocating (unmovable) pages
......
...@@ -569,8 +569,7 @@ This knob is not available when the size of 'struct page' (a structure defined ...@@ -569,8 +569,7 @@ This knob is not available when the size of 'struct page' (a structure defined
in include/linux/mm_types.h) is not power of two (an unusual system config could in include/linux/mm_types.h) is not power of two (an unusual system config could
result in this). result in this).
Enable (set to 1) or disable (set to 0) the feature of optimizing vmemmap pages Enable (set to 1) or disable (set to 0) HugeTLB Vmemmap Optimization (HVO).
associated with each HugeTLB page.
Once enabled, the vmemmap pages of subsequent allocation of HugeTLB pages from Once enabled, the vmemmap pages of subsequent allocation of HugeTLB pages from
buddy allocator will be optimized (7 pages per 2MB HugeTLB page and 4095 pages buddy allocator will be optimized (7 pages per 2MB HugeTLB page and 4095 pages
......
...@@ -7,6 +7,8 @@ A vmemmap diet for HugeTLB and Device DAX ...@@ -7,6 +7,8 @@ A vmemmap diet for HugeTLB and Device DAX
HugeTLB HugeTLB
======= =======
This section is to explain how HugeTLB Vmemmap Optimization (HVO) works.
The struct page structures (page structs) are used to describe a physical The struct page structures (page structs) are used to describe a physical
page frame. By default, there is a one-to-one mapping from a page frame to page frame. By default, there is a one-to-one mapping from a page frame to
it's corresponding page struct. it's corresponding page struct.
......
...@@ -247,8 +247,7 @@ config HUGETLB_PAGE ...@@ -247,8 +247,7 @@ config HUGETLB_PAGE
# #
# Select this config option from the architecture Kconfig, if it is preferred # Select this config option from the architecture Kconfig, if it is preferred
# to enable the feature of minimizing overhead of struct page associated with # to enable the feature of HugeTLB Vmemmap Optimization (HVO).
# each HugeTLB page.
# #
config ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP config ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
bool bool
...@@ -259,14 +258,13 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP ...@@ -259,14 +258,13 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP
depends on SPARSEMEM_VMEMMAP depends on SPARSEMEM_VMEMMAP
config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON
bool "Default optimizing vmemmap pages of HugeTLB to on" bool "HugeTLB Vmemmap Optimization (HVO) defaults to on"
default n default n
depends on HUGETLB_PAGE_OPTIMIZE_VMEMMAP depends on HUGETLB_PAGE_OPTIMIZE_VMEMMAP
help help
When using HUGETLB_PAGE_OPTIMIZE_VMEMMAP, the optimizing unused vmemmap The HugeTLB VmemmapvOptimization (HVO) defaults to off. Say Y here to
pages associated with each HugeTLB page is default off. Say Y here enable HVO by default. It can be disabled via hugetlb_free_vmemmap=off
to enable optimizing vmemmap pages of HugeTLB by default. It can then (boot command line) or hugetlb_optimize_vmemmap (sysctl).
be disabled on the command line via hugetlb_free_vmemmap=off.
config MEMFD_CREATE config MEMFD_CREATE
def_bool TMPFS || HUGETLBFS def_bool TMPFS || HUGETLBFS
......
...@@ -208,8 +208,7 @@ enum pageflags { ...@@ -208,8 +208,7 @@ enum pageflags {
DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key);
/* /*
* If the feature of optimizing vmemmap pages associated with each HugeTLB * If HVO is enabled, the head vmemmap page frame is reused and all of the tail
* page is enabled, the head vmemmap page frame is reused and all of the tail
* vmemmap addresses map to the head vmemmap page frame (furture details can * vmemmap addresses map to the head vmemmap page frame (furture details can
* refer to the figure at the head of the mm/hugetlb_vmemmap.c). In other * refer to the figure at the head of the mm/hugetlb_vmemmap.c). In other
* words, there are more than one page struct with PG_head associated with each * words, there are more than one page struct with PG_head associated with each
......
// SPDX-License-Identifier: GPL-2.0 // SPDX-License-Identifier: GPL-2.0
/* /*
* Optimize vmemmap pages associated with HugeTLB * HugeTLB Vmemmap Optimization (HVO)
* *
* Copyright (c) 2020, Bytedance. All rights reserved. * Copyright (c) 2020, ByteDance. All rights reserved.
* *
* Author: Muchun Song <songmuchun@bytedance.com> * Author: Muchun Song <songmuchun@bytedance.com>
* *
...@@ -156,8 +156,8 @@ void __init hugetlb_vmemmap_init(struct hstate *h) ...@@ -156,8 +156,8 @@ void __init hugetlb_vmemmap_init(struct hstate *h)
/* /*
* There are only (RESERVE_VMEMMAP_SIZE / sizeof(struct page)) struct * There are only (RESERVE_VMEMMAP_SIZE / sizeof(struct page)) struct
* page structs that can be used when CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, * page structs that can be used when HVO is enabled, add a BUILD_BUG_ON
* so add a BUILD_BUG_ON to catch invalid usage of the tail struct page. * to catch invalid usage of the tail page structs.
*/ */
BUILD_BUG_ON(__NR_USED_SUBPAGE >= BUILD_BUG_ON(__NR_USED_SUBPAGE >=
RESERVE_VMEMMAP_SIZE / sizeof(struct page)); RESERVE_VMEMMAP_SIZE / sizeof(struct page));
......
// SPDX-License-Identifier: GPL-2.0 // SPDX-License-Identifier: GPL-2.0
/* /*
* Optimize vmemmap pages associated with HugeTLB * HugeTLB Vmemmap Optimization (HVO)
* *
* Copyright (c) 2020, Bytedance. All rights reserved. * Copyright (c) 2020, ByteDance. All rights reserved.
* *
* Author: Muchun Song <songmuchun@bytedance.com> * Author: Muchun Song <songmuchun@bytedance.com>
*/ */
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment