^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) .. _transhuge:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4) Transparent Hugepage Support
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) ============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7) This document describes design principles for Transparent Hugepage (THP)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) support and its interaction with other parts of the memory management
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) system.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) Design principles
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) =================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) - "graceful fallback": mm components which don't have transparent hugepage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) knowledge fall back to breaking huge pmd mapping into table of ptes and,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) if necessary, split a transparent hugepage. Therefore these components
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) can continue working on the regular pages or regular pte mappings.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) - if a hugepage allocation fails because of memory fragmentation,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) regular pages should be gracefully allocated instead and mixed in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) the same vma without any failure or significant delay and without
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) userland noticing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) - if some task quits and more hugepages become available (either
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) immediately in the buddy or through the VM), guest physical memory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) backed by regular pages should be relocated on hugepages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) automatically (with khugepaged)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) - it doesn't require memory reservation and in turn it uses hugepages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) whenever possible (the only possible reservation here is kernelcore=
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) to avoid unmovable pages to fragment all the memory but such a tweak
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) is not specific to transparent hugepage support and it's a generic
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) feature that applies to all dynamic high order allocations in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) kernel)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) get_user_pages and follow_page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) ==============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) get_user_pages and follow_page if run on a hugepage, will return the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) head or tail pages as usual (exactly as they would do on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) hugetlbfs). Most GUP users will only care about the actual physical
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) address of the page and its temporary pinning to release after the I/O
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) is complete, so they won't ever notice the fact the page is huge. But
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) if any driver is going to mangle over the page structure of the tail
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) page (like for checking page->mapping or other bits that are relevant
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) for the head page and not the tail page), it should be updated to jump
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) to check head page instead. Taking a reference on any head/tail page would
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) prevent the page from being split by anyone.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) .. note::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) these aren't new constraints to the GUP API, and they match the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) same constraints that apply to hugetlbfs too, so any driver capable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) of handling GUP on hugetlbfs will also work fine on transparent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) hugepage backed mappings.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) In case you can't handle compound pages if they're returned by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) follow_page, the FOLL_SPLIT bit can be specified as a parameter to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) follow_page, so that it will split the hugepages before returning
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) them.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) Graceful fallback
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) =================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) Code walking pagetables but unaware about huge pmds can simply call
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) split_huge_pmd(vma, pmd, addr) where the pmd is the one returned by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) pmd_offset. It's trivial to make the code transparent hugepage aware
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) by just grepping for "pmd_offset" and adding split_huge_pmd where
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) missing after pmd_offset returns the pmd. Thanks to the graceful
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) fallback design, with a one liner change, you can avoid to write
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) hundreds if not thousands of lines of complex code to make your code
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) hugepage aware.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) If you're not walking pagetables but you run into a physical hugepage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) that you can't handle natively in your code, you can split it by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) calling split_huge_page(page). This is what the Linux VM does before
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) it tries to swapout the hugepage for example. split_huge_page() can fail
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) if the page is pinned and you must handle this correctly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) Example to make mremap.c transparent hugepage aware with a one liner
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) change::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) diff --git a/mm/mremap.c b/mm/mremap.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) --- a/mm/mremap.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) +++ b/mm/mremap.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) @@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) return NULL;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) pmd = pmd_offset(pud, addr);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) + split_huge_pmd(vma, pmd, addr);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) if (pmd_none_or_clear_bad(pmd))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) return NULL;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) Locking in hugepage aware code
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) ==============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) We want as much code as possible hugepage aware, as calling
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) split_huge_page() or split_huge_pmd() has a cost.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) To make pagetable walks huge pmd aware, all you need to do is to call
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) pmd_trans_huge() on the pmd returned by pmd_offset. You must hold the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) mmap_lock in read (or write) mode to be sure a huge pmd cannot be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) created from under you by khugepaged (khugepaged collapse_huge_page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) takes the mmap_lock in write mode in addition to the anon_vma lock). If
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) pmd_trans_huge returns false, you just fallback in the old code
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) paths. If instead pmd_trans_huge returns true, you have to take the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) page table lock (pmd_lock()) and re-run pmd_trans_huge. Taking the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) page table lock will prevent the huge pmd being converted into a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) regular pmd from under you (split_huge_pmd can run in parallel to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) pagetable walk). If the second pmd_trans_huge returns false, you
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) should just drop the page table lock and fallback to the old code as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) before. Otherwise, you can proceed to process the huge pmd and the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) hugepage natively. Once finished, you can drop the page table lock.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) Refcounts and transparent huge pages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) ====================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) Refcounting on THP is mostly consistent with refcounting on other compound
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) pages:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) - get_page()/put_page() and GUP operate on head page's ->_refcount.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) - ->_refcount in tail pages is always zero: get_page_unless_zero() never
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) succeeds on tail pages.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) - map/unmap of the pages with PTE entry increment/decrement ->_mapcount
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) on relevant sub-page of the compound page.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) - map/unmap of the whole compound page is accounted for in compound_mapcount
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) (stored in first tail page). For file huge pages, we also increment
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) ->_mapcount of all sub-pages in order to have race-free detection of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) last unmap of subpages.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) PageDoubleMap() indicates that the page is *possibly* mapped with PTEs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) For anonymous pages, PageDoubleMap() also indicates ->_mapcount in all
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) subpages is offset up by one. This additional reference is required to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) get race-free detection of unmap of subpages when we have them mapped with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) both PMDs and PTEs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) This optimization is required to lower the overhead of per-subpage mapcount
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) tracking. The alternative is to alter ->_mapcount in all subpages on each
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) map/unmap of the whole compound page.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) For anonymous pages, we set PG_double_map when a PMD of the page is split
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) for the first time, but still have a PMD mapping. The additional references
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) go away with the last compound_mapcount.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) File pages get PG_double_map set on the first map of the page with PTE and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) goes away when the page gets evicted from the page cache.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) split_huge_page internally has to distribute the refcounts in the head
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) page to the tail pages before clearing all PG_head/tail bits from the page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) structures. It can be done easily for refcounts taken by page table
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) entries, but we don't have enough information on how to distribute any
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) additional pins (i.e. from get_user_pages). split_huge_page() fails any
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) requests to split pinned huge pages: it expects page count to be equal to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) the sum of mapcount of all sub-pages plus one (split_huge_page caller must
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) have a reference to the head page).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) split_huge_page uses migration entries to stabilize page->_refcount and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) page->_mapcount of anonymous pages. File pages just get unmapped.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) We are safe against physical memory scanners too: the only legitimate way
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) a scanner can get a reference to a page is get_page_unless_zero().
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) All tail pages have zero ->_refcount until atomic_add(). This prevents the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) scanner from getting a reference to the tail page up to that point. After the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) atomic_add() we don't care about the ->_refcount value. We already know how
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) many references should be uncharged from the head page.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) For head page get_page_unless_zero() will succeed and we don't mind. It's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) clear where references should go after split: it will stay on the head page.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) Note that split_huge_pmd() doesn't have any limitations on refcounting:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) pmd can be split at any point and never fails.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) Partial unmap and deferred_split_huge_page()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) ============================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) Unmapping part of THP (with munmap() or other way) is not going to free
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) memory immediately. Instead, we detect that a subpage of THP is not in use
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) in page_remove_rmap() and queue the THP for splitting if memory pressure
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) comes. Splitting will free up unused subpages.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) Splitting the page right away is not an option due to locking context in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) the place where we can detect partial unmap. It also might be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) counterproductive since in many cases partial unmap happens during exit(2) if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) a THP crosses a VMA boundary.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) The function deferred_split_huge_page() is used to queue a page for splitting.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) The splitting itself will happen when we get memory pressure via shrinker
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) interface.
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags