^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) .. SPDX-License-Identifier: GPL-2.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ======================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4) The x86 kvm shadow mmu
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) ======================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7) The mmu (in arch/x86/kvm, files mmu.[ch] and paging_tmpl.h) is responsible
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) for presenting a standard x86 mmu to the guest, while translating guest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) physical addresses to host physical addresses.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) The mmu code attempts to satisfy the following requirements:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) - correctness:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) the guest should not be able to determine that it is running
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) on an emulated mmu except for timing (we attempt to comply
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) with the specification, not emulate the characteristics of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) a particular implementation such as tlb size)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) - security:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) the guest must not be able to touch host memory not assigned
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) to it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) - performance:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) minimize the performance penalty imposed by the mmu
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) - scaling:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) need to scale to large memory and large vcpu guests
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) - hardware:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) support the full range of x86 virtualization hardware
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) - integration:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) Linux memory management code must be in control of guest memory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) so that swapping, page migration, page merging, transparent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) hugepages, and similar features work without change
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) - dirty tracking:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) report writes to guest memory to enable live migration
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) and framebuffer-based displays
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) - footprint:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) keep the amount of pinned kernel memory low (most memory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) should be shrinkable)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) - reliability:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) avoid multipage or GFP_ATOMIC allocations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) Acronyms
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) ========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) ==== ====================================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) pfn host page frame number
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) hpa host physical address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) hva host virtual address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) gfn guest frame number
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) gpa guest physical address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) gva guest virtual address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) ngpa nested guest physical address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) ngva nested guest virtual address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) pte page table entry (used also to refer generically to paging structure
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) entries)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) gpte guest pte (referring to gfns)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) spte shadow pte (referring to pfns)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) tdp two dimensional paging (vendor neutral term for NPT and EPT)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) ==== ====================================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) Virtual and real hardware supported
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) ===================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) The mmu supports first-generation mmu hardware, which allows an atomic switch
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) of the current paging mode and cr3 during guest entry, as well as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) two-dimensional paging (AMD's NPT and Intel's EPT). The emulated hardware
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) it exposes is the traditional 2/3/4 level x86 mmu, with support for global
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) pages, pae, pse, pse36, cr0.wp, and 1GB pages. Emulated hardware also
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) able to expose NPT capable hardware on NPT capable hosts.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) Translation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) ===========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) The primary job of the mmu is to program the processor's mmu to translate
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) addresses for the guest. Different translations are required at different
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) times:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) - when guest paging is disabled, we translate guest physical addresses to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) host physical addresses (gpa->hpa)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) - when guest paging is enabled, we translate guest virtual addresses, to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) guest physical addresses, to host physical addresses (gva->gpa->hpa)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) - when the guest launches a guest of its own, we translate nested guest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) virtual addresses, to nested guest physical addresses, to guest physical
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) addresses, to host physical addresses (ngva->ngpa->gpa->hpa)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) The primary challenge is to encode between 1 and 3 translations into hardware
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) that support only 1 (traditional) and 2 (tdp) translations. When the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) number of required translations matches the hardware, the mmu operates in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) direct mode; otherwise it operates in shadow mode (see below).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) Memory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) ======
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) Guest memory (gpa) is part of the user address space of the process that is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) using kvm. Userspace defines the translation between guest addresses and user
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) addresses (gpa->hva); note that two gpas may alias to the same hva, but not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) vice versa.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) These hvas may be backed using any method available to the host: anonymous
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) memory, file backed memory, and device memory. Memory might be paged by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) host at any time.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) Events
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) ======
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) The mmu is driven by events, some from the guest, some from the host.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) Guest generated events:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) - writes to control registers (especially cr3)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) - invlpg/invlpga instruction execution
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) - access to missing or protected translations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) Host generated events:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) - changes in the gpa->hpa translation (either through gpa->hva changes or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) through hva->hpa changes)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) - memory pressure (the shrinker)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) Shadow pages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) The principal data structure is the shadow page, 'struct kvm_mmu_page'. A
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) shadow page contains 512 sptes, which can be either leaf or nonleaf sptes. A
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) shadow page may contain a mix of leaf and nonleaf sptes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) A nonleaf spte allows the hardware mmu to reach the leaf pages and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) is not related to a translation directly. It points to other shadow pages.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) A leaf spte corresponds to either one or two translations encoded into
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) one paging structure entry. These are always the lowest level of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) translation stack, with optional higher level translations left to NPT/EPT.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) Leaf ptes point at guest pages.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) The following table shows translations encoded by leaf ptes, with higher-level
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) translations in parentheses:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) Non-nested guests::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) nonpaging: gpa->hpa
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) paging: gva->gpa->hpa
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) paging, tdp: (gva->)gpa->hpa
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) Nested guests::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) non-tdp: ngva->gpa->hpa (*)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) tdp: (ngva->)ngpa->gpa->hpa
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) (*) the guest hypervisor will encode the ngva->gpa translation into its page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) tables if npt is not present
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) Shadow pages contain the following information:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) role.level:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) The level in the shadow paging hierarchy that this shadow page belongs to.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) 1=4k sptes, 2=2M sptes, 3=1G sptes, etc.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) role.direct:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) If set, leaf sptes reachable from this page are for a linear range.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) Examples include real mode translation, large guest pages backed by small
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) host pages, and gpa->hpa translations when NPT or EPT is active.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) The linear range starts at (gfn << PAGE_SHIFT) and its size is determined
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) by role.level (2MB for first level, 1GB for second level, 0.5TB for third
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) level, 256TB for fourth level)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) If clear, this page corresponds to a guest page table denoted by the gfn
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) field.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) role.quadrant:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) When role.gpte_is_8_bytes=0, the guest uses 32-bit gptes while the host uses 64-bit
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) sptes. That means a guest page table contains more ptes than the host,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) so multiple shadow pages are needed to shadow one guest page.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) For first-level shadow pages, role.quadrant can be 0 or 1 and denotes the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) first or second 512-gpte block in the guest page table. For second-level
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) page tables, each 32-bit gpte is converted to two 64-bit sptes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) (since each first-level guest page is shadowed by two first-level
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) shadow pages) so role.quadrant takes values in the range 0..3. Each
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) quadrant maps 1GB virtual address space.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) role.access:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) Inherited guest access permissions from the parent ptes in the form uwx.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) Note execute permission is positive, not negative.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) role.invalid:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) The page is invalid and should not be used. It is a root page that is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) currently pinned (by a cpu hardware register pointing to it); once it is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) unpinned it will be destroyed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) role.gpte_is_8_bytes:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) Reflects the size of the guest PTE for which the page is valid, i.e. '1'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) if 64-bit gptes are in use, '0' if 32-bit gptes are in use.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) role.nxe:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) Contains the value of efer.nxe for which the page is valid.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) role.cr0_wp:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) Contains the value of cr0.wp for which the page is valid.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) role.smep_andnot_wp:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) Contains the value of cr4.smep && !cr0.wp for which the page is valid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) (pages for which this is true are different from other pages; see the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) treatment of cr0.wp=0 below).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) role.smap_andnot_wp:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) Contains the value of cr4.smap && !cr0.wp for which the page is valid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) (pages for which this is true are different from other pages; see the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) treatment of cr0.wp=0 below).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) role.ept_sp:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) This is a virtual flag to denote a shadowed nested EPT page. ept_sp
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) is true if "cr0_wp && smap_andnot_wp", an otherwise invalid combination.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) role.smm:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) Is 1 if the page is valid in system management mode. This field
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) determines which of the kvm_memslots array was used to build this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) shadow page; it is also used to go back from a struct kvm_mmu_page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) to a memslot, through the kvm_memslots_for_spte_role macro and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) __gfn_to_memslot.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) role.ad_disabled:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) Is 1 if the MMU instance cannot use A/D bits. EPT did not have A/D
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) bits before Haswell; shadow EPT page tables also cannot use A/D bits
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) if the L1 hypervisor does not enable them.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) gfn:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209) Either the guest page table containing the translations shadowed by this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) page, or the base page frame for linear translations. See role.direct.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) spt:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212) A pageful of 64-bit sptes containing the translations for this page.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) Accessed by both kvm and hardware.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) The page pointed to by spt will have its page->private pointing back
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215) at the shadow page structure.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) sptes in spt point either at guest pages, or at lower-level shadow pages.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) Specifically, if sp1 and sp2 are shadow pages, then sp1->spt[n] may point
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218) at __pa(sp2->spt). sp2 will point back at sp1 through parent_pte.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) The spt array forms a DAG structure with the shadow page as a node, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220) guest pages as leaves.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) gfns:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) An array of 512 guest frame numbers, one for each present pte. Used to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) perform a reverse map from a pte to a gfn. When role.direct is set, any
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224) element of this array can be calculated from the gfn field when used, in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) this case, the array of gfns is not allocated. See role.direct and gfn.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226) root_count:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) A counter keeping track of how many hardware registers (guest cr3 or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228) pdptrs) are now pointing at the page. While this counter is nonzero, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) page cannot be destroyed. See role.invalid.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) parent_ptes:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231) The reverse mapping for the pte/ptes pointing at this page's spt. If
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) parent_ptes bit 0 is zero, only one spte points at this page and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233) parent_ptes points at this single spte, otherwise, there exists multiple
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234) sptes pointing at this page and (parent_ptes & ~0x1) points at a data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) structure with a list of parent sptes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236) unsync:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237) If true, then the translations in this page may not match the guest's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) translation. This is equivalent to the state of the tlb when a pte is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) changed but before the tlb entry is flushed. Accordingly, unsync ptes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240) are synchronized when the guest executes invlpg or flushes its tlb by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241) other means. Valid for leaf pages.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) unsync_children:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) How many sptes in the page point at pages that are unsync (or have
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) unsynchronized children).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245) unsync_child_bitmap:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246) A bitmap indicating which sptes in spt point (directly or indirectly) at
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) pages that may be unsynchronized. Used to quickly locate all unsychronized
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) pages reachable from a given page.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249) clear_spte_count:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) Only present on 32-bit hosts, where a 64-bit spte cannot be written
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) atomically. The reader uses this while running out of the MMU lock
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252) to detect in-progress updates and retry them until the writer has
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) finished the write.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) write_flooding_count:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) A guest may write to a page table many times, causing a lot of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256) emulations if the page needs to be write-protected (see "Synchronized
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) and unsynchronized pages" below). Leaf pages can be unsynchronized
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258) so that they do not trigger frequent emulation, but this is not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) possible for non-leafs. This field counts the number of emulations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260) since the last time the page table was actually used; if emulation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) is triggered too frequently on this page, KVM will unmap the page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) to avoid emulation in the future.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) Reverse map
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) ===========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) The mmu maintains a reverse mapping whereby all ptes mapping a page can be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268) reached given its gfn. This is used, for example, when swapping out a page.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) Synchronized and unsynchronized pages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271) =====================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273) The guest uses two events to synchronize its tlb and page tables: tlb flushes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274) and page invalidations (invlpg).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276) A tlb flush means that we need to synchronize all sptes reachable from the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277) guest's cr3. This is expensive, so we keep all guest page tables write
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278) protected, and synchronize sptes to gptes when a gpte is written.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280) A special case is when a guest page table is reachable from the current
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281) guest cr3. In this case, the guest is obliged to issue an invlpg instruction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282) before using the translation. We take advantage of that by removing write
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283) protection from the guest page, and allowing the guest to modify it freely.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284) We synchronize modified gptes when the guest invokes invlpg. This reduces
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285) the amount of emulation we have to do when the guest modifies multiple gptes,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286) or when the a guest page is no longer used as a page table and is used for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287) random guest data.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289) As a side effect we have to resynchronize all reachable unsynchronized shadow
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) pages on a tlb flush.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293) Reaction to events
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294) ==================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296) - guest page fault (or npt page fault, or ept violation)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298) This is the most complicated event. The cause of a page fault can be:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300) - a true guest fault (the guest translation won't allow the access) (*)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301) - access to a missing translation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302) - access to a protected translation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303) - when logging dirty pages, memory is write protected
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) - synchronized shadow pages are write protected (*)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305) - access to untranslatable memory (mmio)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307) (*) not applicable in direct mode
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309) Handling a page fault is performed as follows:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311) - if the RSV bit of the error code is set, the page fault is caused by guest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312) accessing MMIO and cached MMIO information is available.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314) - walk shadow page table
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315) - check for valid generation number in the spte (see "Fast invalidation of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) MMIO sptes" below)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317) - cache the information to vcpu->arch.mmio_gva, vcpu->arch.mmio_access and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318) vcpu->arch.mmio_gfn, and call the emulator
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320) - If both P bit and R/W bit of error code are set, this could possibly
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321) be handled as a "fast page fault" (fixed without taking the MMU lock). See
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322) the description in Documentation/virt/kvm/locking.rst.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324) - if needed, walk the guest page tables to determine the guest translation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325) (gva->gpa or ngpa->gpa)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327) - if permissions are insufficient, reflect the fault back to the guest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329) - determine the host page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331) - if this is an mmio request, there is no host page; cache the info to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332) vcpu->arch.mmio_gva, vcpu->arch.mmio_access and vcpu->arch.mmio_gfn
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334) - walk the shadow page table to find the spte for the translation,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335) instantiating missing intermediate page tables as necessary
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) - If this is an mmio request, cache the mmio info to the spte and set some
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338) reserved bit on the spte (see callers of kvm_mmu_set_mmio_spte_mask)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340) - try to unsynchronize the page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342) - if successful, we can let the guest continue and modify the gpte
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344) - emulate the instruction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346) - if failed, unshadow the page and let the guest continue
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348) - update any translations that were modified by the instruction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) invlpg handling:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352) - walk the shadow page hierarchy and drop affected translations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353) - try to reinstantiate the indicated translation in the hope that the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354) guest will use it in the near future
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) Guest control register updates:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358) - mov to cr3
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360) - look up new shadow roots
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361) - synchronize newly reachable shadow pages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363) - mov to cr0/cr4/efer
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365) - set up mmu context for new paging mode
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366) - look up new shadow roots
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367) - synchronize newly reachable shadow pages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369) Host translation updates:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) - mmu notifier called with updated hva
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372) - look up affected sptes through reverse map
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373) - drop (or update) translations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375) Emulating cr0.wp
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376) ================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378) If tdp is not enabled, the host must keep cr0.wp=1 so page write protection
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379) works for the guest kernel, not guest guest userspace. When the guest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 380) cr0.wp=1, this does not present a problem. However when the guest cr0.wp=0,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 381) we cannot map the permissions for gpte.u=1, gpte.w=0 to any spte (the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 382) semantics require allowing any guest kernel access plus user read access).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 383)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 384) We handle this by mapping the permissions to two possible sptes, depending
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 385) on fault type:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 386)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 387) - kernel write fault: spte.u=0, spte.w=1 (allows full kernel access,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 388) disallows user access)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 389) - read fault: spte.u=1, spte.w=0 (allows full read access, disallows kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 390) write access)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 391)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 392) (user write faults generate a #PF)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 393)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 394) In the first case there are two additional complications:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 395)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 396) - if CR4.SMEP is enabled: since we've turned the page into a kernel page,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 397) the kernel may now execute it. We handle this by also setting spte.nx.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 398) If we get a user fetch or read fault, we'll change spte.u=1 and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 399) spte.nx=gpte.nx back. For this to work, KVM forces EFER.NX to 1 when
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 400) shadow paging is in use.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 401) - if CR4.SMAP is disabled: since the page has been changed to a kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 402) page, it can not be reused when CR4.SMAP is enabled. We set
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 403) CR4.SMAP && !CR0.WP into shadow page's role to avoid this case. Note,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 404) here we do not care the case that CR4.SMAP is enabled since KVM will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 405) directly inject #PF to guest due to failed permission check.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 406)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 407) To prevent an spte that was converted into a kernel page with cr0.wp=0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 408) from being written by the kernel after cr0.wp has changed to 1, we make
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 409) the value of cr0.wp part of the page role. This means that an spte created
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 410) with one value of cr0.wp cannot be used when cr0.wp has a different value -
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 411) it will simply be missed by the shadow page lookup code. A similar issue
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 412) exists when an spte created with cr0.wp=0 and cr4.smep=0 is used after
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 413) changing cr4.smep to 1. To avoid this, the value of !cr0.wp && cr4.smep
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 414) is also made a part of the page role.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 415)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 416) Large pages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 417) ===========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 418)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 419) The mmu supports all combinations of large and small guest and host pages.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 420) Supported page sizes include 4k, 2M, 4M, and 1G. 4M pages are treated as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 421) two separate 2M pages, on both guest and host, since the mmu always uses PAE
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 422) paging.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 423)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 424) To instantiate a large spte, four constraints must be satisfied:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 425)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 426) - the spte must point to a large host page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 427) - the guest pte must be a large pte of at least equivalent size (if tdp is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 428) enabled, there is no guest pte and this condition is satisfied)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 429) - if the spte will be writeable, the large page frame may not overlap any
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 430) write-protected pages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 431) - the guest page must be wholly contained by a single memory slot
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 432)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 433) To check the last two conditions, the mmu maintains a ->disallow_lpage set of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 434) arrays for each memory slot and large page size. Every write protected page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 435) causes its disallow_lpage to be incremented, thus preventing instantiation of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 436) a large spte. The frames at the end of an unaligned memory slot have
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 437) artificially inflated ->disallow_lpages so they can never be instantiated.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 438)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 439) Fast invalidation of MMIO sptes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 440) ===============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 441)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 442) As mentioned in "Reaction to events" above, kvm will cache MMIO
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 443) information in leaf sptes. When a new memslot is added or an existing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 444) memslot is changed, this information may become stale and needs to be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 445) invalidated. This also needs to hold the MMU lock while walking all
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 446) shadow pages, and is made more scalable with a similar technique.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 447)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 448) MMIO sptes have a few spare bits, which are used to store a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 449) generation number. The global generation number is stored in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 450) kvm_memslots(kvm)->generation, and increased whenever guest memory info
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 451) changes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 452)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 453) When KVM finds an MMIO spte, it checks the generation number of the spte.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 454) If the generation number of the spte does not equal the global generation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 455) number, it will ignore the cached MMIO information and handle the page
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 456) fault through the slow path.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 457)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 458) Since only 18 bits are used to store generation-number on mmio spte, all
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 459) pages are zapped when there is an overflow.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 460)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 461) Unfortunately, a single memory access might access kvm_memslots(kvm) multiple
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 462) times, the last one happening when the generation number is retrieved and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 463) stored into the MMIO spte. Thus, the MMIO spte might be created based on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 464) out-of-date information, but with an up-to-date generation number.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 465)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 466) To avoid this, the generation number is incremented again after synchronize_srcu
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 467) returns; thus, bit 63 of kvm_memslots(kvm)->generation set to 1 only during a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 468) memslot update, while some SRCU readers might be using the old copy. We do not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 469) want to use an MMIO sptes created with an odd generation number, and we can do
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 470) this without losing a bit in the MMIO spte. The "update in-progress" bit of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 471) generation is not stored in MMIO spte, and is so is implicitly zero when the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 472) generation is extracted out of the spte. If KVM is unlucky and creates an MMIO
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 473) spte while an update is in-progress, the next access to the spte will always be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 474) a cache miss. For example, a subsequent access during the update window will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 475) miss due to the in-progress flag diverging, while an access after the update
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 476) window closes will have a higher generation number (as compared to the spte).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 477)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 478)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 479) Further reading
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 480) ===============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 481)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 482) - NPT presentation from KVM Forum 2008
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 483) https://www.linux-kvm.org/images/c/c8/KvmForum2008%24kdf2008_21.pdf
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags