^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) .. _numaperf:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) =============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4) NUMA Locality
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) =============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7) Some platforms may have multiple types of memory attached to a compute
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) node. These disparate memory ranges may share some characteristics, such
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) as CPU cache coherence, but may have different performance. For example,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) different media types and buses affect bandwidth and latency.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) A system supports such heterogeneous memory by grouping each memory type
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) under different domains, or "nodes", based on locality and performance
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) characteristics. Some memory may share the same node as a CPU, and others
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) are provided as memory only nodes. While memory only nodes do not provide
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) CPUs, they may still be local to one or more compute nodes relative to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) other nodes. The following diagram shows one such example of two compute
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) nodes with local memory and a memory only node for each of compute node::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) +------------------+ +------------------+
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) | Compute Node 0 +-----+ Compute Node 1 |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) | Local Node0 Mem | | Local Node1 Mem |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) +--------+---------+ +--------+---------+
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) | |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) +--------+---------+ +--------+---------+
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) | Slower Node2 Mem | | Slower Node3 Mem |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) +------------------+ +--------+---------+
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) A "memory initiator" is a node containing one or more devices such as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) CPUs or separate memory I/O devices that can initiate memory requests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) A "memory target" is a node containing one or more physical address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) ranges accessible from one or more memory initiators.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) When multiple memory initiators exist, they may not all have the same
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) performance when accessing a given memory target. Each initiator-target
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) pair may be organized into different ranked access classes to represent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) this relationship. The highest performing initiator to a given target
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) is considered to be one of that target's local initiators, and given
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) the highest access class, 0. Any given target may have one or more
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) local initiators, and any given initiator may have multiple local
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) memory targets.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) To aid applications matching memory targets with their initiators, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) kernel provides symlinks to each other. The following example lists the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) relationship for the access class "0" memory initiators and targets::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) # symlinks -v /sys/devices/system/node/nodeX/access0/targets/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) relative: /sys/devices/system/node/nodeX/access0/targets/nodeY -> ../../nodeY
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) # symlinks -v /sys/devices/system/node/nodeY/access0/initiators/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) relative: /sys/devices/system/node/nodeY/access0/initiators/nodeX -> ../../nodeX
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) A memory initiator may have multiple memory targets in the same access
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) class. The target memory's initiators in a given class indicate the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) nodes' access characteristics share the same performance relative to other
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) linked initiator nodes. Each target within an initiator's access class,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) though, do not necessarily perform the same as each other.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) The access class "1" is used to allow differentiation between initiators
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) that are CPUs and hence suitable for generic task scheduling, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) IO initiators such as GPUs and NICs. Unlike access class 0, only
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) nodes containing CPUs are considered.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) ================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) NUMA Performance
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) ================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) Applications may wish to consider which node they want their memory to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) be allocated from based on the node's performance characteristics. If
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) the system provides these attributes, the kernel exports them under the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) node sysfs hierarchy by appending the attributes directory under the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) memory node's access class 0 initiators as follows::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) /sys/devices/system/node/nodeY/access0/initiators/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) These attributes apply only when accessed from nodes that have the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) are linked under the this access's inititiators.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) The performance characteristics the kernel provides for the local initiators
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) are exported are as follows::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) # tree -P "read*|write*" /sys/devices/system/node/nodeY/access0/initiators/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) /sys/devices/system/node/nodeY/access0/initiators/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) |-- read_bandwidth
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) |-- read_latency
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) |-- write_bandwidth
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) `-- write_latency
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) The bandwidth attributes are provided in MiB/second.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) The latency attributes are provided in nanoseconds.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) The values reported here correspond to the rated latency and bandwidth
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) for the platform.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) Access class 1 takes the same form but only includes values for CPU to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) memory activity.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) ==========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) NUMA Cache
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) ==========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) System memory may be constructed in a hierarchy of elements with various
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) performance characteristics in order to provide large address space of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) slower performing memory cached by a smaller higher performing memory. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) system physical addresses memory initiators are aware of are provided
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) by the last memory level in the hierarchy. The system meanwhile uses
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) higher performing memory to transparently cache access to progressively
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) slower levels.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) The term "far memory" is used to denote the last level memory in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) hierarchy. Each increasing cache level provides higher performing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) initiator access, and the term "near memory" represents the fastest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) cache provided by the system.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) This numbering is different than CPU caches where the cache level (ex:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) L1, L2, L3) uses the CPU-side view where each increased level is lower
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) performing. In contrast, the memory cache level is centric to the last
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) level memory, so the higher numbered cache level corresponds to memory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) nearer to the CPU, and further from far memory.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) The memory-side caches are not directly addressable by software. When
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) software accesses a system address, the system will return it from the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) near memory cache if it is present. If it is not present, the system
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) accesses the next level of memory until there is either a hit in that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) cache level, or it reaches far memory.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) An application does not need to know about caching attributes in order
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) to use the system. Software may optionally query the memory cache
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) attributes in order to maximize the performance out of such a setup.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) If the system provides a way for the kernel to discover this information,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) for example with ACPI HMAT (Heterogeneous Memory Attribute Table),
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) the kernel will append these attributes to the NUMA node memory target.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) When the kernel first registers a memory cache with a node, the kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) will create the following directory::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) /sys/devices/system/node/nodeX/memory_side_cache/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) If that directory is not present, the system either does not provide
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) a memory-side cache, or that information is not accessible to the kernel.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) The attributes for each level of cache is provided under its cache
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) level index::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) /sys/devices/system/node/nodeX/memory_side_cache/indexA/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) /sys/devices/system/node/nodeX/memory_side_cache/indexB/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) /sys/devices/system/node/nodeX/memory_side_cache/indexC/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) Each cache level's directory provides its attributes. For example, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) following shows a single cache level and the attributes available for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) software to query::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) # tree sys/devices/system/node/node0/memory_side_cache/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) /sys/devices/system/node/node0/memory_side_cache/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) |-- index1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) | |-- indexing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) | |-- line_size
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) | |-- size
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) | `-- write_policy
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) The "indexing" will be 0 if it is a direct-mapped cache, and non-zero
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) for any other indexed based, multi-way associativity.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) The "line_size" is the number of bytes accessed from the next cache
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) level on a miss.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) The "size" is the number of bytes provided by this cache level.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) The "write_policy" will be 0 for write-back, and non-zero for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) write-through caching.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) ========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) See Also
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) ========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) [1] https://www.uefi.org/sites/default/files/resources/ACPI_6_2.pdf
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) - Section 5.2.27
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags