^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) =========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) Hardware Latency Detector
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) =========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) Introduction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) -------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) The tracer hwlat_detector is a special purpose tracer that is used to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) detect large system latencies induced by the behavior of certain underlying
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) hardware or firmware, independent of Linux itself. The code was developed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) originally to detect SMIs (System Management Interrupts) on x86 systems,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) however there is nothing x86 specific about this patchset. It was
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) originally written for use by the "RT" patch since the Real Time
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) kernel is highly latency sensitive.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) SMIs are not serviced by the Linux kernel, which means that it does not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) even know that they are occuring. SMIs are instead set up by BIOS code
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) and are serviced by BIOS code, usually for "critical" events such as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) management of thermal sensors and fans. Sometimes though, SMIs are used for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) other tasks and those tasks can spend an inordinate amount of time in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) handler (sometimes measured in milliseconds). Obviously this is a problem if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) you are trying to keep event service latencies down in the microsecond range.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) The hardware latency detector works by hogging one of the cpus for configurable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) amounts of time (with interrupts disabled), polling the CPU Time Stamp Counter
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) for some period, then looking for gaps in the TSC data. Any gap indicates a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) time when the polling was interrupted and since the interrupts are disabled,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) the only thing that could do that would be an SMI or other hardware hiccup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) (or an NMI, but those can be tracked).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) Note that the hwlat detector should *NEVER* be used in a production environment.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) It is intended to be run manually to determine if the hardware platform has a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) problem with long system firmware service routines.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) ------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) Write the ASCII text "hwlat" into the current_tracer file of the tracing system
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) (mounted at /sys/kernel/tracing or /sys/kernel/tracing). It is possible to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) redefine the threshold in microseconds (us) above which latency spikes will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) be taken into account.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) Example::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) # echo hwlat > /sys/kernel/tracing/current_tracer
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) # echo 100 > /sys/kernel/tracing/tracing_thresh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) The /sys/kernel/tracing/hwlat_detector interface contains the following files:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) - width - time period to sample with CPUs held (usecs)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) must be less than the total window size (enforced)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) - window - total period of sampling, width being inside (usecs)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) By default the width is set to 500,000 and window to 1,000,000, meaning that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) for every 1,000,000 usecs (1s) the hwlat detector will spin for 500,000 usecs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) (0.5s). If tracing_thresh contains zero when hwlat tracer is enabled, it will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) change to a default of 10 usecs. If any latencies that exceed the threshold is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) observed then the data will be written to the tracing ring buffer.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) The minimum sleep time between periods is 1 millisecond. Even if width
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) is less than 1 millisecond apart from window, to allow the system to not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) be totally starved.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) If tracing_thresh was zero when hwlat detector was started, it will be set
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) back to zero if another tracer is loaded. Note, the last value in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) tracing_thresh that hwlat detector had will be saved and this value will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) be restored in tracing_thresh if it is still zero when hwlat detector is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) started again.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) The following tracing directory files are used by the hwlat_detector:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) in /sys/kernel/tracing:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) - tracing_threshold - minimum latency value to be considered (usecs)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) - tracing_max_latency - maximum hardware latency actually observed (usecs)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) - tracing_cpumask - the CPUs to move the hwlat thread across
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) - hwlat_detector/width - specified amount of time to spin within window (usecs)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) - hwlat_detector/window - amount of time between (width) runs (usecs)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) The hwlat detector's kernel thread will migrate across each CPU specified in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) tracing_cpumask between each window. To limit the migration, either modify
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) tracing_cpumask, or modify the hwlat kernel thread (named [hwlatd]) CPU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) affinity directly, and the migration will stop.
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags