Orange Pi5 kernel

^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  1) ===============================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  2) Softlockup detector and hardlockup detector (aka nmi_watchdog)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  3) ===============================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  4) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  5) The Linux kernel can act as a watchdog to detect both soft and hard
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  6) lockups.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  7) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  8) A 'softlockup' is defined as a bug that causes the kernel to loop in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  9) kernel mode for more than 20 seconds (see "Implementation" below for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) details), without giving other tasks a chance to run. The current
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) stack trace is displayed upon detection and, by default, the system
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) will stay locked up. Alternatively, the kernel can be configured to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) panic; a sysctl, "kernel.softlockup_panic", a kernel parameter,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) "softlockup_panic" (see "Documentation/admin-guide/kernel-parameters.rst" for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) details), and a compile option, "BOOTPARAM_SOFTLOCKUP_PANIC", are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) provided for this.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) A 'hardlockup' is defined as a bug that causes the CPU to loop in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) kernel mode for more than 10 seconds (see "Implementation" below for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) details), without letting other interrupts have a chance to run.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) Similarly to the softlockup case, the current stack trace is displayed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) upon detection and the system will stay locked up unless the default
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) behavior is changed, which can be done through a sysctl,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) 'hardlockup_panic', a compile time knob, "BOOTPARAM_HARDLOCKUP_PANIC",
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) and a kernel parameter, "nmi_watchdog"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) (see "Documentation/admin-guide/kernel-parameters.rst" for details).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) The panic option can be used in combination with panic_timeout (this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) timeout is set through the confusingly named "kernel.panic" sysctl),
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) to cause the system to reboot automatically after a specified amount
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) of time.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) Implementation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) ==============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) The soft and hard lockup detectors are built on top of the hrtimer and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) perf subsystems, respectively. A direct consequence of this is that,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) in principle, they should work in any architecture where these
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) subsystems are present.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) A periodic hrtimer runs to generate interrupts and kick the watchdog
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) task. An NMI perf event is generated every "watchdog_thresh"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) (compile-time initialized to 10 and configurable through sysctl of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) same name) seconds to check for hardlockups. If any CPU in the system
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) does not receive any hrtimer interrupt during that time the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) 'hardlockup detector' (the handler for the NMI perf event) will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) generate a kernel warning or call panic, depending on the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) configuration.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) The watchdog task is a high priority kernel thread that updates a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) timestamp every time it is scheduled. If that timestamp is not updated
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) for 2*watchdog_thresh seconds (the softlockup threshold) the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) 'softlockup detector' (coded inside the hrtimer callback function)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) will dump useful debug information to the system log, after which it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) will call panic if it was instructed to do so or resume execution of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) other kernel code.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) The period of the hrtimer is 2*watchdog_thresh/5, which means it has
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) two or three chances to generate an interrupt before the hardlockup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) detector kicks in.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) As explained above, a kernel knob is provided that allows
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) administrators to configure the period of the hrtimer and the perf
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) event. The right value for a particular environment is a trade-off
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) between fast response to lockups and detection overhead.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) By default, the watchdog runs on all online cores.  However, on a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) kernel configured with NO_HZ_FULL, by default the watchdog runs only
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) on the housekeeping cores, not the cores specified in the "nohz_full"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) boot argument.  If we allowed the watchdog to run by default on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) the "nohz_full" cores, we would have to run timer ticks to activate
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) the scheduler, which would prevent the "nohz_full" functionality
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) from protecting the user code on those cores from the kernel.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) Of course, disabling it by default on the nohz_full cores means that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) when those cores do enter the kernel, by default we will not be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) able to detect if they lock up.  However, allowing the watchdog
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) to continue to run on the housekeeping (non-tickless) cores means
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) that we will continue to detect lockups properly on those cores.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) In either case, the set of cores excluded from running the watchdog
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) may be adjusted via the kernel.watchdog_cpumask sysctl.  For
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) nohz_full cores, this may be useful for debugging a case where the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) kernel seems to be hanging on the nohz_full cores.