^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) ==================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) Kernel Lock Torture Test Operation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ==================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) CONFIG_LOCK_TORTURE_TEST
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) ========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) The CONFIG LOCK_TORTURE_TEST config option provides a kernel module
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) that runs torture tests on core kernel locking primitives. The kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) module, 'locktorture', may be built after the fact on the running
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) kernel to be tested, if desired. The tests periodically output status
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) messages via printk(), which can be examined via the dmesg (perhaps
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) grepping for "torture"). The test is started when the module is loaded,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) and stops when the module is unloaded. This program is based on how RCU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) is tortured, via rcutorture.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) This torture test consists of creating a number of kernel threads which
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) acquire the lock and hold it for specific amount of time, thus simulating
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) different critical region behaviors. The amount of contention on the lock
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) can be simulated by either enlarging this critical region hold time and/or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) creating more kthreads.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) Module Parameters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) =================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) This module has the following parameters:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) Locktorture-specific
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) --------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) nwriters_stress
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) Number of kernel threads that will stress exclusive lock
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) ownership (writers). The default value is twice the number
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) of online CPUs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) nreaders_stress
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) Number of kernel threads that will stress shared lock
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) ownership (readers). The default is the same amount of writer
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) locks. If the user did not specify nwriters_stress, then
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) both readers and writers be the amount of online CPUs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) torture_type
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) Type of lock to torture. By default, only spinlocks will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) be tortured. This module can torture the following locks,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) with string values as follows:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) - "lock_busted":
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) Simulates a buggy lock implementation.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) - "spin_lock":
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) spin_lock() and spin_unlock() pairs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) - "spin_lock_irq":
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) spin_lock_irq() and spin_unlock_irq() pairs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) - "rw_lock":
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) read/write lock() and unlock() rwlock pairs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) - "rw_lock_irq":
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) read/write lock_irq() and unlock_irq()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) rwlock pairs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) - "mutex_lock":
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) mutex_lock() and mutex_unlock() pairs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) - "rtmutex_lock":
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) rtmutex_lock() and rtmutex_unlock() pairs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) Kernel must have CONFIG_RT_MUTEX=y.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) - "rwsem_lock":
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) read/write down() and up() semaphore pairs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) Torture-framework (RCU + locking)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) ---------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) shutdown_secs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) The number of seconds to run the test before terminating
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) the test and powering off the system. The default is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) zero, which disables test termination and system shutdown.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) This capability is useful for automated testing.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) onoff_interval
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) The number of seconds between each attempt to execute a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) randomly selected CPU-hotplug operation. Defaults
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) to zero, which disables CPU hotplugging. In
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) CONFIG_HOTPLUG_CPU=n kernels, locktorture will silently
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) refuse to do any CPU-hotplug operations regardless of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) what value is specified for onoff_interval.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) onoff_holdoff
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) The number of seconds to wait until starting CPU-hotplug
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) operations. This would normally only be used when
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) locktorture was built into the kernel and started
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) automatically at boot time, in which case it is useful
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) in order to avoid confusing boot-time code with CPUs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) coming and going. This parameter is only useful if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) CONFIG_HOTPLUG_CPU is enabled.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) stat_interval
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) Number of seconds between statistics-related printk()s.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) By default, locktorture will report stats every 60 seconds.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) Setting the interval to zero causes the statistics to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) be printed -only- when the module is unloaded.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) stutter
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) The length of time to run the test before pausing for this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) same period of time. Defaults to "stutter=5", so as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) to run and pause for (roughly) five-second intervals.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) Specifying "stutter=0" causes the test to run continuously
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) without pausing.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) shuffle_interval
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) The number of seconds to keep the test threads affinitied
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) to a particular subset of the CPUs, defaults to 3 seconds.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) Used in conjunction with test_no_idle_hz.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) verbose
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) Enable verbose debugging printing, via printk(). Enabled
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) by default. This extra information is mostly related to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) high-level errors and reports from the main 'torture'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) framework.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) Statistics
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) ==========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) Statistics are printed in the following format::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) spin_lock-torture: Writes: Total: 93746064 Max/Min: 0/0 Fail: 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) (A) (B) (C) (D) (E)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) (A): Lock type that is being tortured -- torture_type parameter.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) (B): Number of writer lock acquisitions. If dealing with a read/write
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) primitive a second "Reads" statistics line is printed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) (C): Number of times the lock was acquired.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) (D): Min and max number of times threads failed to acquire the lock.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) (E): true/false values if there were errors acquiring the lock. This should
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) -only- be positive if there is a bug in the locking primitive's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) implementation. Otherwise a lock should never fail (i.e., spin_lock()).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) Of course, the same applies for (C), above. A dummy example of this is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) the "lock_busted" type.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) =====
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) The following script may be used to torture locks::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) #!/bin/sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) modprobe locktorture
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) sleep 3600
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) rmmod locktorture
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) dmesg | grep torture:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) The output can be manually inspected for the error flag of "!!!".
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) One could of course create a more elaborate script that automatically
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) checked for such errors. The "rmmod" command forces a "SUCCESS",
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) "FAILURE", or "RCU_HOTPLUG" indication to be printk()ed. The first
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) two are self-explanatory, while the last indicates that while there
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) were no locking failures, CPU-hotplug problems were detected.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) Also see: Documentation/RCU/torture.rst
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags