Orange Pi5 kernel

^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   1) .. SPDX-License-Identifier: GPL-2.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   2) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   3) ==========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   4) RCU Torture Test Operation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   5) ==========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   6) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   7) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   8) CONFIG_RCU_TORTURE_TEST
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   9) =======================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  10) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  11) The CONFIG_RCU_TORTURE_TEST config option is available for all RCU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  12) implementations.  It creates an rcutorture kernel module that can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  13) be loaded to run a torture test.  The test periodically outputs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  14) status messages via printk(), which can be examined via the dmesg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  15) command (perhaps grepping for "torture").  The test is started
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  16) when the module is loaded, and stops when the module is unloaded.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  17) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  18) Module parameters are prefixed by "rcutorture." in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  19) Documentation/admin-guide/kernel-parameters.txt.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  20) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  21) Output
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  22) ======
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  23) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  24) The statistics output is as follows::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  25) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  26) 	rcu-torture:--- Start of test: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  27) 	rcu-torture: rtc:           (null) ver: 155441 tfle: 0 rta: 155441 rtaf: 8884 rtf: 155440 rtmbe: 0 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 3055767
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  28) 	rcu-torture: Reader Pipe:  727860534 34213 0 0 0 0 0 0 0 0 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  29) 	rcu-torture: Reader Batch:  727877838 17003 0 0 0 0 0 0 0 0 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  30) 	rcu-torture: Free-Block Circulation:  155440 155440 155440 155440 155440 155440 155440 155440 155440 155440 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  31) 	rcu-torture:--- End of test: SUCCESS: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  32) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  33) The command "dmesg | grep torture:" will extract this information on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  34) most systems.  On more esoteric configurations, it may be necessary to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  35) use other commands to access the output of the printk()s used by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  36) the RCU torture test.  The printk()s use KERN_ALERT, so they should
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  37) be evident.  ;-)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  38) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  39) The first and last lines show the rcutorture module parameters, and the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  40) last line shows either "SUCCESS" or "FAILURE", based on rcutorture's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  41) automatic determination as to whether RCU operated correctly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  42) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  43) The entries are as follows:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  44) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  45) *	"rtc": The hexadecimal address of the structure currently visible
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  46) 	to readers.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  47) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  48) *	"ver": The number of times since boot that the RCU writer task
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  49) 	has changed the structure visible to readers.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  50) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  51) *	"tfle": If non-zero, indicates that the "torture freelist"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  52) 	containing structures to be placed into the "rtc" area is empty.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  53) 	This condition is important, since it can fool you into thinking
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  54) 	that RCU is working when it is not.  :-/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  55) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  56) *	"rta": Number of structures allocated from the torture freelist.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  57) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  58) *	"rtaf": Number of allocations from the torture freelist that have
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  59) 	failed due to the list being empty.  It is not unusual for this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  60) 	to be non-zero, but it is bad for it to be a large fraction of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  61) 	the value indicated by "rta".
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  62) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  63) *	"rtf": Number of frees into the torture freelist.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  64) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  65) *	"rtmbe": A non-zero value indicates that rcutorture believes that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  66) 	rcu_assign_pointer() and rcu_dereference() are not working
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  67) 	correctly.  This value should be zero.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  68) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  69) *	"rtbe": A non-zero value indicates that one of the rcu_barrier()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  70) 	family of functions is not working correctly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  71) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  72) *	"rtbke": rcutorture was unable to create the real-time kthreads
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  73) 	used to force RCU priority inversion.  This value should be zero.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  74) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  75) *	"rtbre": Although rcutorture successfully created the kthreads
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  76) 	used to force RCU priority inversion, it was unable to set them
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  77) 	to the real-time priority level of 1.  This value should be zero.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  78) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  79) *	"rtbf": The number of times that RCU priority boosting failed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  80) 	to resolve RCU priority inversion.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  81) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  82) *	"rtb": The number of times that rcutorture attempted to force
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  83) 	an RCU priority inversion condition.  If you are testing RCU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  84) 	priority boosting via the "test_boost" module parameter, this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  85) 	value should be non-zero.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  86) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  87) *	"nt": The number of times rcutorture ran RCU read-side code from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  88) 	within a timer handler.  This value should be non-zero only
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  89) 	if you specified the "irqreader" module parameter.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  90) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  91) *	"Reader Pipe": Histogram of "ages" of structures seen by readers.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  92) 	If any entries past the first two are non-zero, RCU is broken.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  93) 	And rcutorture prints the error flag string "!!!" to make sure
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  94) 	you notice.  The age of a newly allocated structure is zero,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  95) 	it becomes one when removed from reader visibility, and is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  96) 	incremented once per grace period subsequently -- and is freed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  97) 	after passing through (RCU_TORTURE_PIPE_LEN-2) grace periods.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  98) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  99) 	The output displayed above was taken from a correctly working
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) 	RCU.  If you want to see what it looks like when broken, break
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) 	it yourself.  ;-)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) *	"Reader Batch": Another histogram of "ages" of structures seen
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) 	by readers, but in terms of counter flips (or batches) rather
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) 	than in terms of grace periods.  The legal number of non-zero
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) 	entries is again two.  The reason for this separate view is that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) 	it is sometimes easier to get the third entry to show up in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) 	"Reader Batch" list than in the "Reader Pipe" list.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) *	"Free-Block Circulation": Shows the number of torture structures
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) 	that have reached a given point in the pipeline.  The first element
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) 	should closely correspond to the number of structures allocated,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) 	the second to the number that have been removed from reader view,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) 	and all but the last remaining to the corresponding number of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) 	passes through a grace period.  The last entry should be zero,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) 	as it is only incremented if a torture structure's counter
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) 	somehow gets incremented farther than it should.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) Different implementations of RCU can provide implementation-specific
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) additional information.  For example, Tree SRCU provides the following
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) additional line::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) 	srcud-torture: Tree SRCU per-CPU(idx=0): 0(35,-21) 1(-4,24) 2(1,1) 3(-26,20) 4(28,-47) 5(-9,4) 6(-10,14) 7(-14,11) T(1,6)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) This line shows the per-CPU counter state, in this case for Tree SRCU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) using a dynamically allocated srcu_struct (hence "srcud-" rather than
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) "srcu-").  The numbers in parentheses are the values of the "old" and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) "current" counters for the corresponding CPU.  The "idx" value maps the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) "old" and "current" values to the underlying array, and is useful for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) debugging.  The final "T" entry contains the totals of the counters.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) Usage on Specific Kernel Builds
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) ===============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) It is sometimes desirable to torture RCU on a specific kernel build,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) for example, when preparing to put that kernel build into production.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) In that case, the kernel should be built with CONFIG_RCU_TORTURE_TEST=m
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) so that the test can be started using modprobe and terminated using rmmod.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) For example, the following script may be used to torture RCU::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) 	#!/bin/sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) 	modprobe rcutorture
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) 	sleep 3600
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) 	rmmod rcutorture
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) 	dmesg | grep torture:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) The output can be manually inspected for the error flag of "!!!".
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) One could of course create a more elaborate script that automatically
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) checked for such errors.  The "rmmod" command forces a "SUCCESS",
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) "FAILURE", or "RCU_HOTPLUG" indication to be printk()ed.  The first
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) two are self-explanatory, while the last indicates that while there
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) were no RCU failures, CPU-hotplug problems were detected.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) Usage on Mainline Kernels
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) =========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) When using rcutorture to test changes to RCU itself, it is often
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) necessary to build a number of kernels in order to test that change
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) across a broad range of combinations of the relevant Kconfig options
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) and of the relevant kernel boot parameters.  In this situation, use
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) of modprobe and rmmod can be quite time-consuming and error-prone.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) Therefore, the tools/testing/selftests/rcutorture/bin/kvm.sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) script is available for mainline testing for x86, arm64, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) powerpc.  By default, it will run the series of tests specified by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) tools/testing/selftests/rcutorture/configs/rcu/CFLIST, with each test
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) running for 30 minutes within a guest OS using a minimal userspace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) supplied by an automatically generated initrd.  After the tests are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) complete, the resulting build products and console output are analyzed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) for errors and the results of the runs are summarized.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) On larger systems, rcutorture testing can be accelerated by passing the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) --cpus argument to kvm.sh.  For example, on a 64-CPU system, "--cpus 43"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) would use up to 43 CPUs to run tests concurrently, which as of v5.4 would
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) complete all the scenarios in two batches, reducing the time to complete
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) from about eight hours to about one hour (not counting the time to build
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) the sixteen kernels).  The "--dryrun sched" argument will not run tests,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) but rather tell you how the tests would be scheduled into batches.  This
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) can be useful when working out how many CPUs to specify in the --cpus
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) argument.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) Not all changes require that all scenarios be run.  For example, a change
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) to Tree SRCU might run only the SRCU-N and SRCU-P scenarios using the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) --configs argument to kvm.sh as follows:  "--configs 'SRCU-N SRCU-P'".
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) Large systems can run multiple copies of of the full set of scenarios,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) for example, a system with 448 hardware threads can run five instances
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) of the full set concurrently.  To make this happen::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) 	kvm.sh --cpus 448 --configs '5*CFLIST'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) Alternatively, such a system can run 56 concurrent instances of a single
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) eight-CPU scenario::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) 	kvm.sh --cpus 448 --configs '56*TREE04'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) Or 28 concurrent instances of each of two eight-CPU scenarios::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) 	kvm.sh --cpus 448 --configs '28*TREE03 28*TREE04'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) Of course, each concurrent instance will use memory, which can be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) limited using the --memory argument, which defaults to 512M.  Small
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) values for memory may require disabling the callback-flooding tests
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) using the --bootargs parameter discussed below.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) Sometimes additional debugging is useful, and in such cases the --kconfig
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209) parameter to kvm.sh may be used, for example, ``--kconfig 'CONFIG_KASAN=y'``.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) Kernel boot arguments can also be supplied, for example, to control
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212) rcutorture's module parameters.  For example, to test a change to RCU's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) CPU stall-warning code, use "--bootargs 'rcutorture.stall_cpu=30'".
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) This will of course result in the scripting reporting a failure, namely
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215) the resuling RCU CPU stall warning.  As noted above, reducing memory may
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) require disabling rcutorture's callback-flooding tests::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218) 	kvm.sh --cpus 448 --configs '56*TREE04' --memory 128M \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) 		--bootargs 'rcutorture.fwd_progress=0'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) Sometimes all that is needed is a full set of kernel builds.  This is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) what the --buildonly argument does.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224) Finally, the --trust-make argument allows each kernel build to reuse what
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) it can from the previous kernel build.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) There are additional more arcane arguments that are documented in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228) source code of the kvm.sh script.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) If a run contains failures, the number of buildtime and runtime failures
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231) is listed at the end of the kvm.sh output, which you really should redirect
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) to a file.  The build products and console output of each run is kept in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233) tools/testing/selftests/rcutorture/res in timestamped directories.  A
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234) given directory can be supplied to kvm-find-errors.sh in order to have
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) it cycle you through summaries of errors and full error logs.  For example::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237) 	tools/testing/selftests/rcutorture/bin/kvm-find-errors.sh \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) 		tools/testing/selftests/rcutorture/res/2020.01.20-15.54.23
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240) However, it is often more convenient to access the files directly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241) Files pertaining to all scenarios in a run reside in the top-level
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) directory (2020.01.20-15.54.23 in the example above), while per-scenario
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) files reside in a subdirectory named after the scenario (for example,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) "TREE04").  If a given scenario ran more than once (as in "--configs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245) '56*TREE04'" above), the directories corresponding to the second and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246) subsequent runs of that scenario include a sequence number, for example,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) "TREE04.2", "TREE04.3", and so on.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249) The most frequently used file in the top-level directory is testid.txt.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) If the test ran in a git repository, then this file contains the commit
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) that was tested and any uncommitted changes in diff format.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) The most frequently used files in each per-scenario-run directory are:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) .config:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256) 	This file contains the Kconfig options.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258) Make.out:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) 	This contains build output for a specific scenario.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) console.log:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) 	This contains the console output for a specific scenario.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263) 	This file may be examined once the kernel has booted, but
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) 	it might not exist if the build failed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266) vmlinux:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) 	This contains the kernel, which can be useful with tools like
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268) 	objdump and gdb.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) A number of additional files are available, but are less frequently used.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271) Many are intended for debugging of rcutorture itself or of its scripting.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273) As of v5.4, a successful run with the default set of scenarios produces
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274) the following summary at the end of the run on a 12-CPU system::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276)     SRCU-N ------- 804233 GPs (148.932/s) [srcu: g10008272 f0x0 ]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277)     SRCU-P ------- 202320 GPs (37.4667/s) [srcud: g1809476 f0x0 ]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278)     SRCU-t ------- 1122086 GPs (207.794/s) [srcu: g0 f0x0 ]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279)     SRCU-u ------- 1111285 GPs (205.794/s) [srcud: g1 f0x0 ]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280)     TASKS01 ------- 19666 GPs (3.64185/s) [tasks: g0 f0x0 ]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281)     TASKS02 ------- 20541 GPs (3.80389/s) [tasks: g0 f0x0 ]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282)     TASKS03 ------- 19416 GPs (3.59556/s) [tasks: g0 f0x0 ]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283)     TINY01 ------- 836134 GPs (154.84/s) [rcu: g0 f0x0 ] n_max_cbs: 34198
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284)     TINY02 ------- 850371 GPs (157.476/s) [rcu: g0 f0x0 ] n_max_cbs: 2631
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285)     TREE01 ------- 162625 GPs (30.1157/s) [rcu: g1124169 f0x0 ]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286)     TREE02 ------- 333003 GPs (61.6672/s) [rcu: g2647753 f0x0 ] n_max_cbs: 35844
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287)     TREE03 ------- 306623 GPs (56.782/s) [rcu: g2975325 f0x0 ] n_max_cbs: 1496497
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288)     CPU count limited from 16 to 12
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289)     TREE04 ------- 246149 GPs (45.5831/s) [rcu: g1695737 f0x0 ] n_max_cbs: 434961
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290)     TREE05 ------- 314603 GPs (58.2598/s) [rcu: g2257741 f0x2 ] n_max_cbs: 193997
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291)     TREE07 ------- 167347 GPs (30.9902/s) [rcu: g1079021 f0x0 ] n_max_cbs: 478732
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292)     CPU count limited from 16 to 12
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293)     TREE09 ------- 752238 GPs (139.303/s) [rcu: g13075057 f0x0 ] n_max_cbs: 99011