^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) _
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) _ __ _ __ ___ __ _ _ __ __ _ _ __ | |__
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) | '_ \| '_ ` _ \ _____ / _` | '__/ _` | '_ \| '_ \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4) | |_) | | | | | |_____| (_| | | | (_| | |_) | | | |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) | .__/|_| |_| |_| \__, |_| \__,_| .__/|_| |_|
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) |_| |___/ |_|
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) pm-graph: suspend/resume/boot timing analysis tools
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) Version: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) Author: Todd Brandt <todd.e.brandt@intel.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) Home Page: https://01.org/pm-graph
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) Report bugs/issues at bugzilla.kernel.org Tools/pm-graph
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) - https://bugzilla.kernel.org/buglist.cgi?component=pm-graph&product=Tools
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) Full documentation available online & in man pages
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) - Getting Started:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) https://01.org/pm-graph/documentation/getting-started
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) - Config File Format:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) https://01.org/pm-graph/documentation/3-config-file-format
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) - upstream version in git:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) https://github.com/intel/pm-graph/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) Table of Contents
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) - Overview
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) - Setup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) - Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) - Basic Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) - Dev Mode Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) - Proc Mode Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) - Endurance Testing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) - Usage Examples
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) - Configuration Files
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) - Usage Examples
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) - Config File Options
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) - Custom Timeline Entries
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) - Adding/Editing Timeline Functions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) - Adding/Editing Dev Timeline Source Functions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) - Verifying your Custom Functions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) - Testing on consumer linux Operating Systems
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) - Android
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) | OVERVIEW |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) This tool suite is designed to assist kernel and OS developers in optimizing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) their linux stack's suspend/resume & boot time. Using a kernel image built
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) with a few extra options enabled, the tools will execute a suspend or boot,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) and will capture dmesg and ftrace data. This data is transformed into a set of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) timelines and a callgraph to give a quick and detailed view of which devices
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) and kernel processes are taking the most time in suspend/resume & boot.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) | SETUP |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) Package Requirements
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) - runs with python2 or python3, choice is made by /usr/bin/python link
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) - python
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) - python-configparser (for python2 sleepgraph)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) - python-requests (for googlesheet.py)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) - linux-tools-common (for turbostat usage in sleepgraph)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) Ubuntu:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) sudo apt-get install python python-configparser python-requests linux-tools-common
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) Fedora:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) sudo dnf install python python-configparser python-requests linux-tools-common
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) The tools can most easily be installed via git clone and make install
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) $> git clone http://github.com/intel/pm-graph.git
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) $> cd pm-graph
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) $> sudo make install
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) $> man sleepgraph ; man bootgraph
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) Setup involves some minor kernel configuration
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) The following kernel build options are required for all kernels:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) CONFIG_DEVMEM=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) CONFIG_PM_DEBUG=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) CONFIG_PM_SLEEP_DEBUG=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) CONFIG_FTRACE=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) CONFIG_FUNCTION_TRACER=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) CONFIG_FUNCTION_GRAPH_TRACER=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) CONFIG_KPROBES=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) CONFIG_KPROBES_ON_FTRACE=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) In kernel 3.15.0, two patches were upstreamed which enable the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) v3.0 behavior. These patches allow the tool to read all the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) data from trace events instead of from dmesg. You can enable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) this behavior on earlier kernels with these patches:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) (kernel/pre-3.15/enable_trace_events_suspend_resume.patch)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) (kernel/pre-3.15/enable_trace_events_device_pm_callback.patch)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) If you're using a kernel older than 3.15.0, the following
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) additional kernel parameters are required:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) (e.g. in file /etc/default/grub)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) GRUB_CMDLINE_LINUX_DEFAULT="... initcall_debug log_buf_len=32M ..."
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) If you're using a kernel older than 3.11-rc2, the following simple
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) patch must be applied to enable ftrace data:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) in file: kernel/power/suspend.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) in function: int suspend_devices_and_enter(suspend_state_t state)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) remove call to "ftrace_stop();"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) remove call to "ftrace_start();"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) There is a patch which does this for kernel v3.8.0:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) (kernel/pre-3.11-rc2/enable_ftrace_in_suspendresume.patch)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) | USAGE |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) Basic Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) ___________
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) 1) First configure a kernel using the instructions from the previous sections.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) Then build, install, and boot with it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) 2) Open up a terminal window and execute the mode list command:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) %> sudo ./sleepgraph.py -modes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) ['freeze', 'mem', 'disk']
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) Execute a test using one of the available power modes, e.g. mem (S3):
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) %> sudo ./sleepgraph.py -m mem -rtcwake 15
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) or with a config file
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) %> sudo ./sleepgraph.py -config config/suspend.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) When the system comes back you'll see the script finishing up and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) creating the output files in the test subdir. It generates output
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) files in subdirectory: suspend-mmddyy-HHMMSS. The ftrace file can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) be used to regenerate the html timeline with different options
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) HTML output: <hostname>_<mode>.html
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) raw dmesg output: <hostname>_<mode>_dmesg.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) raw ftrace output: <hostname>_<mode>_ftrace.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) View the html in firefox or chrome.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) Dev Mode Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) ______________
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) Developer mode adds information on low level source calls to the timeline.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) The tool sets kprobes on all delay and mutex calls to see which devices
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) are waiting for something and when. It also sets a suite of kprobes on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) subsystem dependent calls to better fill out the timeline.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) The tool will also expose kernel threads that don't normally show up in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) timeline. This is useful in discovering dependent threads to get a better
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) idea of what each device is waiting for. For instance, the scsi_eh thread,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) a.k.a. scsi resume error handler, is what each SATA disk device waits for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) before it can continue resume.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) The timeline will be much larger if run with dev mode, so it can be useful
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) to set the -mindev option to clip out any device blocks that are too small
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) to see easily. The following command will give a nice dev mode run:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) %> sudo ./sleepgraph.py -m mem -rtcwake 15 -mindev 1 -dev
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) or with a config file
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) %> sudo ./sleepgraph.py -config config/suspend-dev.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) Proc Mode Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) _______________
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) Proc mode adds user process info to the timeline. This is done in a manner
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) similar to the bootchart utility, which graphs init processes and their
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) execution as the system boots. This tool option does the same thing but for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) the period before and after suspend/resume.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) In order to see any process info, there needs to be some delay before or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) after resume since processes are frozen in suspend_prepare and thawed in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) resume_complete. The predelay and postdelay args allow you to do this. It
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) can also be useful to run in x2 mode with an x2 delay, this way you can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) see process activity before and after resume, and in between two
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) successive suspend/resumes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) The command can be run like this:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) %> sudo ./sleepgraph.py -m mem -rtcwake 15 -x2 -x2delay 1000 -predelay 1000 -postdelay 1000 -proc
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) or with a config file
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) %> sudo ./sleepgraph.py -config config/suspend-proc.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) | ENDURANCE TESTING |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) The best way to gauge the health of a system is to run a series of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) suspend/resumes over an extended period and analyze the behavior. This can be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) accomplished with sleepgraph's -multi argument. You specify two numbers: the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) number of tests to run OR the duration in days, hours, or minutes, and the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) delay in seconds between them. For instance, -multi 20 5: execute 20 tests with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) a 5 second delay between each, or -multi 24h 0: execute tests over a 24 hour
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209) period with no delay between tests. You can include any other options you like
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) to generate the data you want. It's most useful to collect dev mode timelines
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) as the kprobes don't alter the performance much and you get more insight.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) On completion, the output folder contains a series of folders for the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) individual test data and a set of summary pages in the root. The summary.html
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215) file is a tabular list of the tests with relevant info and links. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) summary-issue.html and summary-devices.html files include data taken from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) all tests on kernel issues and device performance. The folder looks like this:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) suspend-xN-{date}-{time}:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220) summary.html
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) summary-issues.html
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) summary-devices.html
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) suspend-{date}-{time} (1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224) suspend-{date}-{time} (2)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) ...
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) These are the relevant arguments to use for testing:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) -m mode
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) Mode to initiate for suspend e.g. mem, freeze, standby (default: mem).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) -rtcwake t
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233) Use rtcwake to autoresume after t seconds (default: 15).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) -gzip (optional)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236) Gzip the trace and dmesg logs to save space. The tool can also read in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237) gzipped logs for processing. This reduces the multitest folder size.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) -dev (optional)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240) Add kernel source calls and threads to the timeline (default: disabled).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) -multi n d
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) Execute n consecutive tests at d seconds intervals. The outputs will be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) created in a new subdirectory: suspend-xN-{date}-{time}. When the multitest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245) run is done, the -summary command is called automatically to create summary
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246) html files for all the data (unless you use -skiphtml). -skiphtml will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) speed up the testing by not creating timelines or summary html files. You
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) can then run the tool again at a later time with -summary and -genhtml to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249) create the timelines.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) -skiphtml (optional)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252) Run the test and capture the trace logs, but skip the timeline and summary
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) html generation. This can greatly speed up overall testing. You can then
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) copy the data to a faster host machine and run -summary -genhtml to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) generate the timelines and summary.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) These are the relevant commands to use after testing is complete:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) -summary indir
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260) Generate or regenerate the summary for a -multi test run. Creates three
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) files: summary.html, summary-issues.html, and summary-devices.html in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) current folder. summary.html is a table of tests with relevant info sorted
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263) by kernel/host/mode, and links to the test html files. summary-issues.html
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) is a list of kernel issues found in dmesg from all the tests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) summary-devices.html is a list of devices and times from all the tests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) -genhtml
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268) Used with -summary to regenerate any missing html timelines from their
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269) dmesg and ftrace logs. This will require a significant amount of time if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) there are thousands of tests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272) Usage Examples
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273) _______________
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) A multitest is initiated like this:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277) %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279) or you can skip timeline generation in order to speed things up
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281) %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0 -skiphtml
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283) The tool will produce an output folder with all the test subfolders inside.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284) Each test subfolder contains the dmesg/ftrace logs and/or the html timeline
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285) depending on whether you used the -skiphtml option. The root folder contains
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286) the summary.html files.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288) The summary for an existing multitest is generated like this:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) %> cd suspend-x2000-{date}-{time}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291) %> sleepgraph.py -summary .
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293) or if you need to generate the html timelines you can use -genhtml
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295) %> cd suspend-xN-{date}-{time}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296) %> sleepgraph.py -summary . -genhtml
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299) | CONFIGURATION FILES |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302) Since 4.0 we've moved to using config files in lieu of command line options.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303) The config folder contains a collection of typical use cases.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) There are corresponding configs for other power modes:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306) Simple suspend/resume with basic timeline (mem/freeze/standby)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307) config/suspend.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308) config/freeze.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309) config/standby.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311) Dev mode suspend/resume with dev timeline (mem/freeze/standby)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312) config/suspend-dev.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313) config/freeze-dev.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314) config/standby-dev.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) Simple suspend/resume with timeline and callgraph (mem/freeze/standby)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317) config/suspend-callgraph.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318) config/freeze-callgraph.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319) config/standby-callgraph.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321) Sample proc mode x2 run using mem suspend
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322) config/suspend-x2-proc.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324) Sample for editing timeline funcs (moves internal functions into config)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325) config/custom-timeline-functions.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327) Sample debug config for serio subsystem
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328) config/debug-serio-suspend.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331) Usage Examples
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332) ______________
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334) Run a simple mem suspend:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335) %> sudo ./sleepgraph.py -config config/suspend.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) Run a mem suspend with callgraph data:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338) %> sudo ./sleepgraph.py -config config/suspend-callgraph.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340) Run a mem suspend with dev mode detail:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341) %> sudo ./sleepgraph.py -config config/suspend-dev.cfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344) Config File Options
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345) ___________________
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347) [Settings]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349) # Verbosity: print verbose messages (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) verbose: false
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352) # Suspend Mode: e.g. standby, mem, freeze, disk (def: mem)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353) mode: mem
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355) # Output Directory Format: {hostname}, {date}, {time} give current values
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) output-dir: suspend-{hostname}-{date}-{time}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358) # Automatic Wakeup: use rtcwake to wakeup after X seconds (def: infinity)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359) rtcwake: 15
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361) # Add Logs: add the dmesg and ftrace log to the html output (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362) addlogs: false
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364) # Sus/Res Gap: insert a gap between sus & res in the timeline (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365) srgap: false
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367) # Custom Command: Command to execute in lieu of suspend (def: "")
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368) command: echo mem > /sys/power/state
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370) # Proc mode: graph user processes and cpu usage in the timeline (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) proc: false
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373) # Dev mode: graph source functions in the timeline (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374) dev: false
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376) # Suspend/Resume x2: run 2 suspend/resumes back to back (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377) x2: false
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379) # x2 Suspend Delay: time delay between the two test runs in ms (def: 0 ms)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 380) x2delay: 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 381)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 382) # Pre Suspend Delay: nclude an N ms delay before (1st) suspend (def: 0 ms)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 383) predelay: 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 384)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 385) # Post Resume Delay: include an N ms delay after (last) resume (def: 0 ms)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 386) postdelay: 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 387)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 388) # Min Device Length: graph only dev callbacks longer than min (def: 0.001 ms)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 389) mindev: 0.001
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 390)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 391) # Callgraph: gather ftrace callgraph data on all timeline events (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 392) callgraph: false
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 393)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 394) # Expand Callgraph: pre-expand the callgraph treeviews in html (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 395) expandcg: false
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 396)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 397) # Min Callgraph Length: show callgraphs only if longer than min (def: 1 ms)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 398) mincg: 1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 399)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 400) # Timestamp Precision: number of sig digits in timestamps (0:S, [3:ms], 6:us)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 401) timeprec: 3
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 402)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 403) # Device Filter: show only devs whose name/driver includes one of these strings
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 404) devicefilter: _cpu_up,_cpu_down,i915,usb
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 405)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 406) # Override default timeline entries:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 407) # Do not use the internal default functions for timeline entries (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 408) # Set this to true if you intend to only use the ones defined in the config
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 409) override-timeline-functions: true
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 410)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 411) # Override default dev timeline entries:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 412) # Do not use the internal default functions for dev timeline entries (def: false)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 413) # Set this to true if you intend to only use the ones defined in the config
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 414) override-dev-timeline-functions: true
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 415)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 416) # Call Loop Max Gap (dev mode only)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 417) # merge loops of the same call if each is less than maxgap apart (def: 100us)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 418) callloop-maxgap: 0.0001
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 419)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 420) # Call Loop Max Length (dev mode only)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 421) # merge loops of the same call if each is less than maxlen in length (def: 5ms)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 422) callloop-maxlen: 0.005
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 423)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 424) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 425) | CUSTOM TIMELINE ENTRIES |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 426) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 427)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 428) Adding or Editing Timeline Functions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 429) ____________________________________
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 430)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 431) The tool uses an array of function names to fill out empty spaces in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 432) timeline where device callbacks don't appear. For instance, in suspend_prepare
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 433) the tool adds the sys_sync and freeze_processes calls as virtual device blocks
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 434) in the timeline to show you where the time is going. These calls should fill
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 435) the timeline with contiguous data so that most kernel execution is covered.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 436)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 437) It is possible to add new function calls to the timeline by adding them to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 438) the config. It's also possible to copy the internal timeline functions into
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 439) the config so that you can override and edit them. Place them in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 440) timeline_functions_ARCH section with the name of your architecture appended.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 441) i.e. for x86_64: [timeline_functions_x86_64]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 442)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 443) Use the override-timeline-functions option if you only want to use your
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 444) custom calls, or leave it false to append them to the internal ones.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 445)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 446) This section includes a list of functions (set using kprobes) which use both
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 447) symbol data and function arg data. The args are pulled directly from the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 448) stack using this architecture's registers and stack formatting. Each entry
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 449) can include up to four pieces of info: The function name, a format string,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 450) an argument list, and a color. But only a function name is required.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 451)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 452) For a full example config, see config/custom-timeline-functions.cfg. It pulls
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 453) all the internal timeline functions into the config and allows you to edit
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 454) them.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 455)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 456) Entry format:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 457)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 458) function: format{fn_arg1}_{fn_arg2} fn_arg1 fn_arg2 ... [color=purple]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 459)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 460) Required Arguments:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 461)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 462) function: The symbol name for the function you want probed, this is the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 463) minimum required for an entry, it will show up as the function
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 464) name with no arguments.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 465)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 466) example: _cpu_up:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 467)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 468) Optional Arguments:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 469)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 470) format: The format to display the data on the timeline in. Use braces to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 471) enclose the arg names.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 472)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 473) example: CPU_ON[{cpu}]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 474)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 475) color: The color of the entry block in the timeline. The default color is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 476) transparent, so the entry shares the phase color. The color is an
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 477) html color string, either a word, or an RGB.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 478)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 479) example: [color=#CC00CC]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 480)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 481) arglist: A list of arguments from registers/stack addresses. See URL:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 482) https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 483)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 484) example: cpu=%di:s32
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 485)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 486) Here is a full example entry. It displays cpu resume calls in the timeline
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 487) in orange. They will appear as CPU_ON[0], CPU_ON[1], etc.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 488)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 489) [timeline_functions_x86_64]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 490) _cpu_up: CPU_ON[{cpu}] cpu=%di:s32 [color=orange]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 491)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 492)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 493) Adding or Editing Dev Mode Timeline Source Functions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 494) ____________________________________________________
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 495)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 496) In dev mode, the tool uses an array of function names to monitor source
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 497) execution within the timeline entries.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 498)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 499) The function calls are displayed inside the main device/call blocks in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 500) timeline. However, if a function call is not within a main timeline event,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 501) it will spawn an entirely new event named after the caller's kernel thread.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 502) These asynchronous kernel threads will populate in a separate section
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 503) beneath the main device/call section.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 504)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 505) The tool has a set of hard coded calls which focus on the most common use
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 506) cases: msleep, udelay, schedule_timeout, mutex_lock_slowpath, etc. These are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 507) the functions that add a hardcoded time delay to the suspend/resume path.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 508) The tool also includes some common functions native to important
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 509) subsystems: ata, i915, and ACPI, etc.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 510)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 511) It is possible to add new function calls to the dev timeline by adding them
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 512) to the config. It's also possible to copy the internal dev timeline
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 513) functions into the config so that you can override and edit them. Place them
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 514) in the dev_timeline_functions_ARCH section with the name of your architecture
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 515) appended. i.e. for x86_64: [dev_timeline_functions_x86_64]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 516)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 517) Use the override-dev-timeline-functions option if you only want to use your
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 518) custom calls, or leave it false to append them to the internal ones.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 519)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 520) The format is the same as the timeline_functions_x86_64 section. It's a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 521) list of functions (set using kprobes) which use both symbol data and function
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 522) arg data. The args are pulled directly from the stack using this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 523) architecture's registers and stack formatting. Each entry can include up
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 524) to four pieces of info: The function name, a format string, an argument list,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 525) and a color. But only the function name is required.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 526)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 527) For a full example config, see config/custom-timeline-functions.cfg. It pulls
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 528) all the internal dev timeline functions into the config and allows you to edit
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 529) them.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 530)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 531) Here is a full example entry. It displays the ATA port reset calls as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 532) ataN_port_reset in the timeline. This is where most of the SATA disk resume
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 533) time goes, so it can be helpful to see the low level call.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 534)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 535) [dev_timeline_functions_x86_64]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 536) ata_eh_recover: ata{port}_port_reset port=+36(%di):s32 [color=#CC00CC]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 537)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 538)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 539) Verifying your custom functions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 540) _______________________________
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 541)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 542) Once you have a set of functions (kprobes) defined, it can be useful to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 543) perform a quick check to see if you formatted them correctly and if the system
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 544) actually supports them. To do this, run the tool with your config file
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 545) and the -status option. The tool will go through all the kprobes (both
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 546) custom and internal if you haven't overridden them) and actually attempts
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 547) to set them in ftrace. It will then print out success or fail for you.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 548)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 549) Note that kprobes which don't actually exist in the kernel won't stop the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 550) tool, they just wont show up.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 551)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 552) For example:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 553)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 554) sudo ./sleepgraph.py -config config/custom-timeline-functions.cfg -status
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 555) Checking this system (myhostname)...
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 556) have root access: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 557) is sysfs mounted: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 558) is "mem" a valid power mode: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 559) is ftrace supported: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 560) are kprobes supported: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 561) timeline data source: FTRACE (all trace events found)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 562) is rtcwake supported: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 563) verifying timeline kprobes work:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 564) _cpu_down: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 565) _cpu_up: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 566) acpi_pm_finish: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 567) acpi_pm_prepare: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 568) freeze_kernel_threads: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 569) freeze_processes: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 570) sys_sync: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 571) thaw_processes: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 572) verifying dev kprobes work:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 573) __const_udelay: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 574) __mutex_lock_slowpath: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 575) acpi_os_stall: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 576) acpi_ps_parse_aml: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 577) intel_opregion_init: NO
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 578) intel_opregion_register: NO
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 579) intel_opregion_setup: NO
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 580) msleep: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 581) schedule_timeout: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 582) schedule_timeout_uninterruptible: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 583) usleep_range: YES
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 584)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 585)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 586) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 587) | TESTING ON CONSUMER LINUX OPERATING SYSTEMS |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 588) ------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 589)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 590) Android
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 591) _______
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 592)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 593) The easiest way to execute on an android device is to run the android.sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 594) script on the device, then pull the ftrace log back to the host and run
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 595) sleepgraph.py on it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 596)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 597) Here are the steps:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 598)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 599) [download and install the tool on the device]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 600)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 601) host%> wget https://raw.githubusercontent.com/intel/pm-graph/master/tools/android.sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 602) host%> adb connect 192.168.1.6
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 603) host%> adb root
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 604) # push the script to a writeable location
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 605) host%> adb push android.sh /sdcard/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 606)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 607) [check whether the tool will run on your device]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 608)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 609) host%> adb shell
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 610) dev%> cd /sdcard
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 611) dev%> sh android.sh status
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 612) host : asus_t100
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 613) kernel : 3.14.0-i386-dirty
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 614) modes : freeze mem
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 615) rtcwake : supported
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 616) ftrace : supported
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 617) trace events {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 618) suspend_resume: found
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 619) device_pm_callback_end: found
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 620) device_pm_callback_start: found
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 621) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 622) # the above is what you see on a system that's properly patched
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 623)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 624) [execute the suspend]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 625)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 626) # NOTE: The suspend will only work if the screen isn't timed out,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 627) # so you have to press some keys first to wake it up b4 suspend)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 628) dev%> sh android.sh suspend mem
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 629) ------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 630) Suspend/Resume timing test initiated
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 631) ------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 632) hostname : asus_t100
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 633) kernel : 3.14.0-i386-dirty
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 634) mode : mem
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 635) ftrace out : /mnt/shell/emulated/0/ftrace.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 636) dmesg out : /mnt/shell/emulated/0/dmesg.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 637) log file : /mnt/shell/emulated/0/log.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 638) ------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 639) INITIALIZING FTRACE........DONE
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 640) STARTING FTRACE
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 641) SUSPEND START @ 21:24:02 (rtcwake in 10 seconds)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 642) <adb connection will now terminate>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 643)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 644) [retrieve the data from the device]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 645)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 646) # I find that you have to actually kill the adb process and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 647) # reconnect sometimes in order for the connection to work post-suspend
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 648) host%> adb connect 192.168.1.6
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 649) # (required) get the ftrace data, this is the most important piece
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 650) host%> adb pull /sdcard/ftrace.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 651) # (optional) get the dmesg data, this is for debugging
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 652) host%> adb pull /sdcard/dmesg.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 653) # (optional) get the log, which just lists some test times for comparison
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 654) host%> adb pull /sdcard/log.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 655)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 656) [create an output html file using sleepgraph.py]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 657)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 658) host%> sleepgraph.py -ftrace ftrace.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 659)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 660) You should now have an output.html with the android data, enjoy!
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags