^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) .. highlight:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) Debugging kernel and modules via gdb
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4) ====================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7) interfaces allow to debug the Linux kernel and its modules during runtime
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) using gdb. Gdb comes with a powerful scripting interface for python. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) kernel provides a collection of helper scripts that can simplify typical
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) kernel debugging steps. This is a short tutorial about how to enable and use
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) them. It focuses on QEMU/KVM virtual machines as target, but the examples can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) be transferred to the other gdb stubs as well.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) Requirements
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) ------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) - gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) for distributions)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) Setup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) -----
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) - Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) www.qemu.org for more details). For cross-development,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) https://landley.net/aboriginal/bin keeps a pool of machine images and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) toolchains that can be helpful to start from.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) - Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) CONFIG_FRAME_POINTER, keep it enabled.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) - Install that kernel on the guest, turn off KASLR if necessary by adding
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) "nokaslr" to the kernel command line.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) Alternatively, QEMU allows to boot the kernel directly using -kernel,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) -append, -initrd command line switches. This is generally only useful if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) you do not depend on modules. See QEMU documentation for more details on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) this mode. In this case, you should build the kernel with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) - Enable the gdb stub of QEMU/KVM, either
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) - at VM startup time by appending "-s" to the QEMU command line
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) - during runtime by issuing "gdbserver" from the QEMU monitor
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) console
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) - cd /path/to/linux-build
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) - Start gdb: gdb vmlinux
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) Note: Some distros may restrict auto-loading of gdb scripts to known safe
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) add-auto-load-safe-path /path/to/linux-build
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) to ~/.gdbinit. See gdb help for more details.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) - Attach to the booted guest::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) (gdb) target remote :1234
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) Examples of using the Linux-provided gdb helpers
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) ------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) - Load module (and main kernel) symbols::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) (gdb) lx-symbols
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) loading vmlinux
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) scanning for modules in /home/user/linux/build
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) ...
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) - Set a breakpoint on some not yet loaded module function, e.g.::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) (gdb) b btrfs_init_sysfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) Function "btrfs_init_sysfs" not defined.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) Make breakpoint pending on future shared library load? (y or [n]) y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) Breakpoint 1 (btrfs_init_sysfs) pending.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) - Continue the target::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) (gdb) c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) - Load the module on the target and watch the symbols being loaded as well as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) the breakpoint hit::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) - Dump the log buffer of the target kernel::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) (gdb) lx-dmesg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) [ 0.000000] Initializing cgroup subsys cpuset
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) [ 0.000000] Initializing cgroup subsys cpu
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) [ 0.000000] e820: BIOS-provided physical RAM map:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) ....
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) - Examine fields of the current task struct::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) (gdb) p $lx_current().pid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) $1 = 4998
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) (gdb) p $lx_current().comm
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) $2 = "modprobe\000\000\000\000\000\000\000"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) - Make use of the per-cpu function for the current or a specified CPU::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) (gdb) p $lx_per_cpu("runqueues").nr_running
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) $3 = 1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) (gdb) p $lx_per_cpu("runqueues", 2).nr_running
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) $4 = 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) - Dig into hrtimers using the container_of helper::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) (gdb) p *$container_of($next, "struct hrtimer", "node")
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) $5 = {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) node = {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) node = {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) __rb_parent_color = 18446612133355256072,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) rb_right = 0x0 <irq_stack_union>,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) rb_left = 0x0 <irq_stack_union>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) },
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) expires = {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) tv64 = 1835268000000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) },
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) _softexpires = {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) tv64 = 1835268000000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) },
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) function = 0xffffffff81078232 <tick_sched_timer>,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) base = 0xffff88003fd0d6f0,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) state = 1,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) start_pid = 0,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) start_comm = "swapper/2\000\000\000\000\000\000"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) List of commands and functions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) ------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) The number of commands and convenience functions may evolve over the time,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) this is just a snapshot of the initial version::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) (gdb) apropos lx
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) function lx_current -- Return current task
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) function lx_module -- Find module by name and return the module variable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) function lx_per_cpu -- Return per-cpu variable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) function lx_thread_info -- Calculate Linux thread_info from task variable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) lx-dmesg -- Print Linux kernel log buffer
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) lx-lsmod -- List currently loaded modules
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) Detailed help can be obtained via "help <command-name>" for commands and "help
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) function <function-name>" for convenience functions.
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags