Orange Pi5 kernel

^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   1) ===================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   2) Testing BPF on s390
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   3) ===================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   4) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   5) 1. Introduction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   6) ***************
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   7) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   8) IBM Z are mainframe computers, which are descendants of IBM System/360 from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   9) year 1964. They are supported by the Linux kernel under the name "s390". This
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  10) document describes how to test BPF in an s390 QEMU guest.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  11) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  12) 2. One-time setup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  13) *****************
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  14) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  15) The following is required to build and run the test suite:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  16) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  17)   * s390 GCC
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  18)   * s390 development headers and libraries
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  19)   * Clang with BPF support
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  20)   * QEMU with s390 support
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  21)   * Disk image with s390 rootfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  22) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  23) Debian supports installing compiler and libraries for s390 out of the box.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  24) Users of other distros may use debootstrap in order to set up a Debian chroot::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  25) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  26)   sudo debootstrap \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  27)     --variant=minbase \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  28)     --include=sudo \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  29)     testing \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  30)     ./s390-toolchain
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  31)   sudo mount --rbind /dev ./s390-toolchain/dev
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  32)   sudo mount --rbind /proc ./s390-toolchain/proc
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  33)   sudo mount --rbind /sys ./s390-toolchain/sys
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  34)   sudo chroot ./s390-toolchain
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  35) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  36) Once on Debian, the build prerequisites can be installed as follows::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  37) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  38)   sudo dpkg --add-architecture s390x
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  39)   sudo apt-get update
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  40)   sudo apt-get install \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  41)     bc \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  42)     bison \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  43)     cmake \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  44)     debootstrap \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  45)     dwarves \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  46)     flex \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  47)     g++ \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  48)     gcc \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  49)     g++-s390x-linux-gnu \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  50)     gcc-s390x-linux-gnu \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  51)     gdb-multiarch \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  52)     git \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  53)     make \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  54)     python3 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  55)     qemu-system-misc \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  56)     qemu-utils \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  57)     rsync \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  58)     libcap-dev:s390x \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  59)     libelf-dev:s390x \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  60)     libncurses-dev
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  61) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  62) Latest Clang targeting BPF can be installed as follows::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  63) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  64)   git clone https://github.com/llvm/llvm-project.git
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  65)   ln -s ../../clang llvm-project/llvm/tools/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  66)   mkdir llvm-project-build
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  67)   cd llvm-project-build
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  68)   cmake \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  69)     -DLLVM_TARGETS_TO_BUILD=BPF \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  70)     -DCMAKE_BUILD_TYPE=Release \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  71)     -DCMAKE_INSTALL_PREFIX=/opt/clang-bpf \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  72)     ../llvm-project/llvm
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  73)   make
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  74)   sudo make install
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  75)   export PATH=/opt/clang-bpf/bin:$PATH
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  76) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  77) The disk image can be prepared using a loopback mount and debootstrap::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  78) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  79)   qemu-img create -f raw ./s390.img 1G
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  80)   sudo losetup -f ./s390.img
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  81)   sudo mkfs.ext4 /dev/loopX
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  82)   mkdir ./s390.rootfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  83)   sudo mount /dev/loopX ./s390.rootfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  84)   sudo debootstrap \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  85)     --foreign \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  86)     --arch=s390x \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  87)     --variant=minbase \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  88)     --include=" \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  89)       iproute2, \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  90)       iputils-ping, \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  91)       isc-dhcp-client, \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  92)       kmod, \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  93)       libcap2, \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  94)       libelf1, \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  95)       netcat, \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  96)       procps" \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  97)     testing \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  98)     ./s390.rootfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  99)   sudo umount ./s390.rootfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100)   sudo losetup -d /dev/loopX
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) 3. Compilation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) **************
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) In addition to the usual Kconfig options required to run the BPF test suite, it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) is also helpful to select::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108)   CONFIG_NET_9P=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109)   CONFIG_9P_FS=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110)   CONFIG_NET_9P_VIRTIO=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111)   CONFIG_VIRTIO_PCI=y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) as that would enable a very easy way to share files with the s390 virtual
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) machine.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) Compiling kernel, modules and testsuite, as well as preparing gdb scripts to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) simplify debugging, can be done using the following commands::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119)   make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- menuconfig
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120)   make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- bzImage modules scripts_gdb
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121)   make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122)     -C tools/testing/selftests \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123)     TARGETS=bpf \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)     INSTALL_PATH=$PWD/tools/testing/selftests/kselftest_install \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125)     install
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) 4. Running the test suite
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) *************************
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) The virtual machine can be started as follows::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)   qemu-system-s390x \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133)     -cpu max,zpci=on \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134)     -smp 2 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135)     -m 4G \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136)     -kernel linux/arch/s390/boot/compressed/vmlinux \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137)     -drive file=./s390.img,if=virtio,format=raw \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138)     -nographic \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139)     -append 'root=/dev/vda rw console=ttyS1' \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140)     -virtfs local,path=./linux,security_model=none,mount_tag=linux \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141)     -object rng-random,filename=/dev/urandom,id=rng0 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142)     -device virtio-rng-ccw,rng=rng0 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143)     -netdev user,id=net0 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144)     -device virtio-net-ccw,netdev=net0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) When using this on a real IBM Z, ``-enable-kvm`` may be added for better
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) performance. When starting the virtual machine for the first time, disk image
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) setup must be finalized using the following command::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150)   /debootstrap/debootstrap --second-stage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) Directory with the code built on the host as well as ``/proc`` and ``/sys``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) need to be mounted as follows::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155)   mkdir -p /linux
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156)   mount -t 9p linux /linux
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157)   mount -t proc proc /proc
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158)   mount -t sysfs sys /sys
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) After that, the test suite can be run using the following commands::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162)   cd /linux/tools/testing/selftests/kselftest_install
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163)   ./run_kselftest.sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) As usual, tests can be also run individually::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167)   cd /linux/tools/testing/selftests/bpf
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168)   ./test_verifier
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) 5. Debugging
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) ************
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) It is possible to debug the s390 kernel using QEMU GDB stub, which is activated
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) by passing ``-s`` to QEMU.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) It is preferable to turn KASLR off, so that gdb would know where to find the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) kernel image in memory, by building the kernel with::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179)   RANDOMIZE_BASE=n
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) GDB can then be attached using the following command::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183)   gdb-multiarch -ex 'target remote localhost:1234' ./vmlinux
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) 6. Network
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) **********
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) In case one needs to use the network in the virtual machine in order to e.g.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) install additional packages, it can be configured using::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191)   dhclient eth0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) 7. Links
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) ********
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) This document is a compilation of techniques, whose more comprehensive
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) descriptions can be found by following these links:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) - `Debootstrap <https://wiki.debian.org/EmDebian/CrossDebootstrap>`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) - `Multiarch <https://wiki.debian.org/Multiarch/HOWTO>`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) - `Building LLVM <https://llvm.org/docs/CMake.html>`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) - `Cross-compiling the kernel <https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Cross-compiling_the_kernel>`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) - `QEMU s390x Guest Support <https://wiki.qemu.org/Documentation/Platforms/S390X>`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) - `Plan 9 folder sharing over Virtio <https://wiki.qemu.org/Documentation/9psetup>`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) - `Using GDB with QEMU <https://wiki.osdev.org/Kernel_Debugging#Use_GDB_with_QEMU>`_