Orange Pi5 kernel

^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  1) =====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  2) DAWR issues on POWER9
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  3) =====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  4) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  5) On POWER9 the Data Address Watchpoint Register (DAWR) can cause a checkstop
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  6) if it points to cache inhibited (CI) memory. Currently Linux has no way to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  7) disinguish CI memory when configuring the DAWR, so (for now) the DAWR is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  8) disabled by this commit::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  9) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10)     commit 9654153158d3e0684a1bdb76dbababdb7111d5a0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11)     Author: Michael Neuling <mikey@neuling.org>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12)     Date:   Tue Mar 27 15:37:24 2018 +1100
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13)     powerpc: Disable DAWR in the base POWER9 CPU features
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) Technical Details:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) ==================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) DAWR has 6 different ways of being set.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) 1) ptrace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) 2) h_set_mode(DAWR)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) 3) h_set_dabr()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) 4) kvmppc_set_one_reg()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) 5) xmon
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) For ptrace, we now advertise zero breakpoints on POWER9 via the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) PPC_PTRACE_GETHWDBGINFO call. This results in GDB falling back to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) software emulation of the watchpoint (which is slow).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) h_set_mode(DAWR) and h_set_dabr() will now return an error to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) guest on a POWER9 host. Current Linux guests ignore this error, so
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) they will silently not get the DAWR.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) kvmppc_set_one_reg() will store the value in the vcpu but won't
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) actually set it on POWER9 hardware. This is done so we don't break
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) migration from POWER8 to POWER9, at the cost of silently losing the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) DAWR on the migration.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) For xmon, the 'bd' command will return an error on P9.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) Consequences for users
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) ======================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) For GDB watchpoints (ie 'watch' command) on POWER9 bare metal , GDB
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) will accept the command. Unfortunately since there is no hardware
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) support for the watchpoint, GDB will software emulate the watchpoint
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) making it run very slowly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) The same will also be true for any guests started on a POWER9
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) host. The watchpoint will fail and GDB will fall back to software
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) emulation.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) If a guest is started on a POWER8 host, GDB will accept the watchpoint
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) and configure the hardware to use the DAWR. This will run at full
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) speed since it can use the hardware emulation. Unfortunately if this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) guest is migrated to a POWER9 host, the watchpoint will be lost on the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) POWER9. Loads and stores to the watchpoint locations will not be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) trapped in GDB. The watchpoint is remembered, so if the guest is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) migrated back to the POWER8 host, it will start working again.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) Force enabling the DAWR
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) =======================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) Kernels (since ~v5.2) have an option to force enable the DAWR via::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64)   echo Y > /sys/kernel/debug/powerpc/dawr_enable_dangerous
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) This enables the DAWR even on POWER9.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) This is a dangerous setting, USE AT YOUR OWN RISK.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) Some users may not care about a bad user crashing their box
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) (ie. single user/desktop systems) and really want the DAWR.  This
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) allows them to force enable DAWR.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) This flag can also be used to disable DAWR access. Once this is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) cleared, all DAWR access should be cleared immediately and your
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) machine once again safe from crashing.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) Userspace may get confused by toggling this. If DAWR is force
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) enabled/disabled between getting the number of breakpoints (via
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) PTRACE_GETHWDBGINFO) and setting the breakpoint, userspace will get an
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) inconsistent view of what's available. Similarly for guests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) For the DAWR to be enabled in a KVM guest, the DAWR needs to be force
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) enabled in the host AND the guest. For this reason, this won't work on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) POWERVM as it doesn't allow the HCALL to work. Writes of 'Y' to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) dawr_enable_dangerous file will fail if the hypervisor doesn't support
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) writing the DAWR.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) To double check the DAWR is working, run this kernel selftest:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91)   tools/testing/selftests/powerpc/ptrace/ptrace-hwbreak.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) Any errors/failures/skips mean something is wrong.