Orange Pi5 kernel

^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   1) .. SPDX-License-Identifier: GPL-2.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   2) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   3) ===============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   4) Boot Interrupts
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   5) ===============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   6) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   7) :Author: - Sean V Kelley <sean.v.kelley@linux.intel.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   8) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   9) Overview
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  10) ========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  11) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  12) On PCI Express, interrupts are represented with either MSI or inbound
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  13) interrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  14) given Core IO converts the legacy interrupt messages from PCI Express to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  15) MSI interrupts.  If the IO-APIC is disabled (via the mask bits in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  16) IO-APIC table entries), the messages are routed to the legacy PCH. This
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  17) in-band interrupt mechanism was traditionally necessary for systems that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  18) did not support the IO-APIC and for boot. Intel in the past has used the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  19) term "boot interrupts" to describe this mechanism. Further, the PCI Express
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  20) protocol describes this in-band legacy wire-interrupt INTx mechanism for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  21) I/O devices to signal PCI-style level interrupts. The subsequent paragraphs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  22) describe problems with the Core IO handling of INTx message routing to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  23) PCH and mitigation within BIOS and the OS.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  24) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  25) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  26) Issue
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  27) =====
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  28) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  29) When in-band legacy INTx messages are forwarded to the PCH, they in turn
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  30) trigger a new interrupt for which the OS likely lacks a handler. When an
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  31) interrupt goes unhandled over time, they are tracked by the Linux kernel as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  32) Spurious Interrupts. The IRQ will be disabled by the Linux kernel after it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  33) reaches a specific count with the error "nobody cared". This disabled IRQ
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  34) now prevents valid usage by an existing interrupt which may happen to share
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  35) the IRQ line::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  36) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  37)   irq 19: nobody cared (try booting with the "irqpoll" option)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  38)   CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  39)   Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  40)   Call Trace:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  41) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  42)   <IRQ>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  43)    ? dump_stack+0x46/0x5e
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  44)    ? __report_bad_irq+0x2e/0xb0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  45)    ? note_interrupt+0x242/0x290
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  46)    ? nNIKAL100_memoryRead16+0x8/0x10 [nikal]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  47)    ? handle_irq_event_percpu+0x55/0x70
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  48)    ? handle_irq_event+0x4f/0x80
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  49)    ? handle_fasteoi_irq+0x81/0x180
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  50)    ? handle_irq+0x1c/0x30
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  51)    ? do_IRQ+0x41/0xd0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  52)    ? common_interrupt+0x84/0x84
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  53)   </IRQ>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  54) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  55)   handlers:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  56)   irq_default_primary_handler threaded usb_hcd_irq
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  57)   Disabling IRQ #19
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  58) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  59) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  60) Conditions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  61) ==========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  62) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  63) The use of threaded interrupts is the most likely condition to trigger
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  64) this problem today. Threaded interrupts may not be reenabled after the IRQ
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  65) handler wakes. These "one shot" conditions mean that the threaded interrupt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  66) needs to keep the interrupt line masked until the threaded handler has run.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  67) Especially when dealing with high data rate interrupts, the thread needs to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  68) run to completion; otherwise some handlers will end up in stack overflows
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  69) since the interrupt of the issuing device is still active.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  70) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  71) Affected Chipsets
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  72) =================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  73) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  74) The legacy interrupt forwarding mechanism exists today in a number of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  75) devices including but not limited to chipsets from AMD/ATI, Broadcom, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  76) Intel. Changes made through the mitigations below have been applied to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  77) drivers/pci/quirks.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  78) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  79) Starting with ICX there are no longer any IO-APICs in the Core IO's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  80) devices.  IO-APIC is only in the PCH.  Devices connected to the Core IO's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  81) PCIe Root Ports will use native MSI/MSI-X mechanisms.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  82) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  83) Mitigations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  84) ===========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  85) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  86) The mitigations take the form of PCI quirks. The preference has been to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  87) first identify and make use of a means to disable the routing to the PCH.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  88) In such a case a quirk to disable boot interrupt generation can be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  89) added. [1]_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  90) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  91) Intel® 6300ESB I/O Controller Hub
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  92)   Alternate Base Address Register:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  93)    BIE: Boot Interrupt Enable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  94) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  95) 	  ==  ===========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  96) 	  0   Boot interrupt is enabled.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  97) 	  1   Boot interrupt is disabled.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  98) 	  ==  ===========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  99) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) Intel® Sandy Bridge through Sky Lake based Xeon servers:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101)   Coherent Interface Protocol Interrupt Control
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102)    dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) 	  When this bit is set. Local INTx messages received from the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) 	  Intel® Quick Data DMA/PCI Express ports are not routed to legacy
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) 	  PCH - they are either converted into MSI via the integrated IO-APIC
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) 	  (if the IO-APIC mask bit is clear in the appropriate entries)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) 	  or cause no further action (when mask bit is set)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) In the absence of a way to directly disable the routing, another approach
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) has been to make use of PCI Interrupt pin to INTx routing tables for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) purposes of redirecting the interrupt handler to the rerouted interrupt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) line by default.  Therefore, on chipsets where this INTx routing cannot be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) disabled, the Linux kernel will reroute the valid interrupt to its legacy
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) interrupt. This redirection of the handler will prevent the occurrence of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) the spurious interrupt detection which would ordinarily disable the IRQ
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) line due to excessive unhandled counts. [2]_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) disable) the redirection of the interrupt handler to the PCH interrupt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) line. The option can be overridden by either pci=ioapicreroute or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) pci=noioapicreroute. [3]_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) More Documentation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) ==================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) There is an overview of the legacy interrupt handling in several datasheets
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) (6300ESB and 6700PXH below). While largely the same, it provides insight
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) into the evolution of its handling with chipsets.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) Example of disabling of the boot interrupt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) ------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134)       - Intel® 6300ESB I/O Controller Hub (Document # 300641-004US)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) 	5.7.3 Boot Interrupt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) 	https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138)       - Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) 	Datasheet - Volume 2: Registers (Document # 330784-003)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) 	6.6.41 cipintrc Coherent Interface Protocol Interrupt Control
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) 	https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) Example of handler rerouting
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) ----------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146)       - Intel® 6700PXH 64-bit PCI Hub (Document # 302628)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) 	2.15.2 PCI Express Legacy INTx Support and Boot Interrupt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) 	https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) If you have any legacy PCI interrupt questions that aren't answered, email me.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) Cheers,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154)     Sean V Kelley
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155)     sean.v.kelley@linux.intel.com
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) .. [1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) .. [2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) .. [3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/