^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) .. SPDX-License-Identifier: GPL-2.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ==================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4) PCI Error Recovery
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) ==================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) :Authors: - Linas Vepstas <linasvepstas@gmail.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) - Richard Lary <rlary@us.ibm.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) - Mike Mason <mmlnx@us.ibm.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) Many PCI bus controllers are able to detect a variety of hardware
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) PCI errors on the bus, such as parity errors on the data and address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) buses, as well as SERR and PERR errors. Some of the more advanced
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) chipsets are able to deal with these errors; these include PCI-E chipsets,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) and the PCI-host bridges found on IBM Power4, Power5 and Power6-based
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) pSeries boxes. A typical action taken is to disconnect the affected device,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) halting all I/O to it. The goal of a disconnection is to avoid system
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) corruption; for example, to halt system memory corruption due to DMA's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) to "wild" addresses. Typically, a reconnection mechanism is also
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) offered, so that the affected PCI device(s) are reset and put back
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) into working condition. The reset phase requires coordination
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) between the affected device drivers and the PCI controller chip.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) This document describes a generic API for notifying device drivers
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) of a bus disconnection, and then performing error recovery.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) This API is currently implemented in the 2.6.16 and later kernels.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) Reporting and recovery is performed in several steps. First, when
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) a PCI hardware error has resulted in a bus disconnect, that event
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) is reported as soon as possible to all affected device drivers,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) including multiple instances of a device driver on multi-function
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) cards. This allows device drivers to avoid deadlocking in spinloops,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) waiting for some i/o-space register to change, when it never will.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) It also gives the drivers a chance to defer incoming I/O as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) needed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) Next, recovery is performed in several stages. Most of the complexity
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) is forced by the need to handle multi-function devices, that is,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) devices that have multiple device drivers associated with them.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) In the first stage, each driver is allowed to indicate what type
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) of reset it desires, the choices being a simple re-enabling of I/O
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) or requesting a slot reset.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) If any driver requests a slot reset, that is what will be done.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) After a reset and/or a re-enabling of I/O, all drivers are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) again notified, so that they may then perform any device setup/config
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) that may be required. After these have all completed, a final
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) "resume normal operations" event is sent out.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) The biggest reason for choosing a kernel-based implementation rather
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) than a user-space implementation was the need to deal with bus
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) disconnects of PCI devices attached to storage media, and, in particular,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) disconnects from devices holding the root file system. If the root
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) file system is disconnected, a user-space mechanism would have to go
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) through a large number of contortions to complete recovery. Almost all
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) of the current Linux file systems are not tolerant of disconnection
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) from/reconnection to their underlying block device. By contrast,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) bus errors are easy to manage in the device driver. Indeed, most
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) device drivers already handle very similar recovery procedures;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) for example, the SCSI-generic layer already provides significant
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) mechanisms for dealing with SCSI bus errors and SCSI bus resets.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) Detailed Design
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) ===============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) Design and implementation details below, based on a chain of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) public email discussions with Ben Herrenschmidt, circa 5 April 2005.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) The error recovery API support is exposed to the driver in the form of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) a structure of function pointers pointed to by a new field in struct
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) pci_driver. A driver that fails to provide the structure is "non-aware",
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) and the actual recovery steps taken are platform dependent. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) arch/powerpc implementation will simulate a PCI hotplug remove/add.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) This structure has the form::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) struct pci_error_handlers
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) int (*error_detected)(struct pci_dev *dev, pci_channel_state_t);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) int (*mmio_enabled)(struct pci_dev *dev);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) int (*slot_reset)(struct pci_dev *dev);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) void (*resume)(struct pci_dev *dev);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) };
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) The possible channel states are::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) typedef enum {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) pci_channel_io_normal, /* I/O channel is in normal state */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) pci_channel_io_frozen, /* I/O to channel is blocked */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) pci_channel_io_perm_failure, /* PCI card is dead */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) } pci_channel_state_t;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) Possible return values are::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) enum pci_ers_result {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) PCI_ERS_RESULT_NONE, /* no result/none/not supported in device driver */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) PCI_ERS_RESULT_CAN_RECOVER, /* Device driver can recover without slot reset */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) PCI_ERS_RESULT_NEED_RESET, /* Device driver wants slot to be reset. */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) PCI_ERS_RESULT_DISCONNECT, /* Device has completely failed, is unrecoverable */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) PCI_ERS_RESULT_RECOVERED, /* Device driver is fully recovered and operational */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) };
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) A driver does not have to implement all of these callbacks; however,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) if it implements any, it must implement error_detected(). If a callback
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) is not implemented, the corresponding feature is considered unsupported.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) For example, if mmio_enabled() and resume() aren't there, then it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) is assumed that the driver is not doing any direct recovery and requires
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) a slot reset. Typically a driver will want to know about
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) a slot_reset().
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) The actual steps taken by a platform to recover from a PCI error
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) event will be platform-dependent, but will follow the general
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) sequence described below.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) STEP 0: Error Event
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) -------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) A PCI bus error is detected by the PCI hardware. On powerpc, the slot
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) is isolated, in that all I/O is blocked: all reads return 0xffffffff,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) all writes are ignored.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) STEP 1: Notification
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) --------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) Platform calls the error_detected() callback on every instance of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) every driver affected by the error.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) At this point, the device might not be accessible anymore, depending on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) the platform (the slot will be isolated on powerpc). The driver may
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) already have "noticed" the error because of a failing I/O, but this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) is the proper "synchronization point", that is, it gives the driver
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) a chance to cleanup, waiting for pending stuff (timers, whatever, etc...)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) to complete; it can take semaphores, schedule, etc... everything but
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) touch the device. Within this function and after it returns, the driver
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) shouldn't do any new IOs. Called in task context. This is sort of a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) "quiesce" point. See note about interrupts at the end of this doc.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) All drivers participating in this system must implement this call.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) The driver must return one of the following result codes:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) - PCI_ERS_RESULT_CAN_RECOVER
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) Driver returns this if it thinks it might be able to recover
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) the HW by just banging IOs or if it wants to be given
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) a chance to extract some diagnostic information (see
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) mmio_enable, below).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) - PCI_ERS_RESULT_NEED_RESET
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) Driver returns this if it can't recover without a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) slot reset.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) - PCI_ERS_RESULT_DISCONNECT
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) Driver returns this if it doesn't want to recover at all.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) The next step taken will depend on the result codes returned by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) drivers.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) If all drivers on the segment/slot return PCI_ERS_RESULT_CAN_RECOVER,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) then the platform should re-enable IOs on the slot (or do nothing in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) particular, if the platform doesn't isolate slots), and recovery
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) proceeds to STEP 2 (MMIO Enable).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) If any driver requested a slot reset (by returning PCI_ERS_RESULT_NEED_RESET),
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) then recovery proceeds to STEP 4 (Slot Reset).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) If the platform is unable to recover the slot, the next step
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) is STEP 6 (Permanent Failure).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) .. note::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) The current powerpc implementation assumes that a device driver will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) *not* schedule or semaphore in this routine; the current powerpc
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) implementation uses one kernel thread to notify all devices;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) thus, if one device sleeps/schedules, all devices are affected.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) Doing better requires complex multi-threaded logic in the error
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) recovery implementation (e.g. waiting for all notification threads
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) to "join" before proceeding with recovery.) This seems excessively
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) complex and not worth implementing.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) The current powerpc implementation doesn't much care if the device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) attempts I/O at this point, or not. I/O's will fail, returning
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) a value of 0xff on read, and writes will be dropped. If more than
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) EEH_MAX_FAILS I/O's are attempted to a frozen adapter, EEH
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) assumes that the device driver has gone into an infinite loop
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) and prints an error to syslog. A reboot is then required to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) get the device working again.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) STEP 2: MMIO Enabled
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) --------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) The platform re-enables MMIO to the device (but typically not the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) DMA), and then calls the mmio_enabled() callback on all affected
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) device drivers.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) This is the "early recovery" call. IOs are allowed again, but DMA is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) not, with some restrictions. This is NOT a callback for the driver to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) start operations again, only to peek/poke at the device, extract diagnostic
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) information, if any, and eventually do things like trigger a device local
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) reset or some such, but not restart operations. This callback is made if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) all drivers on a segment agree that they can try to recover and if no automatic
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) link reset was performed by the HW. If the platform can't just re-enable IOs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) without a slot reset or a link reset, it will not call this callback, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) instead will have gone directly to STEP 3 (Link Reset) or STEP 4 (Slot Reset)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) .. note::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) The following is proposed; no platform implements this yet:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) Proposal: All I/O's should be done _synchronously_ from within
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) this callback, errors triggered by them will be returned via
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) the normal pci_check_whatever() API, no new error_detected()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209) callback will be issued due to an error happening here. However,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) such an error might cause IOs to be re-blocked for the whole
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) segment, and thus invalidate the recovery that other devices
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212) on the same segment might have done, forcing the whole segment
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) into one of the next states, that is, link reset or slot reset.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215) The driver should return one of the following result codes:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) - PCI_ERS_RESULT_RECOVERED
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) Driver returns this if it thinks the device is fully
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218) functional and thinks it is ready to start
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) normal driver operations again. There is no
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220) guarantee that the driver will actually be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) allowed to proceed, as another driver on the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) same segment might have failed and thus triggered a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) slot reset on platforms that support it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) - PCI_ERS_RESULT_NEED_RESET
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226) Driver returns this if it thinks the device is not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) recoverable in its current state and it needs a slot
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228) reset to proceed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) - PCI_ERS_RESULT_DISCONNECT
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231) Same as above. Total failure, no recovery even after
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) reset driver dead. (To be defined more precisely)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234) The next step taken depends on the results returned by the drivers.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) If all drivers returned PCI_ERS_RESULT_RECOVERED, then the platform
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236) proceeds to either STEP3 (Link Reset) or to STEP 5 (Resume Operations).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) If any driver returned PCI_ERS_RESULT_NEED_RESET, then the platform
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) proceeds to STEP 4 (Slot Reset)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241) STEP 3: Link Reset
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) ------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) The platform resets the link. This is a PCI-Express specific step
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) and is done whenever a fatal error has been detected that can be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245) "solved" by resetting the link.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) STEP 4: Slot Reset
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) ------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) In response to a return value of PCI_ERS_RESULT_NEED_RESET, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) platform will perform a slot reset on the requesting PCI device(s).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252) The actual steps taken by a platform to perform a slot reset
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) will be platform-dependent. Upon completion of slot reset, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) platform will call the device slot_reset() callback.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256) Powerpc platforms implement two levels of slot reset:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) soft reset(default) and fundamental(optional) reset.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) Powerpc soft reset consists of asserting the adapter #RST line and then
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260) restoring the PCI BAR's and PCI configuration header to a state
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) that is equivalent to what it would be after a fresh system
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) power-on followed by power-on BIOS/system firmware initialization.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263) Soft reset is also known as hot-reset.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) Powerpc fundamental reset is supported by PCI Express cards only
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266) and results in device's state machines, hardware logic, port states and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) configuration registers to initialize to their default conditions.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269) For most PCI devices, a soft reset will be sufficient for recovery.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) Optional fundamental reset is provided to support a limited number
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271) of PCI Express devices for which a soft reset is not sufficient
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272) for recovery.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274) If the platform supports PCI hotplug, then the reset might be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) performed by toggling the slot electrical power off/on.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277) It is important for the platform to restore the PCI config space
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278) to the "fresh poweron" state, rather than the "last state". After
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279) a slot reset, the device driver will almost always use its standard
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280) device initialization routines, and an unusual config space setup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281) may result in hung devices, kernel panics, or silent data corruption.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283) This call gives drivers the chance to re-initialize the hardware
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284) (re-download firmware, etc.). At this point, the driver may assume
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285) that the card is in a fresh state and is fully functional. The slot
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286) is unfrozen and the driver has full access to PCI config space,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287) memory mapped I/O space and DMA. Interrupts (Legacy, MSI, or MSI-X)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288) will also be available.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) Drivers should not restart normal I/O processing operations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291) at this point. If all device drivers report success on this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292) callback, the platform will call resume() to complete the sequence,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293) and let the driver restart normal I/O processing.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295) A driver can still return a critical failure for this function if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296) it can't get the device operational after reset. If the platform
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297) previously tried a soft reset, it might now try a hard reset (power
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298) cycle) and then call slot_reset() again. It the device still can't
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299) be recovered, there is nothing more that can be done; the platform
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300) will typically report a "permanent failure" in such a case. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301) device will be considered "dead" in this case.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303) Drivers for multi-function cards will need to coordinate among
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) themselves as to which driver instance will perform any "one-shot"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305) or global device initialization. For example, the Symbios sym53cxx2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306) driver performs device init only from PCI function 0::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308) + if (PCI_FUNC(pdev->devfn) == 0)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309) + sym_reset_scsi_bus(np, 0);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311) Result codes:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312) - PCI_ERS_RESULT_DISCONNECT
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313) Same as above.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315) Drivers for PCI Express cards that require a fundamental reset must
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) set the needs_freset bit in the pci_dev structure in their probe function.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317) For example, the QLogic qla2xxx driver sets the needs_freset bit for certain
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318) PCI card types::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320) + /* Set EEH reset type to fundamental if required by hba */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321) + if (IS_QLA24XX(ha) || IS_QLA25XX(ha) || IS_QLA81XX(ha))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322) + pdev->needs_freset = 1;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323) +
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325) Platform proceeds either to STEP 5 (Resume Operations) or STEP 6 (Permanent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326) Failure).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328) .. note::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330) The current powerpc implementation does not try a power-cycle
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331) reset if the driver returned PCI_ERS_RESULT_DISCONNECT.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332) However, it probably should.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335) STEP 5: Resume Operations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336) -------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) The platform will call the resume() callback on all affected device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338) drivers if all drivers on the segment have returned
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339) PCI_ERS_RESULT_RECOVERED from one of the 3 previous callbacks.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340) The goal of this callback is to tell the driver to restart activity,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341) that everything is back and running. This callback does not return
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342) a result code.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344) At this point, if a new error happens, the platform will restart
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345) a new error recovery sequence.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347) STEP 6: Permanent Failure
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348) -------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349) A "permanent failure" has occurred, and the platform cannot recover
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) the device. The platform will call error_detected() with a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351) pci_channel_state_t value of pci_channel_io_perm_failure.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353) The device driver should, at this point, assume the worst. It should
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354) cancel all pending I/O, refuse all new I/O, returning -EIO to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355) higher layers. The device driver should then clean up all of its
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) memory and remove itself from kernel operations, much as it would
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357) during system shutdown.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359) The platform will typically notify the system operator of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360) permanent failure in some way. If the device is hotplug-capable,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361) the operator will probably want to remove and replace the device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362) Note, however, not all failures are truly "permanent". Some are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363) caused by over-heating, some by a poorly seated card. Many
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364) PCI error events are caused by software bugs, e.g. DMA's to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365) wild addresses or bogus split transactions due to programming
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366) errors. See the discussion in powerpc/eeh-pci-error-recovery.txt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367) for additional detail on real-life experience of the causes of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368) software errors.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) Conclusion; General Remarks
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372) ---------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373) The way the callbacks are called is platform policy. A platform with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374) no slot reset capability may want to just "ignore" drivers that can't
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375) recover (disconnect them) and try to let other cards on the same segment
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376) recover. Keep in mind that in most real life cases, though, there will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377) be only one driver per segment.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379) Now, a note about interrupts. If you get an interrupt and your
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 380) device is dead or has been isolated, there is a problem :)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 381) The current policy is to turn this into a platform policy.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 382) That is, the recovery API only requires that:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 383)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 384) - There is no guarantee that interrupt delivery can proceed from any
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 385) device on the segment starting from the error detection and until the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 386) slot_reset callback is called, at which point interrupts are expected
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 387) to be fully operational.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 388)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 389) - There is no guarantee that interrupt delivery is stopped, that is,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 390) a driver that gets an interrupt after detecting an error, or that detects
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 391) an error within the interrupt handler such that it prevents proper
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 392) ack'ing of the interrupt (and thus removal of the source) should just
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 393) return IRQ_NOTHANDLED. It's up to the platform to deal with that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 394) condition, typically by masking the IRQ source during the duration of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 395) the error handling. It is expected that the platform "knows" which
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 396) interrupts are routed to error-management capable slots and can deal
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 397) with temporarily disabling that IRQ number during error processing (this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 398) isn't terribly complex). That means some IRQ latency for other devices
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 399) sharing the interrupt, but there is simply no other way. High end
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 400) platforms aren't supposed to share interrupts between many devices
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 401) anyway :)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 402)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 403) .. note::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 404)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 405) Implementation details for the powerpc platform are discussed in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 406) the file Documentation/powerpc/eeh-pci-error-recovery.rst
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 407)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 408) As of this writing, there is a growing list of device drivers with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 409) patches implementing error recovery. Not all of these patches are in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 410) mainline yet. These may be used as "examples":
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 411)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 412) - drivers/scsi/ipr
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 413) - drivers/scsi/sym53c8xx_2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 414) - drivers/scsi/qla2xxx
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 415) - drivers/scsi/lpfc
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 416) - drivers/next/bnx2.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 417) - drivers/next/e100.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 418) - drivers/net/e1000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 419) - drivers/net/e1000e
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 420) - drivers/net/ixgb
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 421) - drivers/net/ixgbe
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 422) - drivers/net/cxgb3
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 423) - drivers/net/s2io.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 424)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 425) The End
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 426) -------
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags