^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) ==================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) vfio-ccw: the basic infrastructure
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ==================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) Introduction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) ------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) Here we describe the vfio support for I/O subchannel devices for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) Linux/s390. Motivation for vfio-ccw is to passthrough subchannels to a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) virtual machine, while vfio is the means.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) Different than other hardware architectures, s390 has defined a unified
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) I/O access method, which is so called Channel I/O. It has its own access
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) patterns:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) - Channel programs run asynchronously on a separate (co)processor.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) - The channel subsystem will access any memory designated by the caller
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) in the channel program directly, i.e. there is no iommu involved.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) Thus when we introduce vfio support for these devices, we realize it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) with a mediated device (mdev) implementation. The vfio mdev will be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) added to an iommu group, so as to make itself able to be managed by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) vfio framework. And we add read/write callbacks for special vfio I/O
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) regions to pass the channel programs from the mdev to its parent device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) (the real I/O subchannel device) to do further address translation and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) to perform I/O instructions.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) This document does not intend to explain the s390 I/O architecture in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) every detail. More information/reference could be found here:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) - A good start to know Channel I/O in general:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) https://en.wikipedia.org/wiki/Channel_I/O
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) - s390 architecture:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) s390 Principles of Operation manual (IBM Form. No. SA22-7832)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) - The existing QEMU code which implements a simple emulated channel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) subsystem could also be a good reference. It makes it easier to follow
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) the flow.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) qemu/hw/s390x/css.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) For vfio mediated device framework:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) - Documentation/driver-api/vfio-mediated-device.rst
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) Motivation of vfio-ccw
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) ----------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) Typically, a guest virtualized via QEMU/KVM on s390 only sees
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) paravirtualized virtio devices via the "Virtio Over Channel I/O
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) (virtio-ccw)" transport. This makes virtio devices discoverable via
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) standard operating system algorithms for handling channel devices.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) However this is not enough. On s390 for the majority of devices, which
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) use the standard Channel I/O based mechanism, we also need to provide
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) the functionality of passing through them to a QEMU virtual machine.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) This includes devices that don't have a virtio counterpart (e.g. tape
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) drives) or that have specific characteristics which guests want to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) exploit.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) For passing a device to a guest, we want to use the same interface as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) everybody else, namely vfio. We implement this vfio support for channel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) devices via the vfio mediated device framework and the subchannel device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) driver "vfio_ccw".
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) Access patterns of CCW devices
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) ------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) s390 architecture has implemented a so called channel subsystem, that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) provides a unified view of the devices physically attached to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) systems. Though the s390 hardware platform knows about a huge variety of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) different peripheral attachments like disk devices (aka. DASDs), tapes,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) communication controllers, etc. They can all be accessed by a well
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) defined access method and they are presenting I/O completion a unified
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) way: I/O interruptions.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) All I/O requires the use of channel command words (CCWs). A CCW is an
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) instruction to a specialized I/O channel processor. A channel program is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) a sequence of CCWs which are executed by the I/O channel subsystem. To
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) issue a channel program to the channel subsystem, it is required to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) build an operation request block (ORB), which can be used to point out
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) the format of the CCW and other control information to the system. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) operating system signals the I/O channel subsystem to begin executing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) the channel program with a SSCH (start sub-channel) instruction. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) central processor is then free to proceed with non-I/O instructions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) until interrupted. The I/O completion result is received by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) interrupt handler in the form of interrupt response block (IRB).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) Back to vfio-ccw, in short:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) - ORBs and channel programs are built in guest kernel (with guest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) physical addresses).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) - ORBs and channel programs are passed to the host kernel.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) - Host kernel translates the guest physical addresses to real addresses
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) and starts the I/O with issuing a privileged Channel I/O instruction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) (e.g SSCH).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) - channel programs run asynchronously on a separate processor.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) - I/O completion will be signaled to the host with I/O interruptions.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) And it will be copied as IRB to user space to pass it back to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) guest.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) Physical vfio ccw device and its child mdev
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) -------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) As mentioned above, we realize vfio-ccw with a mdev implementation.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) Channel I/O does not have IOMMU hardware support, so the physical
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) vfio-ccw device does not have an IOMMU level translation or isolation.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) Subchannel I/O instructions are all privileged instructions. When
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) handling the I/O instruction interception, vfio-ccw has the software
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) policing and translation how the channel program is programmed before
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) it gets sent to hardware.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) Within this implementation, we have two drivers for two types of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) devices:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) - The vfio_ccw driver for the physical subchannel device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) This is an I/O subchannel driver for the real subchannel device. It
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) realizes a group of callbacks and registers to the mdev framework as a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) parent (physical) device. As a consequence, mdev provides vfio_ccw a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) generic interface (sysfs) to create mdev devices. A vfio mdev could be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) created by vfio_ccw then and added to the mediated bus. It is the vfio
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) device that added to an IOMMU group and a vfio group.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) vfio_ccw also provides an I/O region to accept channel program
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) request from user space and store I/O interrupt result for user
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) space to retrieve. To notify user space an I/O completion, it offers
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) an interface to setup an eventfd fd for asynchronous signaling.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) - The vfio_mdev driver for the mediated vfio ccw device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) This is provided by the mdev framework. It is a vfio device driver for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) the mdev that created by vfio_ccw.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) It realizes a group of vfio device driver callbacks, adds itself to a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) vfio group, and registers itself to the mdev framework as a mdev
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) driver.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) It uses a vfio iommu backend that uses the existing map and unmap
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) ioctls, but rather than programming them into an IOMMU for a device,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) it simply stores the translations for use by later requests. This
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) means that a device programmed in a VM with guest physical addresses
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) can have the vfio kernel convert that address to process virtual
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) address, pin the page and program the hardware with the host physical
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) address in one step.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) For a mdev, the vfio iommu backend will not pin the pages during the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) VFIO_IOMMU_MAP_DMA ioctl. Mdev framework will only maintain a database
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) of the iova<->vaddr mappings in this operation. And they export a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) vfio_pin_pages and a vfio_unpin_pages interfaces from the vfio iommu
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) backend for the physical devices to pin and unpin pages by demand.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) Below is a high Level block diagram::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) +-------------+
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) | |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) | +---------+ | mdev_register_driver() +--------------+
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) | | Mdev | +<-----------------------+ |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) | | bus | | | vfio_mdev.ko |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) | | driver | +----------------------->+ |<-> VFIO user
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) | +---------+ | probe()/remove() +--------------+ APIs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) | |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) | MDEV CORE |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) | MODULE |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) | mdev.ko |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) | +---------+ | mdev_register_device() +--------------+
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) | |Physical | +<-----------------------+ |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) | | device | | | vfio_ccw.ko |<-> subchannel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) | |interface| +----------------------->+ | device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) | +---------+ | callback +--------------+
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) +-------------+
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) The process of how these work together.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) 1. vfio_ccw.ko drives the physical I/O subchannel, and registers the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) physical device (with callbacks) to mdev framework.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) When vfio_ccw probing the subchannel device, it registers device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) pointer and callbacks to the mdev framework. Mdev related file nodes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) under the device node in sysfs would be created for the subchannel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) device, namely 'mdev_create', 'mdev_destroy' and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) 'mdev_supported_types'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) 2. Create a mediated vfio ccw device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) Use the 'mdev_create' sysfs file, we need to manually create one (and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) only one for our case) mediated device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) 3. vfio_mdev.ko drives the mediated ccw device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) vfio_mdev is also the vfio device drvier. It will probe the mdev and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) add it to an iommu_group and a vfio_group. Then we could pass through
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) the mdev to a guest.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) VFIO-CCW Regions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) ----------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) The vfio-ccw driver exposes MMIO regions to accept requests from and return
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) results to userspace.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) vfio-ccw I/O region
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) -------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) An I/O region is used to accept channel program request from user
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) space and store I/O interrupt result for user space to retrieve. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) definition of the region is::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) struct ccw_io_region {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) #define ORB_AREA_SIZE 12
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) __u8 orb_area[ORB_AREA_SIZE];
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) #define SCSW_AREA_SIZE 12
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) __u8 scsw_area[SCSW_AREA_SIZE];
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) #define IRB_AREA_SIZE 96
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) __u8 irb_area[IRB_AREA_SIZE];
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) __u32 ret_code;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) } __packed;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) This region is always available.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209) While starting an I/O request, orb_area should be filled with the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) guest ORB, and scsw_area should be filled with the SCSW of the Virtual
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) Subchannel.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) irb_area stores the I/O result.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215) ret_code stores a return code for each access of the region. The following
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) values may occur:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218) ``0``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) The operation was successful.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) ``-EOPNOTSUPP``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) The orb specified transport mode or an unidentified IDAW format, or the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) scsw specified a function other than the start function.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) ``-EIO``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226) A request was issued while the device was not in a state ready to accept
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) requests, or an internal error occurred.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) ``-EBUSY``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) The subchannel was status pending or busy, or a request is already active.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) ``-EAGAIN``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233) A request was being processed, and the caller should retry.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) ``-EACCES``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236) The channel path(s) used for the I/O were found to be not operational.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) ``-ENODEV``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) The device was found to be not operational.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241) ``-EINVAL``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) The orb specified a chain longer than 255 ccws, or an internal error
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) occurred.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246) vfio-ccw cmd region
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) -------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249) The vfio-ccw cmd region is used to accept asynchronous instructions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) from userspace::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252) #define VFIO_CCW_ASYNC_CMD_HSCH (1 << 0)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) #define VFIO_CCW_ASYNC_CMD_CSCH (1 << 1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) struct ccw_cmd_region {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) __u32 command;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256) __u32 ret_code;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) } __packed;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) This region is exposed via region type VFIO_REGION_SUBTYPE_CCW_ASYNC_CMD.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) Currently, CLEAR SUBCHANNEL and HALT SUBCHANNEL use this region.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263) command specifies the command to be issued; ret_code stores a return code
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) for each access of the region. The following values may occur:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266) ``0``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) The operation was successful.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269) ``-ENODEV``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) The device was found to be not operational.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272) ``-EINVAL``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273) A command other than halt or clear was specified.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) ``-EIO``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276) A request was issued while the device was not in a state ready to accept
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277) requests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279) ``-EAGAIN``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280) A request was being processed, and the caller should retry.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282) ``-EBUSY``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283) The subchannel was status pending or busy while processing a halt request.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285) vfio-ccw schib region
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286) ---------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288) The vfio-ccw schib region is used to return Subchannel-Information
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289) Block (SCHIB) data to userspace::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291) struct ccw_schib_region {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292) #define SCHIB_AREA_SIZE 52
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293) __u8 schib_area[SCHIB_AREA_SIZE];
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294) } __packed;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296) This region is exposed via region type VFIO_REGION_SUBTYPE_CCW_SCHIB.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298) Reading this region triggers a STORE SUBCHANNEL to be issued to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299) associated hardware.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301) vfio-ccw crw region
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302) ---------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) The vfio-ccw crw region is used to return Channel Report Word (CRW)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305) data to userspace::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307) struct ccw_crw_region {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308) __u32 crw;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309) __u32 pad;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310) } __packed;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312) This region is exposed via region type VFIO_REGION_SUBTYPE_CCW_CRW.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314) Reading this region returns a CRW if one that is relevant for this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315) subchannel (e.g. one reporting changes in channel path state) is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) pending, or all zeroes if not. If multiple CRWs are pending (including
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317) possibly chained CRWs), reading this region again will return the next
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318) one, until no more CRWs are pending and zeroes are returned. This is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319) similar to how STORE CHANNEL REPORT WORD works.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321) vfio-ccw operation details
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322) --------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324) vfio-ccw follows what vfio-pci did on the s390 platform and uses
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325) vfio-iommu-type1 as the vfio iommu backend.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327) * CCW translation APIs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328) A group of APIs (start with `cp_`) to do CCW translation. The CCWs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329) passed in by a user space program are organized with their guest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330) physical memory addresses. These APIs will copy the CCWs into kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331) space, and assemble a runnable kernel channel program by updating the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332) guest physical addresses with their corresponding host physical addresses.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333) Note that we have to use IDALs even for direct-access CCWs, as the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334) referenced memory can be located anywhere, including above 2G.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336) * vfio_ccw device driver
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) This driver utilizes the CCW translation APIs and introduces
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338) vfio_ccw, which is the driver for the I/O subchannel devices you want
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339) to pass through.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340) vfio_ccw implements the following vfio ioctls::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342) VFIO_DEVICE_GET_INFO
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343) VFIO_DEVICE_GET_IRQ_INFO
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344) VFIO_DEVICE_GET_REGION_INFO
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345) VFIO_DEVICE_RESET
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346) VFIO_DEVICE_SET_IRQS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348) This provides an I/O region, so that the user space program can pass a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349) channel program to the kernel, to do further CCW translation before
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) issuing them to a real device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351) This also provides the SET_IRQ ioctl to setup an event notifier to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352) notify the user space program the I/O completion in an asynchronous
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353) way.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355) The use of vfio-ccw is not limited to QEMU, while QEMU is definitely a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) good example to get understand how these patches work. Here is a little
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357) bit more detail how an I/O request triggered by the QEMU guest will be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358) handled (without error handling).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360) Explanation:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362) - Q1-Q7: QEMU side process.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363) - K1-K5: Kernel side process.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365) Q1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366) Get I/O region info during initialization.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368) Q2.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369) Setup event notifier and handler to handle I/O completion.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) ... ...
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373) Q3.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374) Intercept a ssch instruction.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375) Q4.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376) Write the guest channel program and ORB to the I/O region.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378) K1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379) Copy from guest to kernel.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 380) K2.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 381) Translate the guest channel program to a host kernel space
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 382) channel program, which becomes runnable for a real device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 383) K3.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 384) With the necessary information contained in the orb passed in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 385) by QEMU, issue the ccwchain to the device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 386) K4.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 387) Return the ssch CC code.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 388) Q5.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 389) Return the CC code to the guest.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 390)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 391) ... ...
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 392)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 393) K5.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 394) Interrupt handler gets the I/O result and write the result to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 395) the I/O region.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 396) K6.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 397) Signal QEMU to retrieve the result.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 398)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 399) Q6.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 400) Get the signal and event handler reads out the result from the I/O
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 401) region.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 402) Q7.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 403) Update the irb for the guest.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 404)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 405) Limitations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 406) -----------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 407)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 408) The current vfio-ccw implementation focuses on supporting basic commands
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 409) needed to implement block device functionality (read/write) of DASD/ECKD
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 410) device only. Some commands may need special handling in the future, for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 411) example, anything related to path grouping.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 412)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 413) DASD is a kind of storage device. While ECKD is a data recording format.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 414) More information for DASD and ECKD could be found here:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 415) https://en.wikipedia.org/wiki/Direct-access_storage_device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 416) https://en.wikipedia.org/wiki/Count_key_data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 417)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 418) Together with the corresponding work in QEMU, we can bring the passed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 419) through DASD/ECKD device online in a guest now and use it as a block
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 420) device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 421)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 422) The current code allows the guest to start channel programs via
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 423) START SUBCHANNEL, and to issue HALT SUBCHANNEL, CLEAR SUBCHANNEL,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 424) and STORE SUBCHANNEL.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 425)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 426) Currently all channel programs are prefetched, regardless of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 427) p-bit setting in the ORB. As a result, self modifying channel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 428) programs are not supported. For this reason, IPL has to be handled as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 429) a special case by a userspace/guest program; this has been implemented
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 430) in QEMU's s390-ccw bios as of QEMU 4.1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 431)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 432) vfio-ccw supports classic (command mode) channel I/O only. Transport
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 433) mode (HPF) is not supported.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 434)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 435) QDIO subchannels are currently not supported. Classic devices other than
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 436) DASD/ECKD might work, but have not been tested.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 437)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 438) Reference
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 439) ---------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 440) 1. ESA/s390 Principles of Operation manual (IBM Form. No. SA22-7832)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 441) 2. ESA/390 Common I/O Device Commands manual (IBM Form. No. SA22-7204)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 442) 3. https://en.wikipedia.org/wiki/Channel_I/O
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 443) 4. Documentation/s390/cds.rst
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 444) 5. Documentation/driver-api/vfio.rst
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 445) 6. Documentation/driver-api/vfio-mediated-device.rst
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags