^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) ====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) TCM Userspace Design
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) .. Contents:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) 1) Design
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) a) Background
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) b) Benefits
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) c) Design constraints
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) d) Implementation overview
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) i. Mailbox
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) ii. Command ring
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) iii. Data Area
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) e) Device discovery
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) f) Device events
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) g) Other contingencies
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) 2) Writing a user pass-through handler
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) a) Discovering and configuring TCMU uio devices
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) b) Waiting for events on the device(s)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) c) Managing the command ring
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) 3) A final note
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) Design
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) ======
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) TCM is another name for LIO, an in-kernel iSCSI target (server).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) Existing TCM targets run in the kernel. TCMU (TCM in Userspace)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) allows userspace programs to be written which act as iSCSI targets.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) This document describes the design.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) The existing kernel provides modules for different SCSI transport
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) protocols. TCM also modularizes the data storage. There are existing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) modules for file, block device, RAM or using another SCSI device as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) storage. These are called "backstores" or "storage engines". These
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) built-in modules are implemented entirely as kernel code.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) Background
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) ----------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) In addition to modularizing the transport protocol used for carrying
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) SCSI commands ("fabrics"), the Linux kernel target, LIO, also modularizes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) the actual data storage as well. These are referred to as "backstores"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) or "storage engines". The target comes with backstores that allow a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) file, a block device, RAM, or another SCSI device to be used for the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) local storage needed for the exported SCSI LUN. Like the rest of LIO,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) these are implemented entirely as kernel code.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) These backstores cover the most common use cases, but not all. One new
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) use case that other non-kernel target solutions, such as tgt, are able
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) to support is using Gluster's GLFS or Ceph's RBD as a backstore. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) target then serves as a translator, allowing initiators to store data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) in these non-traditional networked storage systems, while still only
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) using standard protocols themselves.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) If the target is a userspace process, supporting these is easy. tgt,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) for example, needs only a small adapter module for each, because the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) modules just use the available userspace libraries for RBD and GLFS.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) Adding support for these backstores in LIO is considerably more
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) difficult, because LIO is entirely kernel code. Instead of undertaking
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) the significant work to port the GLFS or RBD APIs and protocols to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) kernel, another approach is to create a userspace pass-through
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) backstore for LIO, "TCMU".
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) Benefits
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) --------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) In addition to allowing relatively easy support for RBD and GLFS, TCMU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) will also allow easier development of new backstores. TCMU combines
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) with the LIO loopback fabric to become something similar to FUSE
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) (Filesystem in Userspace), but at the SCSI layer instead of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) filesystem layer. A SUSE, if you will.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) The disadvantage is there are more distinct components to configure, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) potentially to malfunction. This is unavoidable, but hopefully not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) fatal if we're careful to keep things as simple as possible.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) Design constraints
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) ------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) - Good performance: high throughput, low latency
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) - Cleanly handle if userspace:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) 1) never attaches
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) 2) hangs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) 3) dies
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) 4) misbehaves
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) - Allow future flexibility in user & kernel implementations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) - Be reasonably memory-efficient
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) - Simple to configure & run
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) - Simple to write a userspace backend
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) Implementation overview
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) -----------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) The core of the TCMU interface is a memory region that is shared
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) between kernel and userspace. Within this region is: a control area
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) (mailbox); a lockless producer/consumer circular buffer for commands
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) to be passed up, and status returned; and an in/out data buffer area.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) TCMU uses the pre-existing UIO subsystem. UIO allows device driver
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) development in userspace, and this is conceptually very close to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) TCMU use case, except instead of a physical device, TCMU implements a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) memory-mapped layout designed for SCSI commands. Using UIO also
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) benefits TCMU by handling device introspection (e.g. a way for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) userspace to determine how large the shared region is) and signaling
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) mechanisms in both directions.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) There are no embedded pointers in the memory region. Everything is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) expressed as an offset from the region's starting address. This allows
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) the ring to still work if the user process dies and is restarted with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) the region mapped at a different virtual address.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) See target_core_user.h for the struct definitions.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) The Mailbox
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) -----------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) The mailbox is always at the start of the shared memory region, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) contains a version, details about the starting offset and size of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) command ring, and head and tail pointers to be used by the kernel and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) userspace (respectively) to put commands on the ring, and indicate
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) when the commands are completed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) version - 1 (userspace should abort if otherwise)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) flags:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) - TCMU_MAILBOX_FLAG_CAP_OOOC:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) indicates out-of-order completion is supported.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) See "The Command Ring" for details.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) cmdr_off
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) The offset of the start of the command ring from the start
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) of the memory region, to account for the mailbox size.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) cmdr_size
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) The size of the command ring. This does *not* need to be a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) power of two.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) cmd_head
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) Modified by the kernel to indicate when a command has been
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) placed on the ring.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) cmd_tail
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) Modified by userspace to indicate when it has completed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) processing of a command.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) The Command Ring
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) ----------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) Commands are placed on the ring by the kernel incrementing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) mailbox.cmd_head by the size of the command, modulo cmdr_size, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) then signaling userspace via uio_event_notify(). Once the command is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) completed, userspace updates mailbox.cmd_tail in the same way and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) signals the kernel via a 4-byte write(). When cmd_head equals
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) cmd_tail, the ring is empty -- no commands are currently waiting to be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) processed by userspace.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) TCMU commands are 8-byte aligned. They start with a common header
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) containing "len_op", a 32-bit value that stores the length, as well as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) the opcode in the lowest unused bits. It also contains cmd_id and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) flags fields for setting by the kernel (kflags) and userspace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) (uflags).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) Currently only two opcodes are defined, TCMU_OP_CMD and TCMU_OP_PAD.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) When the opcode is CMD, the entry in the command ring is a struct
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) tcmu_cmd_entry. Userspace finds the SCSI CDB (Command Data Block) via
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) tcmu_cmd_entry.req.cdb_off. This is an offset from the start of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) overall shared memory region, not the entry. The data in/out buffers
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) are accessible via tht req.iov[] array. iov_cnt contains the number of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) entries in iov[] needed to describe either the Data-In or Data-Out
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) buffers. For bidirectional commands, iov_cnt specifies how many iovec
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) entries cover the Data-Out area, and iov_bidi_cnt specifies how many
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) iovec entries immediately after that in iov[] cover the Data-In
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) area. Just like other fields, iov.iov_base is an offset from the start
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) of the region.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) When completing a command, userspace sets rsp.scsi_status, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) rsp.sense_buffer if necessary. Userspace then increments
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) kernel via the UIO method, a 4-byte write to the file descriptor.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) If TCMU_MAILBOX_FLAG_CAP_OOOC is set for mailbox->flags, kernel is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) capable of handling out-of-order completions. In this case, userspace can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) handle command in different order other than original. Since kernel would
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) still process the commands in the same order it appeared in the command
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) ring, userspace need to update the cmd->id when completing the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) command(a.k.a steal the original command's entry).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) When the opcode is PAD, userspace only updates cmd_tail as above --
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) it's a no-op. (The kernel inserts PAD entries to ensure each CMD entry
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) is contiguous within the command ring.)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) More opcodes may be added in the future. If userspace encounters an
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) opcode it does not handle, it must set UNKNOWN_OP bit (bit 0) in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) hdr.uflags, update cmd_tail, and proceed with processing additional
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) commands, if any.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) The Data Area
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) -------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) This is shared-memory space after the command ring. The organization
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) of this area is not defined in the TCMU interface, and userspace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) should access only the parts referenced by pending iovs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) Device Discovery
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212) ----------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) Other devices may be using UIO besides TCMU. Unrelated user processes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215) may also be handling different sets of TCMU devices. TCMU userspace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) processes must find their devices by scanning sysfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) class/uio/uio*/name. For TCMU devices, these names will be of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218) format::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220) tcm-user/<hba_num>/<device_name>/<subtype>/<path>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) where "tcm-user" is common for all TCMU-backed UIO devices. <hba_num>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) and <device_name> allow userspace to find the device's path in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224) kernel target's configfs tree. Assuming the usual mount point, it is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) found at::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) /sys/kernel/config/target/core/user_<hba_num>/<device_name>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) This location contains attributes such as "hw_block_size", that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) userspace needs to know for correct operation.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) <subtype> will be a userspace-process-unique string to identify the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233) TCMU device as expecting to be backed by a certain handler, and <path>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234) will be an additional handler-specific string for the user process to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) configure the device, if needed. The name cannot contain ':', due to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236) LIO limitations.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) For all devices so discovered, the user handler opens /dev/uioX and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) calls mmap()::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241) mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) where size must be equal to the value read from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) /sys/class/uio/uioX/maps/map0/size.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) Device Events
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) -------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) If a new device is added or removed, a notification will be broadcast
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) over netlink, using a generic netlink family name of "TCM-USER" and a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252) multicast group named "config". This will include the UIO name as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) described in the previous section, as well as the UIO minor
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) number. This should allow userspace to identify both the UIO device and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) the LIO device, so that after determining the device is supported
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256) (based on subtype) it can take the appropriate action.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) Other contingencies
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260) -------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) Userspace handler process never attaches:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) - TCMU will post commands, and then abort them after a timeout period
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) (30 seconds.)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) Userspace handler process is killed:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269) - It is still possible to restart and re-connect to TCMU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) devices. Command ring is preserved. However, after the timeout period,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271) the kernel will abort pending tasks.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273) Userspace handler process hangs:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) - The kernel will abort pending tasks after a timeout period.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277) Userspace handler process is malicious:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279) - The process can trivially break the handling of devices it controls,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280) but should not be able to access kernel memory outside its shared
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281) memory areas.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284) Writing a user pass-through handler (with example code)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285) =======================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287) A user process handing a TCMU device must support the following:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289) a) Discovering and configuring TCMU uio devices
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) b) Waiting for events on the device(s)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291) c) Managing the command ring: Parsing operations and commands,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292) performing work as needed, setting response fields (scsi_status and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293) possibly sense_buffer), updating cmd_tail, and notifying the kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294) that work has been finished
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296) First, consider instead writing a plugin for tcmu-runner. tcmu-runner
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297) implements all of this, and provides a higher-level API for plugin
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298) authors.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300) TCMU is designed so that multiple unrelated processes can manage TCMU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301) devices separately. All handlers should make sure to only open their
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302) devices, based opon a known subtype string.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) a) Discovering and configuring TCMU UIO devices::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306) /* error checking omitted for brevity */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308) int fd, dev_fd;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309) char buf[256];
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310) unsigned long long map_len;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311) void *map;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313) fd = open("/sys/class/uio/uio0/name", O_RDONLY);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314) ret = read(fd, buf, sizeof(buf));
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315) close(fd);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318) /* we only want uio devices whose name is a format we expect */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319) if (strncmp(buf, "tcm-user", 8))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320) exit(-1);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322) /* Further checking for subtype also needed here */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324) fd = open(/sys/class/uio/%s/maps/map0/size, O_RDONLY);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325) ret = read(fd, buf, sizeof(buf));
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326) close(fd);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327) str_buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329) map_len = strtoull(buf, NULL, 0);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331) dev_fd = open("/dev/uio0", O_RDWR);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332) map = mmap(NULL, map_len, PROT_READ|PROT_WRITE, MAP_SHARED, dev_fd, 0);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335) b) Waiting for events on the device(s)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) while (1) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338) char buf[4];
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340) int ret = read(dev_fd, buf, 4); /* will block */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342) handle_device_events(dev_fd, map);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346) c) Managing the command ring::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348) #include <linux/target_core_user.h>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) int handle_device_events(int fd, void *map)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352) struct tcmu_mailbox *mb = map;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353) struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354) int did_some_work = 0;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) /* Process events from cmd ring until we catch up with cmd_head */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357) while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359) if (tcmu_hdr_get_op(ent->hdr.len_op) == TCMU_OP_CMD) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360) uint8_t *cdb = (void *)mb + ent->req.cdb_off;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361) bool success = true;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363) /* Handle command here. */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364) printf("SCSI opcode: 0x%x\n", cdb[0]);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366) /* Set response fields */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367) if (success)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368) ent->rsp.scsi_status = SCSI_NO_SENSE;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369) else {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370) /* Also fill in rsp->sense_buffer here */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) ent->rsp.scsi_status = SCSI_CHECK_CONDITION;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374) else if (tcmu_hdr_get_op(ent->hdr.len_op) != TCMU_OP_PAD) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375) /* Tell the kernel we didn't handle unknown opcodes */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376) ent->hdr.uflags |= TCMU_UFLAG_UNKNOWN_OP;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378) else {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379) /* Do nothing for PAD entries except update cmd_tail */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 380) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 381)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 382) /* update cmd_tail */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 383) mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 384) ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 385) did_some_work = 1;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 386) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 387)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 388) /* Notify the kernel that work has been finished */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 389) if (did_some_work) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 390) uint32_t buf = 0;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 391)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 392) write(fd, &buf, 4);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 393) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 394)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 395) return 0;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 396) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 397)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 398)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 399) A final note
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 400) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 401)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 402) Please be careful to return codes as defined by the SCSI
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 403) specifications. These are different than some values defined in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 404) scsi/scsi.h include file. For example, CHECK CONDITION's status code
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 405) is 2, not 1.
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags