^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) ==========================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) Xillybus driver for generic FPGA interface
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ==========================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) :Author: Eli Billauer, Xillybus Ltd. (http://xillybus.com)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) :Email: eli.billauer@gmail.com or as advertised on Xillybus' site.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) .. Contents:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) - Introduction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) -- Background
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) -- Xillybus Overview
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) - Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) -- User interface
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) -- Synchronization
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) -- Seekable pipes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) - Internals
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) -- Source code organization
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) -- Pipe attributes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) -- Host never reads from the FPGA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) -- Channels, pipes, and the message channel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) -- Data streaming
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) -- Data granularity
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) -- Probing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) -- Buffer allocation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) -- The "nonempty" message (supporting poll)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) Introduction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) Background
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) ----------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) An FPGA (Field Programmable Gate Array) is a piece of logic hardware, which
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) can be programmed to become virtually anything that is usually found as a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) dedicated chipset: For instance, a display adapter, network interface card,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) or even a processor with its peripherals. FPGAs are the LEGO of hardware:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) Based upon certain building blocks, you make your own toys the way you like
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) them. It's usually pointless to reimplement something that is already
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) available on the market as a chipset, so FPGAs are mostly used when some
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) special functionality is needed, and the production volume is relatively low
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) (hence not justifying the development of an ASIC).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) The challenge with FPGAs is that everything is implemented at a very low
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) level, even lower than assembly language. In order to allow FPGA designers to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) focus on their specific project, and not reinvent the wheel over and over
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) again, pre-designed building blocks, IP cores, are often used. These are the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) FPGA parallels of library functions. IP cores may implement certain
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) mathematical functions, a functional unit (e.g. a USB interface), an entire
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) processor (e.g. ARM) or anything that might come handy. Think of them as a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) building block, with electrical wires dangling on the sides for connection to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) other blocks.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) One of the daunting tasks in FPGA design is communicating with a fullblown
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) operating system (actually, with the processor running it): Implementing the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) low-level bus protocol and the somewhat higher-level interface with the host
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) (registers, interrupts, DMA etc.) is a project in itself. When the FPGA's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) function is a well-known one (e.g. a video adapter card, or a NIC), it can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) make sense to design the FPGA's interface logic specifically for the project.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) A special driver is then written to present the FPGA as a well-known interface
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) to the kernel and/or user space. In that case, there is no reason to treat the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) FPGA differently than any device on the bus.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) It's however common that the desired data communication doesn't fit any well-
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) known peripheral function. Also, the effort of designing an elegant
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) abstraction for the data exchange is often considered too big. In those cases,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) a quicker and possibly less elegant solution is sought: The driver is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) effectively written as a user space program, leaving the kernel space part
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) with just elementary data transport. This still requires designing some
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) interface logic for the FPGA, and write a simple ad-hoc driver for the kernel.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) Xillybus Overview
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) -----------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) Xillybus is an IP core and a Linux driver. Together, they form a kit for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) elementary data transport between an FPGA and the host, providing pipe-like
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) data streams with a straightforward user interface. It's intended as a low-
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) effort solution for mixed FPGA-host projects, for which it makes sense to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) have the project-specific part of the driver running in a user-space program.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) Since the communication requirements may vary significantly from one FPGA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) project to another (the number of data pipes needed in each direction and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) their attributes), there isn't one specific chunk of logic being the Xillybus
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) IP core. Rather, the IP core is configured and built based upon a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) specification given by its end user.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) Xillybus presents independent data streams, which resemble pipes or TCP/IP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) communication to the user. At the host side, a character device file is used
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) just like any pipe file. On the FPGA side, hardware FIFOs are used to stream
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) the data. This is contrary to a common method of communicating through fixed-
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) sized buffers (even though such buffers are used by Xillybus under the hood).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) There may be more than a hundred of these streams on a single IP core, but
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) also no more than one, depending on the configuration.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) In order to ease the deployment of the Xillybus IP core, it contains a simple
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) data structure which completely defines the core's configuration. The Linux
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) driver fetches this data structure during its initialization process, and sets
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) up the DMA buffers and character devices accordingly. As a result, a single
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) driver is used to work out of the box with any Xillybus IP core.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) The data structure just mentioned should not be confused with PCI's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) configuration space or the Flattened Device Tree.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) Usage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) =====
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) User interface
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) --------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) On the host, all interface with Xillybus is done through /dev/xillybus_*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) device files, which are generated automatically as the drivers loads. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) names of these files depend on the IP core that is loaded in the FPGA (see
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) Probing below). To communicate with the FPGA, open the device file that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) corresponds to the hardware FIFO you want to send data or receive data from,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) and use plain write() or read() calls, just like with a regular pipe. In
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) particular, it makes perfect sense to go::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) $ cat mydata > /dev/xillybus_thisfifo
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) $ cat /dev/xillybus_thatfifo > hisdata
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) possibly pressing CTRL-C as some stage, even though the xillybus_* pipes have
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) the capability to send an EOF (but may not use it).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) The driver and hardware are designed to behave sensibly as pipes, including:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) * Supporting non-blocking I/O (by setting O_NONBLOCK on open() ).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) * Supporting poll() and select().
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) * Being bandwidth efficient under load (using DMA) but also handle small
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) pieces of data sent across (like TCP/IP) by autoflushing.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) A device file can be read only, write only or bidirectional. Bidirectional
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) device files are treated like two independent pipes (except for sharing a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) "channel" structure in the implementation code).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) Synchronization
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) ---------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) Xillybus pipes are configured (on the IP core) to be either synchronous or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) asynchronous. For a synchronous pipe, write() returns successfully only after
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) some data has been submitted and acknowledged by the FPGA. This slows down
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) bulk data transfers, and is nearly impossible for use with streams that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) require data at a constant rate: There is no data transmitted to the FPGA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) between write() calls, in particular when the process loses the CPU.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) When a pipe is configured asynchronous, write() returns if there was enough
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) room in the buffers to store any of the data in the buffers.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) For FPGA to host pipes, asynchronous pipes allow data transfer from the FPGA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) as soon as the respective device file is opened, regardless of if the data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) has been requested by a read() call. On synchronous pipes, only the amount
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) of data requested by a read() call is transmitted.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) In summary, for synchronous pipes, data between the host and FPGA is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) transmitted only to satisfy the read() or write() call currently handled
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) by the driver, and those calls wait for the transmission to complete before
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) returning.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) Note that the synchronization attribute has nothing to do with the possibility
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) that read() or write() completes less bytes than requested. There is a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) separate configuration flag ("allowpartial") that determines whether such a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) partial completion is allowed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) Seekable pipes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) --------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) A synchronous pipe can be configured to have the stream's position exposed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) to the user logic at the FPGA. Such a pipe is also seekable on the host API.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) With this feature, a memory or register interface can be attached on the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) FPGA side to the seekable stream. Reading or writing to a certain address in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) the attached memory is done by seeking to the desired address, and calling
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) read() or write() as required.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) Internals
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) =========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) Source code organization
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) ------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) The Xillybus driver consists of a core module, xillybus_core.c, and modules
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) that depend on the specific bus interface (xillybus_of.c and xillybus_pcie.c).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) The bus specific modules are those probed when a suitable device is found by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) the kernel. Since the DMA mapping and synchronization functions, which are bus
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) dependent by their nature, are used by the core module, a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) xilly_endpoint_hardware structure is passed to the core module on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) initialization. This structure is populated with pointers to wrapper functions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) which execute the DMA-related operations on the bus.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) Pipe attributes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) ---------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) Each pipe has a number of attributes which are set when the FPGA component
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) (IP core) is built. They are fetched from the IDT (the data structure which
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) defines the core's configuration, see Probing below) by xilly_setupchannels()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) in xillybus_core.c as follows:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) * is_writebuf: The pipe's direction. A non-zero value means it's an FPGA to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) host pipe (the FPGA "writes").
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) * channelnum: The pipe's identification number in communication between the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) host and FPGA.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) * format: The underlying data width. See Data Granularity below.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212) * allowpartial: A non-zero value means that a read() or write() (whichever
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) applies) may return with less than the requested number of bytes. The common
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) choice is a non-zero value, to match standard UNIX behavior.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) * synchronous: A non-zero value means that the pipe is synchronous. See
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) Synchronization above.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) * bufsize: Each DMA buffer's size. Always a power of two.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) * bufnum: The number of buffers allocated for this pipe. Always a power of two.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) * exclusive_open: A non-zero value forces exclusive opening of the associated
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224) device file. If the device file is bidirectional, and already opened only in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) one direction, the opposite direction may be opened once.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) * seekable: A non-zero value indicates that the pipe is seekable. See
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228) Seekable pipes above.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) * supports_nonempty: A non-zero value (which is typical) indicates that the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231) hardware will send the messages that are necessary to support select() and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) poll() for this pipe.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234) Host never reads from the FPGA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) ------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237) Even though PCI Express is hotpluggable in general, a typical motherboard
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) doesn't expect a card to go away all of the sudden. But since the PCIe card
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) is based upon reprogrammable logic, a sudden disappearance from the bus is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240) quite likely as a result of an accidental reprogramming of the FPGA while the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241) host is up. In practice, nothing happens immediately in such a situation. But
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) if the host attempts to read from an address that is mapped to the PCI Express
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) device, that leads to an immediate freeze of the system on some motherboards,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) even though the PCIe standard requires a graceful recovery.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246) In order to avoid these freezes, the Xillybus driver refrains completely from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) reading from the device's register space. All communication from the FPGA to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) the host is done through DMA. In particular, the Interrupt Service Routine
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249) doesn't follow the common practice of checking a status register when it's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) invoked. Rather, the FPGA prepares a small buffer which contains short
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) messages, which inform the host what the interrupt was about.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) This mechanism is used on non-PCIe buses as well for the sake of uniformity.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256) Channels, pipes, and the message channel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) ----------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) Each of the (possibly bidirectional) pipes presented to the user is allocated
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260) a data channel between the FPGA and the host. The distinction between channels
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) and pipes is necessary only because of channel 0, which is used for interrupt-
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) related messages from the FPGA, and has no pipe attached to it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) Data streaming
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) --------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) Even though a non-segmented data stream is presented to the user at both
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268) sides, the implementation relies on a set of DMA buffers which is allocated
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269) for each channel. For the sake of illustration, let's take the FPGA to host
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) direction: As data streams into the respective channel's interface in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271) FPGA, the Xillybus IP core writes it to one of the DMA buffers. When the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272) buffer is full, the FPGA informs the host about that (appending a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273) XILLYMSG_OPCODE_RELEASEBUF message channel 0 and sending an interrupt if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274) necessary). The host responds by making the data available for reading through
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) the character device. When all data has been read, the host writes on the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276) FPGA's buffer control register, allowing the buffer's overwriting. Flow
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277) control mechanisms exist on both sides to prevent underflows and overflows.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279) This is not good enough for creating a TCP/IP-like stream: If the data flow
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280) stops momentarily before a DMA buffer is filled, the intuitive expectation is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281) that the partial data in buffer will arrive anyhow, despite the buffer not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282) being completed. This is implemented by adding a field in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283) XILLYMSG_OPCODE_RELEASEBUF message, through which the FPGA informs not just
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284) which buffer is submitted, but how much data it contains.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286) But the FPGA will submit a partially filled buffer only if directed to do so
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287) by the host. This situation occurs when the read() method has been blocking
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288) for XILLY_RX_TIMEOUT jiffies (currently 10 ms), after which the host commands
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289) the FPGA to submit a DMA buffer as soon as it can. This timeout mechanism
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) balances between bus bandwidth efficiency (preventing a lot of partially
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291) filled buffers being sent) and a latency held fairly low for tails of data.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293) A similar setting is used in the host to FPGA direction. The handling of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294) partial DMA buffers is somewhat different, though. The user can tell the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295) driver to submit all data it has in the buffers to the FPGA, by issuing a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296) write() with the byte count set to zero. This is similar to a flush request,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297) but it doesn't block. There is also an autoflushing mechanism, which triggers
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298) an equivalent flush roughly XILLY_RX_TIMEOUT jiffies after the last write().
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299) This allows the user to be oblivious about the underlying buffering mechanism
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300) and yet enjoy a stream-like interface.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302) Note that the issue of partial buffer flushing is irrelevant for pipes having
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303) the "synchronous" attribute nonzero, since synchronous pipes don't allow data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) to lay around in the DMA buffers between read() and write() anyhow.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306) Data granularity
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307) ----------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309) The data arrives or is sent at the FPGA as 8, 16 or 32 bit wide words, as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310) configured by the "format" attribute. Whenever possible, the driver attempts
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311) to hide this when the pipe is accessed differently from its natural alignment.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312) For example, reading single bytes from a pipe with 32 bit granularity works
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313) with no issues. Writing single bytes to pipes with 16 or 32 bit granularity
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314) will also work, but the driver can't send partially completed words to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315) FPGA, so the transmission of up to one word may be held until it's fully
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) occupied with user data.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318) This somewhat complicates the handling of host to FPGA streams, because
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319) when a buffer is flushed, it may contain up to 3 bytes don't form a word in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320) the FPGA, and hence can't be sent. To prevent loss of data, these leftover
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321) bytes need to be moved to the next buffer. The parts in xillybus_core.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322) that mention "leftovers" in some way are related to this complication.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324) Probing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325) -------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327) As mentioned earlier, the number of pipes that are created when the driver
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328) loads and their attributes depend on the Xillybus IP core in the FPGA. During
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329) the driver's initialization, a blob containing configuration info, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330) Interface Description Table (IDT), is sent from the FPGA to the host. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331) bootstrap process is done in three phases:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333) 1. Acquire the length of the IDT, so a buffer can be allocated for it. This
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334) is done by sending a quiesce command to the device, since the acknowledge
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335) for this command contains the IDT's buffer length.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) 2. Acquire the IDT itself.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339) 3. Create the interfaces according to the IDT.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341) Buffer allocation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342) -----------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344) In order to simplify the logic that prevents illegal boundary crossings of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345) PCIe packets, the following rule applies: If a buffer is smaller than 4kB,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346) it must not cross a 4kB boundary. Otherwise, it must be 4kB aligned. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347) xilly_setupchannels() functions allocates these buffers by requesting whole
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348) pages from the kernel, and diving them into DMA buffers as necessary. Since
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349) all buffers' sizes are powers of two, it's possible to pack any set of such
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) buffers, with a maximal waste of one page of memory.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352) All buffers are allocated when the driver is loaded. This is necessary,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353) since large continuous physical memory segments are sometimes requested,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354) which are more likely to be available when the system is freshly booted.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) The allocation of buffer memory takes place in the same order they appear in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357) the IDT. The driver relies on a rule that the pipes are sorted with decreasing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358) buffer size in the IDT. If a requested buffer is larger or equal to a page,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359) the necessary number of pages is requested from the kernel, and these are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360) used for this buffer. If the requested buffer is smaller than a page, one
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361) single page is requested from the kernel, and that page is partially used.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362) Or, if there already is a partially used page at hand, the buffer is packed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363) into that page. It can be shown that all pages requested from the kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364) (except possibly for the last) are 100% utilized this way.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366) The "nonempty" message (supporting poll)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367) ----------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369) In order to support the "poll" method (and hence select() ), there is a small
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370) catch regarding the FPGA to host direction: The FPGA may have filled a DMA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) buffer with some data, but not submitted that buffer. If the host waited for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372) the buffer's submission by the FPGA, there would be a possibility that the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373) FPGA side has sent data, but a select() call would still block, because the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374) host has not received any notification about this. This is solved with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375) XILLYMSG_OPCODE_NONEMPTY messages sent by the FPGA when a channel goes from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376) completely empty to containing some data.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378) These messages are used only to support poll() and select(). The IP core can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379) be configured not to send them for a slight reduction of bandwidth.
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags