^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) ********************************
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) RDMA Network Block Device (RNBD)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ********************************
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) Introduction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) ------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) RNBD (RDMA Network Block Device) is a pair of kernel modules
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) (client and server) that allow for remote access of a block device on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) the server over RTRS protocol using the RDMA (InfiniBand, RoCE, iWARP)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) transport. After being mapped, the remote block devices can be accessed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) on the client side as local block devices.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) I/O is transferred between client and server by the RTRS transport
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) modules. The administration of RNBD and RTRS modules is done via
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) sysfs entries.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) Requirements
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) ------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) RTRS kernel modules
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) Quick Start
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) -----------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) Server side:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) # modprobe rnbd_server
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) Client side:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) # modprobe rnbd_client
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) # echo "sessname=blya path=ip:10.50.100.66 device_path=/dev/ram0" > \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) /sys/devices/virtual/rnbd-client/ctl/map_device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) Where "sessname=" is a session name, a string to identify the session
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) on client and on server sides; "path=" is a destination IP address or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) a pair of a source and a destination IPs, separated by comma. Multiple
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) "path=" options can be specified in order to use multipath (see RTRS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) description for details); "device_path=" is the block device to be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) mapped from the server side. After the session to the server machine is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) established, the mapped device will appear on the client side under
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) /dev/rnbd<N>.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) RNBD-Server Module Parameters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) =============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) dev_search_path
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) ---------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) When a device is mapped from the client, the server generates the path
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) to the block device on the server side by concatenating dev_search_path
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) and the "device_path" that was specified in the map_device operation.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) The default dev_search_path is: "/".
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) dev_search_path option can also contain %SESSNAME% in order to provide
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) different device namespaces for different sessions. See "device_path"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) option for details.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) ============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) Protocol (rnbd/rnbd-proto.h)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) ============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) 1. Before mapping first device from a given server, client sends an
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) RNBD_MSG_SESS_INFO to the server. Server responds with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) RNBD_MSG_SESS_INFO_RSP. Currently the messages only contain the protocol
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) version for backward compatibility.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) 2. Client requests to open a device by sending RNBD_MSG_OPEN message. This
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) contains the path to the device and access mode (read-only or writable).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) Server responds to the message with RNBD_MSG_OPEN_RSP. This contains
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) a 32 bit device id to be used for IOs and device "geometry" related
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) information: side, max_hw_sectors, etc.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) 3. Client attaches RNBD_MSG_IO to each IO message send to a device. This
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) message contains device id, provided by server in his rnbd_msg_open_rsp,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) sector to be accessed, read-write flags and bi_size.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) 4. Client closes a device by sending RNBD_MSG_CLOSE which contains only the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) device id provided by the server.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) =========================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) Contributors List(in alphabetical order)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) =========================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) Danil Kipnis <danil.kipnis@profitbricks.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) Fabian Holler <mail@fholler.de>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) Jack Wang <jinpu.wang@profitbricks.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) Kleber Souza <kleber.souza@profitbricks.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) Lutz Pogrell <lutz.pogrell@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) Milind Dumbare <Milind.dumbare@gmail.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) Roman Penyaev <roman.penyaev@profitbricks.com>