^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) ===================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) Setting up NFS/RDMA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ===================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) :Author:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) NetApp and Open Grid Computing (May 29, 2008)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) .. warning::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) This document is probably obsolete.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) Overview
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) ========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) This document describes how to install and setup the Linux NFS/RDMA client
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) and server software.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) The NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA server
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) was first included in the following release, Linux 2.6.25.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) In our testing, we have obtained excellent performance results (full 10Gbit
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) wire bandwidth at minimal client CPU) under many workloads. The code passes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) the full Connectathon test suite and operates over both Infiniband and iWARP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) RDMA adapters.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) Getting Help
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) If you get stuck, you can ask questions on the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) nfs-rdma-devel@lists.sourceforge.net mailing list.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) Installation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) These instructions are a step by step guide to building a machine for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) use with NFS/RDMA.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) - Install an RDMA device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) Any device supported by the drivers in drivers/infiniband/hw is acceptable.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) Testing has been performed using several Mellanox-based IB cards, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) Ammasso AMS1100 iWARP adapter, and the Chelsio cxgb3 iWARP adapter.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) - Install a Linux distribution and tools
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) The first kernel release to contain both the NFS/RDMA client and server was
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) Linux 2.6.25 Therefore, a distribution compatible with this and subsequent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) Linux kernel release should be installed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) The procedures described in this document have been tested with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) distributions from Red Hat's Fedora Project (http://fedora.redhat.com/).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) - Install nfs-utils-1.1.2 or greater on the client
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) An NFS/RDMA mount point can be obtained by using the mount.nfs command in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) version with support for NFS/RDMA mounts, but for various reasons we
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) recommend using nfs-utils-1.1.2 or greater). To see which version of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) mount.nfs you are using, type:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) $ /sbin/mount.nfs -V
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) If the version is less than 1.1.2 or the command does not exist,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) you should install the latest version of nfs-utils.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) Download the latest package from: https://www.kernel.org/pub/linux/utils/nfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) Uncompress the package and follow the installation instructions.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) If you will not need the idmapper and gssd executables (you do not need
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) these to create an NFS/RDMA enabled mount command), the installation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) process can be simplified by disabling these features when running
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) configure:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) $ ./configure --disable-gss --disable-nfsv4
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) To build nfs-utils you will need the tcp_wrappers package installed. For
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) more information on this see the package's README and INSTALL files.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) After building the nfs-utils package, there will be a mount.nfs binary in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) the utils/mount directory. This binary can be used to initiate NFS v2, v3,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) or v4 mounts. To initiate a v4 mount, the binary must be called
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) mount.nfs4. The standard technique is to create a symlink called
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) mount.nfs4 to mount.nfs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) This mount.nfs binary should be installed at /sbin/mount.nfs as follows:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) In this location, mount.nfs will be invoked automatically for NFS mounts
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) by the system mount command.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) .. note::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) on the NFS client machine. You do not need this specific version of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) nfs-utils on the server. Furthermore, only the mount.nfs command from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) nfs-utils-1.1.2 is needed on the client.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) - Install a Linux kernel with NFS/RDMA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) The NFS/RDMA client and server are both included in the mainline Linux
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) kernel version 2.6.25 and later. This and other versions of the Linux
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) kernel can be found at: https://www.kernel.org/pub/linux/kernel/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) Download the sources and place them in an appropriate location.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) - Configure the RDMA stack
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) Make sure your kernel configuration has RDMA support enabled. Under
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) Device Drivers -> InfiniBand support, update the kernel configuration
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) to enable InfiniBand support [NOTE: the option name is misleading. Enabling
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) InfiniBand support is required for all RDMA devices (IB, iWARP, etc.)].
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) Enable the appropriate IB HCA support (mlx4, mthca, ehca, ipath, etc.) or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) iWARP adapter support (amso, cxgb3, etc.).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) If you are using InfiniBand, be sure to enable IP-over-InfiniBand support.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) - Configure the NFS client and server
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) Your kernel configuration must also have NFS file system support and/or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) NFS server support enabled. These and other NFS related configuration
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) options can be found under File Systems -> Network File Systems.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) - Build, install, reboot
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) The NFS/RDMA code will be enabled automatically if NFS and RDMA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) are turned on. The NFS/RDMA client and server are configured via the hidden
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) SUNRPC_XPRT_RDMA config option that depends on SUNRPC and INFINIBAND. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) value of SUNRPC_XPRT_RDMA will be:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) #. N if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) and server will not be built
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) #. M if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) in this case the NFS/RDMA client and server will be built as modules
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) #. Y if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) and server will be built into the kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) Therefore, if you have followed the steps above and turned no NFS and RDMA,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) the NFS/RDMA client and server will be built.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) Build a new kernel, install it, boot it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) Check RDMA and NFS Setup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) ========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) Before configuring the NFS/RDMA software, it is a good idea to test
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) your new kernel to ensure that the kernel is working correctly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) In particular, it is a good idea to verify that the RDMA stack
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) is functioning as expected and standard NFS over TCP/IP and/or UDP/IP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) is working properly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) - Check RDMA Setup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) If you built the RDMA components as modules, load them at
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) card:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) $ modprobe ib_mthca
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) $ modprobe ib_ipoib
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) If you are using InfiniBand, make sure there is a Subnet Manager (SM)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) running on the network. If your IB switch has an embedded SM, you can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) use it. Otherwise, you will need to run an SM, such as OpenSM, on one
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) of your end nodes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) If an SM is running on your network, you should see the following:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) $ cat /sys/class/infiniband/driverX/ports/1/state
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) 4: ACTIVE
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) where driverX is mthca0, ipath5, ehca3, etc.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) To further test the InfiniBand software stack, use IPoIB (this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) assumes you have two IB hosts named host1 and host2):
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) host1$ ip link set dev ib0 up
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) host1$ ip address add dev ib0 a.b.c.x
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) host2$ ip link set dev ib0 up
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) host2$ ip address add dev ib0 a.b.c.y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) host1$ ping a.b.c.y
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) host2$ ping a.b.c.x
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) For other device types, follow the appropriate procedures.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) - Check NFS Setup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) For the NFS components enabled above (client and/or server),
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) test their functionality over standard Ethernet using TCP/IP or UDP/IP.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) NFS/RDMA Setup
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) ==============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) We recommend that you use two machines, one to act as the client and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209) one to act as the server.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) One time configuration:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212) -----------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) - On the server system, configure the /etc/exports file and start the NFS/RDMA server.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) Exports entries with the following formats have been tested::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218) /vol0 192.168.0.47(fsid=0,rw,async,insecure,no_root_squash)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) /vol0 192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) The IP address(es) is(are) the client's IPoIB address for an InfiniBand
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) HCA or the client's iWARP address(es) for an RNIC.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224) .. note::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) The "insecure" option must be used because the NFS/RDMA client does
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226) not use a reserved port.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228) Each time a machine boots:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) --------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231) - Load and configure the RDMA drivers
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233) For InfiniBand using a Mellanox adapter:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237) $ modprobe ib_mthca
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) $ modprobe ib_ipoib
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) $ ip li set dev ib0 up
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240) $ ip addr add dev ib0 a.b.c.d
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) .. note::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) Please use unique addresses for the client and server!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245) - Start the NFS server
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) kernel config), load the RDMA transport module:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252) $ modprobe svcrdma
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) Regardless of how the server was built (module or built-in), start the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) server:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) $ /etc/init.d/nfs start
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) $ service nfs start
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) Instruct the server to listen on the RDMA transport:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271) $ echo rdma 20049 > /proc/fs/nfsd/portlist
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273) - On the client system
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276) kernel config), load the RDMA client module:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280) $ modprobe xprtrdma.ko
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282) Regardless of how the client was built (module or built-in), use this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283) command to mount the NFS/RDMA server:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285) .. code-block:: sh
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287) $ mount -o rdma,port=20049 <IPoIB-server-name-or-address>:/<export> /mnt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289) To verify that the mount is using RDMA, run "cat /proc/mounts" and check
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) the "proto" field for the given mount.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292) Congratulations! You're using NFS/RDMA!
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags