^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) ====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) Userspace MAD access
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) Device files
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) Each port of each InfiniBand device has a "umad" device and an
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) "issm" device attached. For example, a two-port HCA will have two
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) umad devices and two issm devices, while a switch will have one
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) device of each type (for switch port 0).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) Creating MAD agents
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) ===================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) A MAD agent can be created by filling in a struct ib_user_mad_reg_req
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) and then calling the IB_USER_MAD_REGISTER_AGENT ioctl on a file
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) descriptor for the appropriate device file. If the registration
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) request succeeds, a 32-bit id will be returned in the structure.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) For example::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) struct ib_user_mad_reg_req req = { /* ... */ };
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) ret = ioctl(fd, IB_USER_MAD_REGISTER_AGENT, (char *) &req);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) if (!ret)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) my_agent = req.id;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) else
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) perror("agent register");
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) Agents can be unregistered with the IB_USER_MAD_UNREGISTER_AGENT
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) ioctl. Also, all agents registered through a file descriptor will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) be unregistered when the descriptor is closed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) 2014
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) a new registration ioctl is now provided which allows additional
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) fields to be provided during registration.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) Users of this registration call are implicitly setting the use of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) pkey_index (see below).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) Receiving MADs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) ==============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) MADs are received using read(). The receive side now supports
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) RMPP. The buffer passed to read() must be at least one
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) struct ib_user_mad + 256 bytes. For example:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) If the buffer passed is not large enough to hold the received
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) MAD (RMPP), the errno is set to ENOSPC and the length of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) buffer needed is set in mad.length.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) Example for normal MAD (non RMPP) reads::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) struct ib_user_mad *mad;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) mad = malloc(sizeof *mad + 256);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) ret = read(fd, mad, sizeof *mad + 256);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) if (ret != sizeof mad + 256) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) perror("read");
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) free(mad);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) Example for RMPP reads::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) struct ib_user_mad *mad;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) mad = malloc(sizeof *mad + 256);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) ret = read(fd, mad, sizeof *mad + 256);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) if (ret == -ENOSPC)) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) length = mad.length;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) free(mad);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) mad = malloc(sizeof *mad + length);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) ret = read(fd, mad, sizeof *mad + length);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) if (ret < 0) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) perror("read");
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) free(mad);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) In addition to the actual MAD contents, the other struct ib_user_mad
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) fields will be filled in with information on the received MAD. For
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) example, the remote LID will be in mad.lid.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) If a send times out, a receive will be generated with mad.status set
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) to ETIMEDOUT. Otherwise when a MAD has been successfully received,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) mad.status will be 0.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) poll()/select() may be used to wait until a MAD can be read.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) Sending MADs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) MADs are sent using write(). The agent ID for sending should be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) filled into the id field of the MAD, the destination LID should be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) filled into the lid field, and so on. The send side does support
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) RMPP so arbitrary length MAD can be sent. For example::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) struct ib_user_mad *mad;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) mad = malloc(sizeof *mad + mad_length);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) /* fill in mad->data */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) mad->hdr.id = my_agent; /* req.id from agent registration */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) mad->hdr.lid = my_dest; /* in network byte order... */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) /* etc. */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) ret = write(fd, &mad, sizeof *mad + mad_length);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) if (ret != sizeof *mad + mad_length)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) perror("write");
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) Transaction IDs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) ===============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) Users of the umad devices can use the lower 32 bits of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) transaction ID field (that is, the least significant half of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) field in network byte order) in MADs being sent to match
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) request/response pairs. The upper 32 bits are reserved for use by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) the kernel and will be overwritten before a MAD is sent.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) P_Key Index Handling
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) ====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) The old ib_umad interface did not allow setting the P_Key index for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) MADs that are sent and did not provide a way for obtaining the P_Key
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) index of received MADs. A new layout for struct ib_user_mad_hdr
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) with a pkey_index member has been defined; however, to preserve binary
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) compatibility with older applications, this new layout will not be used
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) unless one of IB_USER_MAD_ENABLE_PKEY or IB_USER_MAD_REGISTER_AGENT2 ioctl's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) are called before a file descriptor is used for anything else.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) In September 2008, the IB_USER_MAD_ABI_VERSION will be incremented
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) to 6, the new layout of struct ib_user_mad_hdr will be used by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) default, and the IB_USER_MAD_ENABLE_PKEY ioctl will be removed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) Setting IsSM Capability Bit
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) ===========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) To set the IsSM capability bit for a port, simply open the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) corresponding issm device file. If the IsSM bit is already set,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) then the open call will block until the bit is cleared (or return
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) immediately with errno set to EAGAIN if the O_NONBLOCK flag is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) passed to open()). The IsSM bit will be cleared when the issm file
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) is closed. No read, write or other operations can be performed on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) the issm file.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) /dev files
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) ==========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) To create the appropriate character device files automatically with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) udev, a rule like::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) KERNEL=="umad*", NAME="infiniband/%k"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) KERNEL=="issm*", NAME="infiniband/%k"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) can be used. This will create device nodes named::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) /dev/infiniband/umad0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) /dev/infiniband/issm0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) for the first port, and so on. The InfiniBand device and port
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) associated with these devices can be determined from the files::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) /sys/class/infiniband_mad/umad0/ibdev
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) /sys/class/infiniband_mad/umad0/port
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) and::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) /sys/class/infiniband_mad/issm0/ibdev
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) /sys/class/infiniband_mad/issm0/port
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags