^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) What: /sys/class/rtrs-client
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) Description: When a user of RTRS API creates a new session, a directory entry with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) the name of that session is created under /sys/class/rtrs-client/<session-name>/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) What: /sys/class/rtrs-client/<session-name>/add_path
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) Description: RW, adds a new path (connection) to an existing session. Expected format is the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) following::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) <[source addr,]destination addr>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) *addr ::= [ ip:<ipv4|ipv6> | gid:<gid> ]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) What: /sys/class/rtrs-client/<session-name>/max_reconnect_attempts
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) Description: Maximum number reconnect attempts the client should make before giving up
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) after connection breaks unexpectedly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) What: /sys/class/rtrs-client/<session-name>/mp_policy
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) Description: Multipath policy specifies which path should be selected on each IO:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) round-robin (0):
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) select path in per CPU round-robin manner.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) min-inflight (1):
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) select path with minimum inflights.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) What: /sys/class/rtrs-client/<session-name>/paths/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) Description: Each path belonging to a given session is listed here by its source and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) destination address. When a new path is added to a session by writing to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) the "add_path" entry, a directory <src@dst> is created.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/state
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) Description: RO, Contains "connected" if the session is connected to the peer and fully
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) functional. Otherwise the file contains "disconnected"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/reconnect
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) Description: Write "1" to the file in order to reconnect the path.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) Operation is blocking and returns 0 if reconnect was successful.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/disconnect
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) Description: Write "1" to the file in order to disconnect the path.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) Operation blocks until RTRS path is disconnected.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/remove_path
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) Description: Write "1" to the file in order to disconnected and remove the path
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) from the session. Operation blocks until the path is disconnected
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) and removed from the session.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/hca_name
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) Description: RO, Contains the the name of HCA the connection established on.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/hca_port
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) Description: RO, Contains the port number of active port traffic is going through.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/src_addr
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) Description: RO, Contains the source address of the path
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/dst_addr
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) Description: RO, Contains the destination address of the path
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/stats/reset_all
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) Description: RW, Read will return usage help, write 0 will clear all the statistics.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/stats/cpu_migration
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) Description: RTRS expects that each HCA IRQ is pinned to a separate CPU. If it's
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) not the case, the processing of an I/O response could be processed on a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) different CPU than where it was originally submitted. This file shows
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) how many interrupts where generated on a non expected CPU.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) "from:"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) is the CPU on which the IRQ was expected, but not generated.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) "to:"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) is the CPU on which the IRQ was generated, but not expected.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/stats/reconnects
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) Description: Contains 2 unsigned int values, the first one records number of successful
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) reconnects in the path lifetime, the second one records number of failed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) reconnects in the path lifetime.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/stats/rdma
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) Date: Feb 2020
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) KernelVersion: 5.7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) Description: Contains statistics regarding rdma operations and inflight operations.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) The output consists of 6 values::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) <read-count> <read-total-size> <write-count> \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) <write-total-size> <inflights> <failovered>