^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) SNMP counter
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) This document explains the meaning of SNMP counters.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7) General IPv4 counters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) =====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) All layer 4 packets and ICMP packets will change these counters, but
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) these counters won't be changed by layer 2 packets (such as STP) or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) ARP packets.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) * IpInReceives
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) Defined in `RFC1213 ipInReceives`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) .. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) The number of packets received by the IP layer. It gets increasing at the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) beginning of ip_rcv function, always be updated together with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) IpExtInOctets. It will be increased even if the packet is dropped
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) later (e.g. due to the IP header is invalid or the checksum is wrong
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) and so on). It indicates the number of aggregated segments after
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) GRO/LRO.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) * IpInDelivers
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) Defined in `RFC1213 ipInDelivers`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) .. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) ICMP and so on. If no one listens on a raw socket, only kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) supported protocols will be delivered, if someone listens on the raw
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) socket, all valid IP packets will be delivered.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) * IpOutRequests
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39) Defined in `RFC1213 ipOutRequests`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) .. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) The number of packets sent via IP layer, for both single cast and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) multicast packets, and would always be updated together with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) IpExtOutOctets.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) * IpExtInOctets and IpExtOutOctets
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) They are Linux kernel extensions, no RFC definitions. Please note,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) RFC1213 indeed defines ifInOctets and ifOutOctets, but they
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) are different things. The ifInOctets and ifOutOctets include the MAC
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) layer header size but IpExtInOctets and IpExtOutOctets don't, they
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) only include the IP layer header and the IP layer data.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) * IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) They indicate the number of four kinds of ECN IP packets, please refer
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) `Explicit Congestion Notification`_ for more details.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) .. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) These 4 counters calculate how many packets received per ECN
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) status. They count the real frame number regardless the LRO/GRO. So
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) for the same packet, you might find that IpInReceives count 1, but
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) IpExtInNoECTPkts counts 2 or more.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) * IpInHdrErrors
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) dropped due to the IP header error. It might happen in both IP input
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) and IP forward paths.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) .. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) * IpInAddrErrors
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) scenarios: (1) The IP address is invalid. (2) The destination IP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) address is not a local address and IP forwarding is not enabled
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) .. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) * IpExtInNoRoutes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) This counter means the packet is dropped when the IP stack receives a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) packet and can't find a route for it from the route table. It might
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) happen when IP forwarding is enabled and the destination IP address is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) not a local address and there is no route for the destination IP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) address.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) * IpInUnknownProtos
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) layer 4 protocol is unsupported by kernel. If an application is using
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) raw socket, kernel will always deliver the packet to the raw socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) and this counter won't be increased.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) .. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) * IpExtInTruncatedPkts
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) For IPv4 packet, it means the actual data size is smaller than the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) "Total Length" field in the IPv4 header.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) * IpInDiscards
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) in the IP receiving path and due to kernel internal reasons (e.g. no
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) enough memory).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) .. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) * IpOutDiscards
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) dropped in the IP sending path and due to kernel internal reasons.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) .. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) * IpOutNoRoutes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) dropped in the IP sending path and no route is found for it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) .. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) ICMP counters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) =============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) * IcmpInMsgs and IcmpOutMsgs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) .. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) .. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) As mentioned in the RFC1213, these two counters include errors, they
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) would be increased even if the ICMP packet has an invalid type. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) ICMP output path will check the header of a raw socket, so the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) IcmpOutMsgs would still be updated if the IP header is constructed by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) a userspace program.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) * ICMP named types
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) | These counters include most of common ICMP types, they are:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) | IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) | IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) | IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) | IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) | IcmpInRedirects: `RFC1213 icmpInRedirects`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) | IcmpInEchos: `RFC1213 icmpInEchos`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) | IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) | IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) | IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) | IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) | IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) | IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) | IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) | IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) | IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) | IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) | IcmpOutEchos: `RFC1213 icmpOutEchos`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) | IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) | IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) | IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) | IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) | IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) .. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) .. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) .. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) .. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) .. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) .. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) .. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) .. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) .. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) .. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) .. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) .. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) .. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) .. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) .. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) .. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) .. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) .. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) .. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) .. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) .. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) .. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) straightforward. The 'In' counter means kernel receives such a packet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) and the 'Out' counter means kernel sends such a packet.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) * ICMP numeric types
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) ICMP type number. These counters track all kinds of ICMP packets. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) ICMP type number definition could be found in the `ICMP parameters`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) document.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) .. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) For example, if the Linux kernel sends an ICMP Echo packet, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) packet, IcmpMsgInType0 would increase 1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) * IcmpInCsumErrors
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212) This counter indicates the checksum of the ICMP packet is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) before updating IcmpMsgInType[N]. If a packet has bad checksum, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215) IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) * IcmpInErrors and IcmpOutErrors
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) .. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) .. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224) When an error occurs in the ICMP packet handler path, these two
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) counters would be updated. The receiving packet path use IcmpInErrors
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226) and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) is increased, IcmpInErrors would always be increased too.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) relationship of the ICMP counters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) ---------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231) The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) are updated at the same time. The sum of IcmpMsgInType[N] plus
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233) IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234) receives an ICMP packet, kernel follows below logic:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236) 1. increase IcmpInMsgs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237) 2. if has any error, update IcmpInErrors and finish the process
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) 3. update IcmpMsgOutType[N]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) 4. handle the packet depending on the type, if has any error, update
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240) IcmpInErrors and finish the process
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) So if all errors occur in step (2), IcmpInMsgs should be equal to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) step (4), IcmpInMsgs should be equal to the sum of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245) IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246) IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) IcmpInErrors.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249) General TCP counters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) ====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) * TcpInSegs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) Defined in `RFC1213 tcpInSegs`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) .. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) The number of packets received by the TCP layer. As mentioned in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258) RFC1213, it includes the packets received in error, such as checksum
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) error, invalid TCP header and so on. Only one error won't be included:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260) if the layer 2 destination address is not the NIC's layer 2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) address. It might happen if the packet is a multicast or broadcast
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) packet, or the NIC is in promiscuous mode. In these situations, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263) packets would be delivered to the TCP layer, but the TCP layer will discard
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) these packets before increasing TcpInSegs. The TcpInSegs counter
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266) counter would only increase 1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268) * TcpOutSegs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) Defined in `RFC1213 tcpOutSegs`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272) .. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274) The number of packets sent by the TCP layer. As mentioned in RFC1213,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) it excludes the retransmitted packets. But it includes the SYN, ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276) and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277) GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278) increase 2.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280) * TcpActiveOpens
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282) Defined in `RFC1213 tcpActiveOpens`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284) .. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286) It means the TCP layer sends a SYN, and come into the SYN-SENT
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287) state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288) increase 1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) * TcpPassiveOpens
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292) Defined in `RFC1213 tcpPassiveOpens`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294) .. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296) It means the TCP layer receives a SYN, replies a SYN+ACK, come into
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297) the SYN-RCVD state.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299) * TcpExtTCPRcvCoalesce
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301) When packets are received by the TCP layer and are not be read by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302) application, the TCP layer will try to merge them. This counter
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303) indicate how many packets are merged in such situation. If GRO is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) enabled, lots of packets would be merged by GRO, these packets
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305) wouldn't be counted to TcpExtTCPRcvCoalesce.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307) * TcpExtTCPAutoCorking
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309) When sending packets, the TCP layer will try to merge small packets to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310) a bigger one. This counter increase 1 for every packet merged in such
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311) situation. Please refer to the LWN article for more details:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312) https://lwn.net/Articles/576263/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314) * TcpExtTCPOrigDataSent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317) explaination below::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319) TCPOrigDataSent: number of outgoing packets with original data (excluding
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320) retransmission but including data-in-SYN). This counter is different from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321) TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322) more useful to track the TCP retransmission rate.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324) * TCPSynRetrans
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326) This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327) explaination below::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329) TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330) retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332) * TCPFastOpenActiveFail
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334) This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335) explaination below::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338) the remote does not accept it or the attempts timed out.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340) .. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342) * TcpExtListenOverflows and TcpExtListenDrops
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344) When kernel receives a SYN from a client, and if the TCP accept queue
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345) is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346) At the same time kernel will also add 1 to TcpExtListenDrops. When a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347) TCP socket is in LISTEN state, and kernel need to drop a packet,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348) kernel would always add 1 to TcpExtListenDrops. So increase
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349) TcpExtListenOverflows would let TcpExtListenDrops increasing at the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) same time, but TcpExtListenDrops would also increase without
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351) TcpExtListenOverflows increasing, e.g. a memory allocation fail would
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352) also let TcpExtListenDrops increase.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354) Note: The above explanation is based on kernel 4.10 or above version, on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355) an old kernel, the TCP stack has different behavior when TCP accept
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) queue is full. On the old kernel, TCP stack won't drop the SYN, it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357) would complete the 3-way handshake. As the accept queue is full, TCP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358) stack will keep the socket in the TCP half-open queue. As it is in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359) half open queue, TCP stack will send SYN+ACK on an exponential backoff
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360) timer, after client replies ACK, TCP stack checks whether the accept
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361) queue is still full, if it is not full, moves the socket to the accept
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362) queue, if it is full, keeps the socket in the half-open queue, at next
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363) time client replies ACK, this socket will get another chance to move
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364) to the accept queue.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367) TCP Fast Open
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368) =============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369) * TcpEstabResets
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) Defined in `RFC1213 tcpEstabResets`_.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373) .. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375) * TcpAttemptFails
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377) Defined in `RFC1213 tcpAttemptFails`_.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379) .. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 380)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 381) * TcpOutRsts
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 382)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 383) Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 384) the 'segments sent containing the RST flag', but in linux kernel, this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 385) couner indicates the segments kerenl tried to send. The sending
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 386) process might be failed due to some errors (e.g. memory alloc failed).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 387)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 388) .. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 389)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 390) * TcpExtTCPSpuriousRtxHostQueues
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 391)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 392) When the TCP stack wants to retransmit a packet, and finds that packet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 393) is not lost in the network, but the packet is not sent yet, the TCP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 394) stack would give up the retransmission and update this counter. It
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 395) might happen if a packet stays too long time in a qdisc or driver
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 396) queue.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 397)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 398) * TcpEstabResets
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 399)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 400) The socket receives a RST packet in Establish or CloseWait state.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 401)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 402) * TcpExtTCPKeepAlive
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 403)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 404) This counter indicates many keepalive packets were sent. The keepalive
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 405) won't be enabled by default. A userspace program could enable it by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 406) setting the SO_KEEPALIVE socket option.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 407)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 408) * TcpExtTCPSpuriousRTOs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 409)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 410) The spurious retransmission timeout detected by the `F-RTO`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 411) algorithm.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 412)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 413) .. _F-RTO: https://tools.ietf.org/html/rfc5682
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 414)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 415) TCP Fast Path
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 416) =============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 417) When kernel receives a TCP packet, it has two paths to handler the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 418) packet, one is fast path, another is slow path. The comment in kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 419) code provides a good explanation of them, I pasted them below::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 420)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 421) It is split into a fast path and a slow path. The fast path is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 422) disabled when:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 423)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 424) - A zero window was announced from us
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 425) - zero window probing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 426) is only handled properly on the slow path.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 427) - Out of order segments arrived.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 428) - Urgent data is expected.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 429) - There is no buffer space left
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 430) - Unexpected TCP flags/window values/header lengths are received
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 431) (detected by checking the TCP header against pred_flags)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 432) - Data is sent in both directions. The fast path only supports pure senders
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 433) or pure receivers (this means either the sequence number or the ack
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 434) value must stay constant)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 435) - Unexpected TCP option.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 436)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 437) Kernel will try to use fast path unless any of the above conditions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 438) are satisfied. If the packets are out of order, kernel will handle
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 439) them in slow path, which means the performance might be not very
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 440) good. Kernel would also come into slow path if the "Delayed ack" is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 441) used, because when using "Delayed ack", the data is sent in both
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 442) directions. When the TCP window scale option is not used, kernel will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 443) try to enable fast path immediately when the connection comes into the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 444) established state, but if the TCP window scale option is used, kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 445) will disable the fast path at first, and try to enable it after kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 446) receives packets.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 447)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 448) * TcpExtTCPPureAcks and TcpExtTCPHPAcks
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 449)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 450) If a packet set ACK flag and has no data, it is a pure ACK packet, if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 451) kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 452) if kernel handles it in the slow path, TcpExtTCPPureAcks will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 453) increase 1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 454)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 455) * TcpExtTCPHPHits
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 456)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 457) If a TCP packet has data (which means it is not a pure ACK packet),
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 458) and this packet is handled in the fast path, TcpExtTCPHPHits will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 459) increase 1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 460)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 461)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 462) TCP abort
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 463) =========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 464) * TcpExtTCPAbortOnData
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 465)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 466) It means TCP layer has data in flight, but need to close the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 467) connection. So TCP layer sends a RST to the other side, indicate the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 468) connection is not closed very graceful. An easy way to increase this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 469) counter is using the SO_LINGER option. Please refer to the SO_LINGER
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 470) section of the `socket man page`_:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 471)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 472) .. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 473)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 474) By default, when an application closes a connection, the close function
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 475) will return immediately and kernel will try to send the in-flight data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 476) async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 477) to a positive number, the close function won't return immediately, but
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 478) wait for the in-flight data are acked by the other side, the max wait
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 479) time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 480) when the application closes a connection, kernel will send a RST
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 481) immediately and increase the TcpExtTCPAbortOnData counter.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 482)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 483) * TcpExtTCPAbortOnClose
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 484)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 485) This counter means the application has unread data in the TCP layer when
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 486) the application wants to close the TCP connection. In such a situation,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 487) kernel will send a RST to the other side of the TCP connection.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 488)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 489) * TcpExtTCPAbortOnMemory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 490)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 491) When an application closes a TCP connection, kernel still need to track
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 492) the connection, let it complete the TCP disconnect process. E.g. an
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 493) app calls the close method of a socket, kernel sends fin to the other
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 494) side of the connection, then the app has no relationship with the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 495) socket any more, but kernel need to keep the socket, this socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 496) becomes an orphan socket, kernel waits for the reply of the other side,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 497) and would come to the TIME_WAIT state finally. When kernel has no
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 498) enough memory to keep the orphan socket, kernel would send an RST to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 499) the other side, and delete the socket, in such situation, kernel will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 500) increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 501) TcpExtTCPAbortOnMemory:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 502)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 503) 1. the memory used by the TCP protocol is higher than the third value of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 504) the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 505)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 506) .. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 507)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 508) 2. the orphan socket count is higher than net.ipv4.tcp_max_orphans
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 509)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 510)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 511) * TcpExtTCPAbortOnTimeout
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 512)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 513) This counter will increase when any of the TCP timers expire. In such
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 514) situation, kernel won't send RST, just give up the connection.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 515)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 516) * TcpExtTCPAbortOnLinger
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 517)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 518) When a TCP connection comes into FIN_WAIT_2 state, instead of waiting
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 519) for the fin packet from the other side, kernel could send a RST and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 520) delete the socket immediately. This is not the default behavior of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 521) Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 522) you could let kernel follow this behavior.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 523)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 524) * TcpExtTCPAbortFailed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 525)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 526) The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 527) satisfied. If an internal error occurs during this process,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 528) TcpExtTCPAbortFailed will be increased.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 529)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 530) .. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 531)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 532) TCP Hybrid Slow Start
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 533) =====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 534) The Hybrid Slow Start algorithm is an enhancement of the traditional
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 535) TCP congestion window Slow Start algorithm. It uses two pieces of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 536) information to detect whether the max bandwidth of the TCP path is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 537) approached. The two pieces of information are ACK train length and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 538) increase in packet delay. For detail information, please refer the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 539) `Hybrid Slow Start paper`_. Either ACK train length or packet delay
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 540) hits a specific threshold, the congestion control algorithm will come
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 541) into the Congestion Avoidance state. Until v4.20, two congestion
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 542) control algorithms are using Hybrid Slow Start, they are cubic (the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 543) default congestion control algorithm) and cdg. Four snmp counters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 544) relate with the Hybrid Slow Start algorithm.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 545)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 546) .. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 547)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 548) * TcpExtTCPHystartTrainDetect
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 549)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 550) How many times the ACK train length threshold is detected
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 551)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 552) * TcpExtTCPHystartTrainCwnd
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 553)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 554) The sum of CWND detected by ACK train length. Dividing this value by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 555) TcpExtTCPHystartTrainDetect is the average CWND which detected by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 556) ACK train length.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 557)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 558) * TcpExtTCPHystartDelayDetect
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 559)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 560) How many times the packet delay threshold is detected.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 561)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 562) * TcpExtTCPHystartDelayCwnd
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 563)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 564) The sum of CWND detected by packet delay. Dividing this value by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 565) TcpExtTCPHystartDelayDetect is the average CWND which detected by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 566) packet delay.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 567)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 568) TCP retransmission and congestion control
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 569) =========================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 570) The TCP protocol has two retransmission mechanisms: SACK and fast
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 571) recovery. They are exclusive with each other. When SACK is enabled,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 572) the kernel TCP stack would use SACK, or kernel would use fast
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 573) recovery. The SACK is a TCP option, which is defined in `RFC2018`_,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 574) the fast recovery is defined in `RFC6582`_, which is also called
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 575) 'Reno'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 576)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 577) The TCP congestion control is a big and complex topic. To understand
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 578) the related snmp counter, we need to know the states of the congestion
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 579) control state machine. There are 5 states: Open, Disorder, CWR,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 580) Recovery and Loss. For details about these states, please refer page 5
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 581) and page 6 of this document:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 582) https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 583)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 584) .. _RFC2018: https://tools.ietf.org/html/rfc2018
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 585) .. _RFC6582: https://tools.ietf.org/html/rfc6582
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 586)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 587) * TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 588)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 589) When the congestion control comes into Recovery state, if sack is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 590) used, TcpExtTCPSackRecovery increases 1, if sack is not used,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 591) TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 592) stack begins to retransmit the lost packets.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 593)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 594) * TcpExtTCPSACKReneging
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 595)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 596) A packet was acknowledged by SACK, but the receiver has dropped this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 597) packet, so the sender needs to retransmit this packet. In this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 598) situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 599) could drop a packet which has been acknowledged by SACK, although it is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 600) unusual, it is allowed by the TCP protocol. The sender doesn't really
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 601) know what happened on the receiver side. The sender just waits until
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 602) the RTO expires for this packet, then the sender assumes this packet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 603) has been dropped by the receiver.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 604)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 605) * TcpExtTCPRenoReorder
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 606)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 607) The reorder packet is detected by fast recovery. It would only be used
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 608) if SACK is disabled. The fast recovery algorithm detects recorder by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 609) the duplicate ACK number. E.g., if retransmission is triggered, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 610) the original retransmitted packet is not lost, it is just out of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 611) order, the receiver would acknowledge multiple times, one for the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 612) retransmitted packet, another for the arriving of the original out of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 613) order packet. Thus the sender would find more ACks than its
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 614) expectation, and the sender knows out of order occurs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 615)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 616) * TcpExtTCPTSReorder
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 617)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 618) The reorder packet is detected when a hole is filled. E.g., assume the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 619) sender sends packet 1,2,3,4,5, and the receiving order is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 620) 1,2,4,5,3. When the sender receives the ACK of packet 3 (which will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 621) fill the hole), two conditions will let TcpExtTCPTSReorder increase
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 622) 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 623) 3 is retransmitted but the timestamp of the packet 3's ACK is earlier
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 624) than the retransmission timestamp.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 625)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 626) * TcpExtTCPSACKReorder
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 627)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 628) The reorder packet detected by SACK. The SACK has two methods to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 629) detect reorder: (1) DSACK is received by the sender. It means the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 630) sender sends the same packet more than one times. And the only reason
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 631) is the sender believes an out of order packet is lost so it sends the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 632) packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 633) the sender has received SACKs for packet 2 and 5, now the sender
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 634) receives SACK for packet 4 and the sender doesn't retransmit the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 635) packet yet, the sender would know packet 4 is out of order. The TCP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 636) stack of kernel will increase TcpExtTCPSACKReorder for both of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 637) above scenarios.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 638)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 639) * TcpExtTCPSlowStartRetrans
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 640)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 641) The TCP stack wants to retransmit a packet and the congestion control
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 642) state is 'Loss'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 643)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 644) * TcpExtTCPFastRetrans
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 645)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 646) The TCP stack wants to retransmit a packet and the congestion control
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 647) state is not 'Loss'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 648)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 649) * TcpExtTCPLostRetransmit
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 650)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 651) A SACK points out that a retransmission packet is lost again.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 652)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 653) * TcpExtTCPRetransFail
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 654)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 655) The TCP stack tries to deliver a retransmission packet to lower layers
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 656) but the lower layers return an error.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 657)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 658) * TcpExtTCPSynRetrans
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 659)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 660) The TCP stack retransmits a SYN packet.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 661)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 662) DSACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 663) =====
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 664) The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 665) duplicate packets to the sender. There are two kinds of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 666) duplications: (1) a packet which has been acknowledged is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 667) duplicate. (2) an out of order packet is duplicate. The TCP stack
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 668) counts these two kinds of duplications on both receiver side and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 669) sender side.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 670)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 671) .. _RFC2883 : https://tools.ietf.org/html/rfc2883
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 672)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 673) * TcpExtTCPDSACKOldSent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 674)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 675) The TCP stack receives a duplicate packet which has been acked, so it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 676) sends a DSACK to the sender.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 677)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 678) * TcpExtTCPDSACKOfoSent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 679)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 680) The TCP stack receives an out of order duplicate packet, so it sends a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 681) DSACK to the sender.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 682)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 683) * TcpExtTCPDSACKRecv
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 684)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 685) The TCP stack receives a DSACK, which indicates an acknowledged
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 686) duplicate packet is received.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 687)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 688) * TcpExtTCPDSACKOfoRecv
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 689)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 690) The TCP stack receives a DSACK, which indicate an out of order
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 691) duplicate packet is received.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 692)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 693) invalid SACK and DSACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 694) ======================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 695) When a SACK (or DSACK) block is invalid, a corresponding counter would
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 696) be updated. The validation method is base on the start/end sequence
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 697) number of the SACK block. For more details, please refer the comment
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 698) of the function tcp_is_sackblock_valid in the kernel source code. A
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 699) SACK option could have up to 4 blocks, they are checked
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 700) individually. E.g., if 3 blocks of a SACk is invalid, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 701) corresponding counter would be updated 3 times. The comment of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 702) `Add counters for discarded SACK blocks`_ patch has additional
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 703) explaination:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 704)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 705) .. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18f02545a9a16c9a89778b91a162ad16d510bb32
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 706)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 707) * TcpExtTCPSACKDiscard
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 708)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 709) This counter indicates how many SACK blocks are invalid. If the invalid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 710) SACK block is caused by ACK recording, the TCP stack will only ignore
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 711) it and won't update this counter.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 712)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 713) * TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 714)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 715) When a DSACK block is invalid, one of these two counters would be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 716) updated. Which counter will be updated depends on the undo_marker flag
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 717) of the TCP socket. If the undo_marker is not set, the TCP stack isn't
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 718) likely to re-transmit any packets, and we still receive an invalid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 719) DSACK block, the reason might be that the packet is duplicated in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 720) middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 721) will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 722) will be updated. As implied in its name, it might be an old packet.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 723)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 724) SACK shift
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 725) ==========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 726) The linux networking stack stores data in sk_buff struct (skb for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 727) short). If a SACK block acrosses multiple skb, the TCP stack will try
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 728) to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 729) 10 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 730) 15 in skb2 would be moved to skb1. This operation is 'shift'. If a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 731) SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 732) seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 733) discard, this operation is 'merge'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 734)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 735) * TcpExtTCPSackShifted
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 736)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 737) A skb is shifted
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 738)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 739) * TcpExtTCPSackMerged
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 740)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 741) A skb is merged
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 742)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 743) * TcpExtTCPSackShiftFallback
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 744)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 745) A skb should be shifted or merged, but the TCP stack doesn't do it for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 746) some reasons.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 747)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 748) TCP out of order
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 749) ================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 750) * TcpExtTCPOFOQueue
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 751)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 752) The TCP layer receives an out of order packet and has enough memory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 753) to queue it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 754)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 755) * TcpExtTCPOFODrop
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 756)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 757) The TCP layer receives an out of order packet but doesn't have enough
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 758) memory, so drops it. Such packets won't be counted into
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 759) TcpExtTCPOFOQueue.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 760)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 761) * TcpExtTCPOFOMerge
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 762)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 763) The received out of order packet has an overlay with the previous
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 764) packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 765) packets will also be counted into TcpExtTCPOFOQueue.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 766)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 767) TCP PAWS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 768) ========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 769) PAWS (Protection Against Wrapped Sequence numbers) is an algorithm
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 770) which is used to drop old packets. It depends on the TCP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 771) timestamps. For detail information, please refer the `timestamp wiki`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 772) and the `RFC of PAWS`_.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 773)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 774) .. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 775) .. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 776)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 777) * TcpExtPAWSActive
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 778)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 779) Packets are dropped by PAWS in Syn-Sent status.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 780)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 781) * TcpExtPAWSEstab
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 782)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 783) Packets are dropped by PAWS in any status other than Syn-Sent.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 784)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 785) TCP ACK skip
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 786) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 787) In some scenarios, kernel would avoid sending duplicate ACKs too
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 788) frequently. Please find more details in the tcp_invalid_ratelimit
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 789) section of the `sysctl document`_. When kernel decides to skip an ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 790) due to tcp_invalid_ratelimit, kernel would update one of below
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 791) counters to indicate the ACK is skipped in which scenario. The ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 792) would only be skipped if the received packet is either a SYN packet or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 793) it has no data.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 794)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 795) .. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 796)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 797) * TcpExtTCPACKSkippedSynRecv
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 798)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 799) The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 800) TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 801) waiting for an ACK. Generally, the TCP stack doesn't need to send ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 802) in the Syn-Recv status. But in several scenarios, the TCP stack need
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 803) to send an ACK. E.g., the TCP stack receives the same SYN packet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 804) repeately, the received packet does not pass the PAWS check, or the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 805) received packet sequence number is out of window. In these scenarios,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 806) the TCP stack needs to send ACK. If the ACk sending frequency is higher than
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 807) tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 808) increase TcpExtTCPACKSkippedSynRecv.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 809)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 810)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 811) * TcpExtTCPACKSkippedPAWS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 812)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 813) The ACK is skipped due to PAWS (Protect Against Wrapped Sequence
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 814) numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 815) or Time-Wait statuses, the skipped ACK would be counted to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 816) TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 817) TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 818) would be counted to TcpExtTCPACKSkippedPAWS.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 819)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 820) * TcpExtTCPACKSkippedSeq
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 821)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 822) The sequence number is out of window and the timestamp passes the PAWS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 823) check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 824)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 825) * TcpExtTCPACKSkippedFinWait2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 826)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 827) The ACK is skipped in Fin-Wait-2 status, the reason would be either
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 828) PAWS check fails or the received sequence number is out of window.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 829)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 830) * TcpExtTCPACKSkippedTimeWait
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 831)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 832) Tha ACK is skipped in Time-Wait status, the reason would be either
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 833) PAWS check failed or the received sequence number is out of window.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 834)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 835) * TcpExtTCPACKSkippedChallenge
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 836)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 837) The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 838) 3 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 839) `RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 840) three scenarios, In some TCP status, the linux TCP stack would also
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 841) send challenge ACKs if the ACK number is before the first
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 842) unacknowledged number (more strict than `RFC 5961 section 5.2`_).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 843)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 844) .. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 845) .. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 846) .. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 847)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 848) TCP receive window
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 849) ==================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 850) * TcpExtTCPWantZeroWindowAdv
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 851)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 852) Depending on current memory usage, the TCP stack tries to set receive
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 853) window to zero. But the receive window might still be a no-zero
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 854) value. For example, if the previous window size is 10, and the TCP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 855) stack receives 3 bytes, the current window size would be 7 even if the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 856) window size calculated by the memory usage is zero.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 857)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 858) * TcpExtTCPToZeroWindowAdv
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 859)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 860) The TCP receive window is set to zero from a no-zero value.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 861)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 862) * TcpExtTCPFromZeroWindowAdv
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 863)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 864) The TCP receive window is set to no-zero value from zero.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 865)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 866)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 867) Delayed ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 868) ===========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 869) The TCP Delayed ACK is a technique which is used for reducing the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 870) packet count in the network. For more details, please refer the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 871) `Delayed ACK wiki`_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 872)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 873) .. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 874)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 875) * TcpExtDelayedACKs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 876)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 877) A delayed ACK timer expires. The TCP stack will send a pure ACK packet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 878) and exit the delayed ACK mode.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 879)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 880) * TcpExtDelayedACKLocked
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 881)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 882) A delayed ACK timer expires, but the TCP stack can't send an ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 883) immediately due to the socket is locked by a userspace program. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 884) TCP stack will send a pure ACK later (after the userspace program
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 885) unlock the socket). When the TCP stack sends the pure ACK later, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 886) TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 887) mode.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 888)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 889) * TcpExtDelayedACKLost
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 890)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 891) It will be updated when the TCP stack receives a packet which has been
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 892) ACKed. A Delayed ACK loss might cause this issue, but it would also be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 893) triggered by other reasons, such as a packet is duplicated in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 894) network.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 895)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 896) Tail Loss Probe (TLP)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 897) =====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 898) TLP is an algorithm which is used to detect TCP packet loss. For more
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 899) details, please refer the `TLP paper`_.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 900)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 901) .. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 902)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 903) * TcpExtTCPLossProbes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 904)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 905) A TLP probe packet is sent.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 906)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 907) * TcpExtTCPLossProbeRecovery
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 908)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 909) A packet loss is detected and recovered by TLP.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 910)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 911) TCP Fast Open description
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 912) =========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 913) TCP Fast Open is a technology which allows data transfer before the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 914) 3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 915) general description.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 916)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 917) .. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 918)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 919) * TcpExtTCPFastOpenActive
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 920)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 921) When the TCP stack receives an ACK packet in the SYN-SENT status, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 922) the ACK packet acknowledges the data in the SYN packet, the TCP stack
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 923) understand the TFO cookie is accepted by the other side, then it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 924) updates this counter.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 925)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 926) * TcpExtTCPFastOpenActiveFail
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 927)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 928) This counter indicates that the TCP stack initiated a TCP Fast Open,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 929) but it failed. This counter would be updated in three scenarios: (1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 930) the other side doesn't acknowledge the data in the SYN packet. (2) The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 931) SYN packet which has the TFO cookie is timeout at least once. (3)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 932) after the 3-way handshake, the retransmission timeout happens
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 933) net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 934) fast open after the handshake.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 935)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 936) * TcpExtTCPFastOpenPassive
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 937)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 938) This counter indicates how many times the TCP stack accepts the fast
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 939) open request.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 940)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 941) * TcpExtTCPFastOpenPassiveFail
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 942)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 943) This counter indicates how many times the TCP stack rejects the fast
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 944) open request. It is caused by either the TFO cookie is invalid or the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 945) TCP stack finds an error during the socket creating process.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 946)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 947) * TcpExtTCPFastOpenListenOverflow
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 948)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 949) When the pending fast open request number is larger than
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 950) fastopenq->max_qlen, the TCP stack will reject the fast open request
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 951) and update this counter. When this counter is updated, the TCP stack
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 952) won't update TcpExtTCPFastOpenPassive or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 953) TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 954) TCP_FASTOPEN socket operation and it could not be larger than
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 955) net.core.somaxconn. For example:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 956)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 957) setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 958)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 959) * TcpExtTCPFastOpenCookieReqd
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 960)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 961) This counter indicates how many times a client wants to request a TFO
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 962) cookie.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 963)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 964) SYN cookies
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 965) ===========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 966) SYN cookies are used to mitigate SYN flood, for details, please refer
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 967) the `SYN cookies wiki`_.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 968)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 969) .. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 970)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 971) * TcpExtSyncookiesSent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 972)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 973) It indicates how many SYN cookies are sent.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 974)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 975) * TcpExtSyncookiesRecv
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 976)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 977) How many reply packets of the SYN cookies the TCP stack receives.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 978)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 979) * TcpExtSyncookiesFailed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 980)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 981) The MSS decoded from the SYN cookie is invalid. When this counter is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 982) updated, the received packet won't be treated as a SYN cookie and the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 983) TcpExtSyncookiesRecv counter wont be updated.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 984)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 985) Challenge ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 986) =============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 987) For details of challenge ACK, please refer the explaination of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 988) TcpExtTCPACKSkippedChallenge.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 989)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 990) * TcpExtTCPChallengeACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 991)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 992) The number of challenge acks sent.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 993)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 994) * TcpExtTCPSYNChallenge
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 995)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 996) The number of challenge acks sent in response to SYN packets. After
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 997) updates this counter, the TCP stack might send a challenge ACK and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 998) update the TcpExtTCPChallengeACK counter, or it might also skip to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 999) send the challenge and update the TcpExtTCPACKSkippedChallenge.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1000)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1001) prune
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1002) =====
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1003) When a socket is under memory pressure, the TCP stack will try to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1004) reclaim memory from the receiving queue and out of order queue. One of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1005) the reclaiming method is 'collapse', which means allocate a big sbk,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1006) copy the contiguous skbs to the single big skb, and free these
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1007) contiguous skbs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1008)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1009) * TcpExtPruneCalled
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1010)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1011) The TCP stack tries to reclaim memory for a socket. After updates this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1012) counter, the TCP stack will try to collapse the out of order queue and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1013) the receiving queue. If the memory is still not enough, the TCP stack
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1014) will try to discard packets from the out of order queue (and update the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1015) TcpExtOfoPruned counter)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1016)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1017) * TcpExtOfoPruned
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1018)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1019) The TCP stack tries to discard packet on the out of order queue.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1020)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1021) * TcpExtRcvPruned
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1022)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1023) After 'collapse' and discard packets from the out of order queue, if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1024) the actually used memory is still larger than the max allowed memory,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1025) this counter will be updated. It means the 'prune' fails.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1026)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1027) * TcpExtTCPRcvCollapsed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1028)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1029) This counter indicates how many skbs are freed during 'collapse'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1030)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1031) examples
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1032) ========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1033)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1034) ping test
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1035) ---------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1036) Run the ping command against the public dns server 8.8.8.8::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1037)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1038) nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1039) PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1040) 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1041)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1042) --- 8.8.8.8 ping statistics ---
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1043) 1 packets transmitted, 1 received, 0% packet loss, time 0ms
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1044) rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1045)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1046) The nstayt result::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1047)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1048) nstatuser@nstat-a:~$ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1049) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1050) IpInReceives 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1051) IpInDelivers 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1052) IpOutRequests 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1053) IcmpInMsgs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1054) IcmpInEchoReps 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1055) IcmpOutMsgs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1056) IcmpOutEchos 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1057) IcmpMsgInType0 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1058) IcmpMsgOutType8 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1059) IpExtInOctets 84 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1060) IpExtOutOctets 84 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1061) IpExtInNoECTPkts 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1062)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1063) The Linux server sent an ICMP Echo packet, so IpOutRequests,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1064) IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1065) server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1066) IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1067) was passed to the ICMP layer via IP layer, so IpInDelivers was
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1068) increased 1. The default ping data size is 48, so an ICMP Echo packet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1069) and its corresponding Echo Reply packet are constructed by:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1070)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1071) * 14 bytes MAC header
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1072) * 20 bytes IP header
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1073) * 16 bytes ICMP header
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1074) * 48 bytes data (default value of the ping command)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1075)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1076) So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1077)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1078) tcp 3-way handshake
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1079) -------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1080) On server side, we run::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1081)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1082) nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1083) Listening on [0.0.0.0] (family 0, port 9000)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1084)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1085) On client side, we run::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1086)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1087) nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1088) Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1089)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1090) The server listened on tcp 9000 port, the client connected to it, they
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1091) completed the 3-way handshake.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1092)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1093) On server side, we can find below nstat output::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1094)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1095) nstatuser@nstat-b:~$ nstat | grep -i tcp
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1096) TcpPassiveOpens 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1097) TcpInSegs 2 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1098) TcpOutSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1099) TcpExtTCPPureAcks 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1100)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1101) On client side, we can find below nstat output::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1102)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1103) nstatuser@nstat-a:~$ nstat | grep -i tcp
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1104) TcpActiveOpens 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1105) TcpInSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1106) TcpOutSegs 2 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1107)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1108) When the server received the first SYN, it replied a SYN+ACK, and came into
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1109) SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1110) SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1111) packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1112) of the 3-way handshake is a pure ACK without data, so
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1113) TcpExtTCPPureAcks increased 1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1114)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1115) When the client sent SYN, the client came into the SYN-SENT state, so
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1116) TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1117) ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1118) 1, TcpOutSegs increased 2.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1119)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1120) TCP normal traffic
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1121) ------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1122) Run nc on server::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1123)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1124) nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1125) Listening on [0.0.0.0] (family 0, port 9000)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1126)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1127) Run nc on client::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1128)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1129) nstatuser@nstat-a:~$ nc -v nstat-b 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1130) Connection to nstat-b 9000 port [tcp/*] succeeded!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1131)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1132) Input a string in the nc client ('hello' in our example)::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1133)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1134) nstatuser@nstat-a:~$ nc -v nstat-b 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1135) Connection to nstat-b 9000 port [tcp/*] succeeded!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1136) hello
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1137)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1138) The client side nstat output::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1139)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1140) nstatuser@nstat-a:~$ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1141) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1142) IpInReceives 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1143) IpInDelivers 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1144) IpOutRequests 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1145) TcpInSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1146) TcpOutSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1147) TcpExtTCPPureAcks 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1148) TcpExtTCPOrigDataSent 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1149) IpExtInOctets 52 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1150) IpExtOutOctets 58 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1151) IpExtInNoECTPkts 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1152)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1153) The server side nstat output::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1154)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1155) nstatuser@nstat-b:~$ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1156) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1157) IpInReceives 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1158) IpInDelivers 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1159) IpOutRequests 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1160) TcpInSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1161) TcpOutSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1162) IpExtInOctets 58 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1163) IpExtOutOctets 52 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1164) IpExtInNoECTPkts 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1165)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1166) Input a string in nc client side again ('world' in our exmaple)::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1167)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1168) nstatuser@nstat-a:~$ nc -v nstat-b 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1169) Connection to nstat-b 9000 port [tcp/*] succeeded!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1170) hello
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1171) world
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1172)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1173) Client side nstat output::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1174)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1175) nstatuser@nstat-a:~$ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1176) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1177) IpInReceives 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1178) IpInDelivers 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1179) IpOutRequests 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1180) TcpInSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1181) TcpOutSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1182) TcpExtTCPHPAcks 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1183) TcpExtTCPOrigDataSent 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1184) IpExtInOctets 52 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1185) IpExtOutOctets 58 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1186) IpExtInNoECTPkts 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1187)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1188)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1189) Server side nstat output::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1190)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1191) nstatuser@nstat-b:~$ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1192) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1193) IpInReceives 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1194) IpInDelivers 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1195) IpOutRequests 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1196) TcpInSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1197) TcpOutSegs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1198) TcpExtTCPHPHits 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1199) IpExtInOctets 58 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1200) IpExtOutOctets 52 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1201) IpExtInNoECTPkts 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1202)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1203) Compare the first client-side nstat and the second client-side nstat,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1204) we could find one difference: the first one had a 'TcpExtTCPPureAcks',
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1205) but the second one had a 'TcpExtTCPHPAcks'. The first server-side
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1206) nstat and the second server-side nstat had a difference too: the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1207) second server-side nstat had a TcpExtTCPHPHits, but the first
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1208) server-side nstat didn't have it. The network traffic patterns were
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1209) exactly the same: the client sent a packet to the server, the server
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1210) replied an ACK. But kernel handled them in different ways. When the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1211) TCP window scale option is not used, kernel will try to enable fast
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1212) path immediately when the connection comes into the established state,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1213) but if the TCP window scale option is used, kernel will disable the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1214) fast path at first, and try to enable it after kerenl receives
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1215) packets. We could use the 'ss' command to verify whether the window
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1216) scale option is used. e.g. run below command on either server or
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1217) client::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1218)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1219) nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1220) Netid Recv-Q Send-Q Local Address:Port Peer Address:Port
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1221) tcp 0 0 192.168.122.250:40654 192.168.122.251:9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1222) ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1223)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1224) The 'wscale:7,7' means both server and client set the window scale
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1225) option to 7. Now we could explain the nstat output in our test:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1226)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1227) In the first nstat output of client side, the client sent a packet, server
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1228) reply an ACK, when kernel handled this ACK, the fast path was not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1229) enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1230)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1231) In the second nstat output of client side, the client sent a packet again,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1232) and received another ACK from the server, in this time, the fast path is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1233) enabled, and the ACK was qualified for fast path, so it was handled by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1234) the fast path, so this ACK was counted into TcpExtTCPHPAcks.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1235)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1236) In the first nstat output of server side, fast path was not enabled,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1237) so there was no 'TcpExtTCPHPHits'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1238)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1239) In the second nstat output of server side, the fast path was enabled,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1240) and the packet received from client qualified for fast path, so it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1241) was counted into 'TcpExtTCPHPHits'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1242)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1243) TcpExtTCPAbortOnClose
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1244) ---------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1245) On the server side, we run below python script::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1246)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1247) import socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1248) import time
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1249)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1250) port = 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1251)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1252) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1253) s.bind(('0.0.0.0', port))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1254) s.listen(1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1255) sock, addr = s.accept()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1256) while True:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1257) time.sleep(9999999)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1258)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1259) This python script listen on 9000 port, but doesn't read anything from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1260) the connection.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1261)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1262) On the client side, we send the string "hello" by nc::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1263)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1264) nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1265)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1266) Then, we come back to the server side, the server has received the "hello"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1267) packet, and the TCP layer has acked this packet, but the application didn't
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1268) read it yet. We type Ctrl-C to terminate the server script. Then we
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1269) could find TcpExtTCPAbortOnClose increased 1 on the server side::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1270)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1271) nstatuser@nstat-b:~$ nstat | grep -i abort
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1272) TcpExtTCPAbortOnClose 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1273)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1274) If we run tcpdump on the server side, we could find the server sent a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1275) RST after we type Ctrl-C.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1276)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1277) TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1278) ---------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1279) Below is an example which let the orphan socket count be higher than
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1280) net.ipv4.tcp_max_orphans.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1281) Change tcp_max_orphans to a smaller value on client::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1282)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1283) sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1284)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1285) Client code (create 64 connection to server)::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1286)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1287) nstatuser@nstat-a:~$ cat client_orphan.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1288) import socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1289) import time
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1290)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1291) server = 'nstat-b' # server address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1292) port = 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1293)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1294) count = 64
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1295)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1296) connection_list = []
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1297)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1298) for i in range(64):
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1299) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1300) s.connect((server, port))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1301) connection_list.append(s)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1302) print("connection_count: %d" % len(connection_list))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1303)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1304) while True:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1305) time.sleep(99999)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1306)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1307) Server code (accept 64 connection from client)::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1308)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1309) nstatuser@nstat-b:~$ cat server_orphan.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1310) import socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1311) import time
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1312)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1313) port = 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1314) count = 64
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1315)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1316) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1317) s.bind(('0.0.0.0', port))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1318) s.listen(count)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1319) connection_list = []
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1320) while True:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1321) sock, addr = s.accept()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1322) connection_list.append((sock, addr))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1323) print("connection_count: %d" % len(connection_list))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1324)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1325) Run the python scripts on server and client.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1326)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1327) On server::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1328)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1329) python3 server_orphan.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1330)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1331) On client::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1332)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1333) python3 client_orphan.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1334)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1335) Run iptables on server::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1336)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1337) sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1338)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1339) Type Ctrl-C on client, stop client_orphan.py.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1340)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1341) Check TcpExtTCPAbortOnMemory on client::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1342)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1343) nstatuser@nstat-a:~$ nstat | grep -i abort
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1344) TcpExtTCPAbortOnMemory 54 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1345)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1346) Check orphane socket count on client::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1347)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1348) nstatuser@nstat-a:~$ ss -s
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1349) Total: 131 (kernel 0)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1350) TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1351)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1352) Transport Total IP IPv6
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1353) * 0 - -
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1354) RAW 1 0 1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1355) UDP 1 1 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1356) TCP 14 13 1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1357) INET 16 14 2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1358) FRAG 0 0 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1359)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1360) The explanation of the test: after run server_orphan.py and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1361) client_orphan.py, we set up 64 connections between server and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1362) client. Run the iptables command, the server will drop all packets from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1363) the client, type Ctrl-C on client_orphan.py, the system of the client
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1364) would try to close these connections, and before they are closed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1365) gracefully, these connections became orphan sockets. As the iptables
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1366) of the server blocked packets from the client, the server won't receive fin
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1367) from the client, so all connection on clients would be stuck on FIN_WAIT_1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1368) stage, so they will keep as orphan sockets until timeout. We have echo
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1369) 10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1370) only keep 10 orphan sockets, for all other orphan sockets, the client
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1371) system sent RST for them and delete them. We have 64 connections, so
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1372) the 'ss -s' command shows the system has 10 orphan sockets, and the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1373) value of TcpExtTCPAbortOnMemory was 54.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1374)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1375) An additional explanation about orphan socket count: You could find the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1376) exactly orphan socket count by the 'ss -s' command, but when kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1377) decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1378) doesn't always check the exactly orphan socket count. For increasing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1379) performance, kernel checks an approximate count firstly, if the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1380) approximate count is more than tcp_max_orphans, kernel checks the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1381) exact count again. So if the approximate count is less than
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1382) tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1383) would find TcpExtTCPAbortOnMemory is not increased at all. If
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1384) tcp_max_orphans is large enough, it won't occur, but if you decrease
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1385) tcp_max_orphans to a small value like our test, you might find this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1386) issue. So in our test, the client set up 64 connections although the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1387) tcp_max_orphans is 10. If the client only set up 11 connections, we
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1388) can't find the change of TcpExtTCPAbortOnMemory.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1389)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1390) Continue the previous test, we wait for several minutes. Because of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1391) iptables on the server blocked the traffic, the server wouldn't receive
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1392) fin, and all the client's orphan sockets would timeout on the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1393) FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1394) 10 timeout on the client::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1395)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1396) nstatuser@nstat-a:~$ nstat | grep -i abort
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1397) TcpExtTCPAbortOnTimeout 10 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1398)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1399) TcpExtTCPAbortOnLinger
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1400) ----------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1401) The server side code::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1402)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1403) nstatuser@nstat-b:~$ cat server_linger.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1404) import socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1405) import time
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1406)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1407) port = 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1408)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1409) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1410) s.bind(('0.0.0.0', port))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1411) s.listen(1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1412) sock, addr = s.accept()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1413) while True:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1414) time.sleep(9999999)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1415)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1416) The client side code::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1417)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1418) nstatuser@nstat-a:~$ cat client_linger.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1419) import socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1420) import struct
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1421)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1422) server = 'nstat-b' # server address
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1423) port = 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1424)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1425) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1426) s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1427) s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1428) s.connect((server, port))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1429) s.close()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1430)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1431) Run server_linger.py on server::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1432)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1433) nstatuser@nstat-b:~$ python3 server_linger.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1434)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1435) Run client_linger.py on client::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1436)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1437) nstatuser@nstat-a:~$ python3 client_linger.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1438)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1439) After run client_linger.py, check the output of nstat::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1440)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1441) nstatuser@nstat-a:~$ nstat | grep -i abort
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1442) TcpExtTCPAbortOnLinger 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1443)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1444) TcpExtTCPRcvCoalesce
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1445) --------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1446) On the server, we run a program which listen on TCP port 9000, but
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1447) doesn't read any data::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1448)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1449) import socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1450) import time
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1451) port = 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1452) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1453) s.bind(('0.0.0.0', port))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1454) s.listen(1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1455) sock, addr = s.accept()
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1456) while True:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1457) time.sleep(9999999)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1458)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1459) Save the above code as server_coalesce.py, and run::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1460)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1461) python3 server_coalesce.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1462)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1463) On the client, save below code as client_coalesce.py::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1464)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1465) import socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1466) server = 'nstat-b'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1467) port = 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1468) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1469) s.connect((server, port))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1470)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1471) Run::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1472)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1473) nstatuser@nstat-a:~$ python3 -i client_coalesce.py
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1474)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1475) We use '-i' to come into the interactive mode, then a packet::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1476)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1477) >>> s.send(b'foo')
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1478) 3
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1479)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1480) Send a packet again::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1481)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1482) >>> s.send(b'bar')
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1483) 3
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1484)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1485) On the server, run nstat::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1486)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1487) ubuntu@nstat-b:~$ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1488) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1489) IpInReceives 2 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1490) IpInDelivers 2 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1491) IpOutRequests 2 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1492) TcpInSegs 2 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1493) TcpOutSegs 2 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1494) TcpExtTCPRcvCoalesce 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1495) IpExtInOctets 110 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1496) IpExtOutOctets 104 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1497) IpExtInNoECTPkts 2 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1498)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1499) The client sent two packets, server didn't read any data. When
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1500) the second packet arrived at server, the first packet was still in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1501) the receiving queue. So the TCP layer merged the two packets, and we
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1502) could find the TcpExtTCPRcvCoalesce increased 1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1503)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1504) TcpExtListenOverflows and TcpExtListenDrops
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1505) -------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1506) On server, run the nc command, listen on port 9000::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1507)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1508) nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1509) Listening on [0.0.0.0] (family 0, port 9000)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1510)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1511) On client, run 3 nc commands in different terminals::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1512)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1513) nstatuser@nstat-a:~$ nc -v nstat-b 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1514) Connection to nstat-b 9000 port [tcp/*] succeeded!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1515)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1516) The nc command only accepts 1 connection, and the accept queue length
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1517) is 1. On current linux implementation, set queue length to n means the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1518) actual queue length is n+1. Now we create 3 connections, 1 is accepted
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1519) by nc, 2 in accepted queue, so the accept queue is full.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1520)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1521) Before running the 4th nc, we clean the nstat history on the server::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1522)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1523) nstatuser@nstat-b:~$ nstat -n
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1524)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1525) Run the 4th nc on the client::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1526)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1527) nstatuser@nstat-a:~$ nc -v nstat-b 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1528)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1529) If the nc server is running on kernel 4.10 or higher version, you
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1530) won't see the "Connection to ... succeeded!" string, because kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1531) will drop the SYN if the accept queue is full. If the nc client is running
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1532) on an old kernel, you would see that the connection is succeeded,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1533) because kernel would complete the 3 way handshake and keep the socket
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1534) on half open queue. I did the test on kernel 4.15. Below is the nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1535) on the server::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1536)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1537) nstatuser@nstat-b:~$ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1538) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1539) IpInReceives 4 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1540) IpInDelivers 4 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1541) TcpInSegs 4 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1542) TcpExtListenOverflows 4 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1543) TcpExtListenDrops 4 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1544) IpExtInOctets 240 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1545) IpExtInNoECTPkts 4 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1546)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1547) Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1548) between the 4th nc and the nstat was longer, the value of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1549) TcpExtListenOverflows and TcpExtListenDrops would be larger, because
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1550) the SYN of the 4th nc was dropped, the client was retrying.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1551)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1552) IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1553) -------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1554) server A IP address: 192.168.122.250
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1555) server B IP address: 192.168.122.251
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1556) Prepare on server A, add a route to server B::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1557)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1558) $ sudo ip route add 8.8.8.8/32 via 192.168.122.251
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1559)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1560) Prepare on server B, disable send_redirects for all interfaces::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1561)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1562) $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1563) $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1564) $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1565) $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1566)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1567) We want to let sever A send a packet to 8.8.8.8, and route the packet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1568) to server B. When server B receives such packet, it might send a ICMP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1569) Redirect message to server A, set send_redirects to 0 will disable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1570) this behavior.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1571)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1572) First, generate InAddrErrors. On server B, we disable IP forwarding::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1573)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1574) $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1575)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1576) On server A, we send packets to 8.8.8.8::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1577)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1578) $ nc -v 8.8.8.8 53
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1579)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1580) On server B, we check the output of nstat::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1581)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1582) $ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1583) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1584) IpInReceives 3 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1585) IpInAddrErrors 3 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1586) IpExtInOctets 180 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1587) IpExtInNoECTPkts 3 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1588)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1589) As we have let server A route 8.8.8.8 to server B, and we disabled IP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1590) forwarding on server B, Server A sent packets to server B, then server B
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1591) dropped packets and increased IpInAddrErrors. As the nc command would
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1592) re-send the SYN packet if it didn't receive a SYN+ACK, we could find
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1593) multiple IpInAddrErrors.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1594)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1595) Second, generate IpExtInNoRoutes. On server B, we enable IP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1596) forwarding::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1597)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1598) $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1599)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1600) Check the route table of server B and remove the default route::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1601)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1602) $ ip route show
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1603) default via 192.168.122.1 dev ens3 proto static
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1604) 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1605) $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1606)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1607) On server A, we contact 8.8.8.8 again::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1608)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1609) $ nc -v 8.8.8.8 53
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1610) nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1611)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1612) On server B, run nstat::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1613)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1614) $ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1615) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1616) IpInReceives 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1617) IpOutRequests 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1618) IcmpOutMsgs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1619) IcmpOutDestUnreachs 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1620) IcmpMsgOutType3 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1621) IpExtInNoRoutes 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1622) IpExtInOctets 60 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1623) IpExtOutOctets 88 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1624) IpExtInNoECTPkts 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1625)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1626) We enabled IP forwarding on server B, when server B received a packet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1627) which destination IP address is 8.8.8.8, server B will try to forward
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1628) this packet. We have deleted the default route, there was no route for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1629) 8.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1630) Destination Unreachable" message to server A.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1631)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1632) Third, generate IpOutNoRoutes. Run ping command on server B::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1633)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1634) $ ping -c 1 8.8.8.8
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1635) connect: Network is unreachable
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1636)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1637) Run nstat on server B::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1638)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1639) $ nstat
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1640) #kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1641) IpOutNoRoutes 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1642)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1643) We have deleted the default route on server B. Server B couldn't find
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1644) a route for the 8.8.8.8 IP address, so server B increased
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1645) IpOutNoRoutes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1646)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1647) TcpExtTCPACKSkippedSynRecv
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1648) --------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1649) In this test, we send 3 same SYN packets from client to server. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1650) first SYN will let server create a socket, set it to Syn-Recv status,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1651) and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1652) again, and record the reply time (the duplicate ACK reply time). The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1653) third SYN will let server check the previous duplicate ACK reply time,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1654) and decide to skip the duplicate ACK, then increase the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1655) TcpExtTCPACKSkippedSynRecv counter.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1656)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1657) Run tcpdump to capture a SYN packet::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1658)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1659) nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1660) tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1661)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1662) Open another terminal, run nc command::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1663)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1664) nstatuser@nstat-a:~$ nc nstat-b 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1665)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1666) As the nstat-b didn't listen on port 9000, it should reply a RST, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1667) the nc command exited immediately. It was enough for the tcpdump
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1668) command to capture a SYN packet. A linux server might use hardware
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1669) offload for the TCP checksum, so the checksum in the /tmp/syn.pcap
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1670) might be not correct. We call tcprewrite to fix it::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1671)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1672) nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1673)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1674) On nstat-b, we run nc to listen on port 9000::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1675)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1676) nstatuser@nstat-b:~$ nc -lkv 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1677) Listening on [0.0.0.0] (family 0, port 9000)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1678)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1679) On nstat-a, we blocked the packet from port 9000, or nstat-a would send
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1680) RST to nstat-b::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1681)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1682) nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1683)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1684) Send 3 SYN repeatly to nstat-b::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1685)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1686) nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1687)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1688) Check snmp cunter on nstat-b::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1689)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1690) nstatuser@nstat-b:~$ nstat | grep -i skip
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1691) TcpExtTCPACKSkippedSynRecv 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1692)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1693) As we expected, TcpExtTCPACKSkippedSynRecv is 1.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1694)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1695) TcpExtTCPACKSkippedPAWS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1696) -----------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1697) To trigger PAWS, we could send an old SYN.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1698)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1699) On nstat-b, let nc listen on port 9000::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1700)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1701) nstatuser@nstat-b:~$ nc -lkv 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1702) Listening on [0.0.0.0] (family 0, port 9000)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1703)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1704) On nstat-a, run tcpdump to capture a SYN::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1705)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1706) nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1707) tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1708)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1709) On nstat-a, run nc as a client to connect nstat-b::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1710)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1711) nstatuser@nstat-a:~$ nc -v nstat-b 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1712) Connection to nstat-b 9000 port [tcp/*] succeeded!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1713)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1714) Now the tcpdump has captured the SYN and exit. We should fix the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1715) checksum::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1716)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1717) nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1718)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1719) Send the SYN packet twice::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1720)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1721) nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1722)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1723) On nstat-b, check the snmp counter::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1724)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1725) nstatuser@nstat-b:~$ nstat | grep -i skip
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1726) TcpExtTCPACKSkippedPAWS 1 0.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1727)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1728) We sent two SYN via tcpreplay, both of them would let PAWS check
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1729) failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1730) for the second SYN, and updated TcpExtTCPACKSkippedPAWS.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1731)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1732) TcpExtTCPACKSkippedSeq
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1733) ----------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1734) To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1735) timestamp (to pass PAWS check) but the sequence number is out of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1736) window. The linux TCP stack would avoid to skip if the packet has
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1737) data, so we need a pure ACK packet. To generate such a packet, we
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1738) could create two sockets: one on port 9000, another on port 9001. Then
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1739) we capture an ACK on port 9001, change the source/destination port
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1740) numbers to match the port 9000 socket. Then we could trigger
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1741) TcpExtTCPACKSkippedSeq via this packet.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1742)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1743) On nstat-b, open two terminals, run two nc commands to listen on both
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1744) port 9000 and port 9001::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1745)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1746) nstatuser@nstat-b:~$ nc -lkv 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1747) Listening on [0.0.0.0] (family 0, port 9000)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1748)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1749) nstatuser@nstat-b:~$ nc -lkv 9001
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1750) Listening on [0.0.0.0] (family 0, port 9001)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1751)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1752) On nstat-a, run two nc clients::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1753)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1754) nstatuser@nstat-a:~$ nc -v nstat-b 9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1755) Connection to nstat-b 9000 port [tcp/*] succeeded!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1756)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1757) nstatuser@nstat-a:~$ nc -v nstat-b 9001
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1758) Connection to nstat-b 9001 port [tcp/*] succeeded!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1759)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1760) On nstat-a, run tcpdump to capture an ACK::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1761)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1762) nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1763) tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1764)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1765) On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1766) string 'foo' in our example::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1767)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1768) nstatuser@nstat-b:~$ nc -lkv 9001
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1769) Listening on [0.0.0.0] (family 0, port 9001)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1770) Connection from nstat-a 42132 received!
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1771) foo
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1772)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1773) On nstat-a, the tcpdump should have caputred the ACK. We should check
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1774) the source port numbers of the two nc clients::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1775)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1776) nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1777) State Recv-Q Send-Q Local Address:Port Peer Address:Port
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1778) ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1779) ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1780)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1781) Run tcprewrite, change port 9001 to port 9000, chagne port 42132 to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1782) port 50208::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1783)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1784) nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1785)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1786) Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1787)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1788) nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1789)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1790) Check TcpExtTCPACKSkippedSeq on nstat-b::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1791)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1792) nstatuser@nstat-b:~$ nstat | grep -i skip
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1793) TcpExtTCPACKSkippedSeq 1 0.0
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags