^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) =====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) I/O statistics fields
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) =====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) Since 2.4.20 (and some versions before, with patches), and 2.5.45,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) more extensive disk statistics have been introduced to help measure disk
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7) activity. Tools such as ``sar`` and ``iostat`` typically interpret these and do
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) the work for you, but in case you are interested in creating your own
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) tools, the fields are explained here.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) In 2.4 now, the information is found as additional fields in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) ``/proc/partitions``. In 2.6 and upper, the same information is found in two
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) places: one is in the file ``/proc/diskstats``, and the other is within
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) the sysfs file system, which must be mounted in order to obtain
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) the information. Throughout this document we'll assume that sysfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) is mounted on ``/sys``, although of course it may be mounted anywhere.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) Both ``/proc/diskstats`` and sysfs use the same source for the information
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) and so should not differ.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) Here are examples of these different formats::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) 2.4:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) 3 0 39082680 hda 446216 784926 9550688 4382310 424847 312726 5922052 19310380 0 3376340 23705160
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) 3 1 9221278 hda1 35486 0 35496 38030 0 0 0 0 0 38030 38030
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) 2.6+ sysfs:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) 446216 784926 9550688 4382310 424847 312726 5922052 19310380 0 3376340 23705160
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) 35486 38030 38030 38030
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) 2.6+ diskstats:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) 3 0 hda 446216 784926 9550688 4382310 424847 312726 5922052 19310380 0 3376340 23705160
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) 3 1 hda1 35486 38030 38030 38030
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) 4.18+ diskstats:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) 3 0 hda 446216 784926 9550688 4382310 424847 312726 5922052 19310380 0 3376340 23705160 0 0 0 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) On 2.4 you might execute ``grep 'hda ' /proc/partitions``. On 2.6+, you have
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) a choice of ``cat /sys/block/hda/stat`` or ``grep 'hda ' /proc/diskstats``.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) The advantage of one over the other is that the sysfs choice works well
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) if you are watching a known, small set of disks. ``/proc/diskstats`` may
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) be a better choice if you are watching a large number of disks because
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) you'll avoid the overhead of 50, 100, or 500 or more opens/closes with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) each snapshot of your disk statistics.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) In 2.4, the statistics fields are those after the device name. In
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) the above example, the first field of statistics would be 446216.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) By contrast, in 2.6+ if you look at ``/sys/block/hda/stat``, you'll
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) find just the 15 fields, beginning with 446216. If you look at
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) ``/proc/diskstats``, the 15 fields will be preceded by the major and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) minor device numbers, and device name. Each of these formats provides
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) 15 fields of statistics, each meaning exactly the same things.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) All fields except field 9 are cumulative since boot. Field 9 should
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) go to zero as I/Os complete; all others only increase (unless they
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) overflow and wrap). Wrapping might eventually occur on a very busy
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) or long-lived system; so applications should be prepared to deal with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) it. Regarding wrapping, the types of the fields are either unsigned
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) int (32 bit) or unsigned long (32-bit or 64-bit, depending on your
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) machine) as noted per-field below. Unless your observations are very
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) spread in time, these fields should not wrap twice before you notice it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) Each set of stats only applies to the indicated device; if you want
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) system-wide stats you'll have to find all the devices and sum them all up.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) Field 1 -- # of reads completed (unsigned long)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66) This is the total number of reads completed successfully.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) Field 2 -- # of reads merged, field 6 -- # of writes merged (unsigned long)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) Reads and writes which are adjacent to each other may be merged for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) efficiency. Thus two 4K reads may become one 8K read before it is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) ultimately handed to the disk, and so it will be counted (and queued)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) as only one I/O. This field lets you know how often this was done.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) Field 3 -- # of sectors read (unsigned long)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) This is the total number of sectors read successfully.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) Field 4 -- # of milliseconds spent reading (unsigned int)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) This is the total number of milliseconds spent by all reads (as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) measured from __make_request() to end_that_request_last()).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) Field 5 -- # of writes completed (unsigned long)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) This is the total number of writes completed successfully.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) Field 6 -- # of writes merged (unsigned long)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) See the description of field 2.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) Field 7 -- # of sectors written (unsigned long)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) This is the total number of sectors written successfully.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) Field 8 -- # of milliseconds spent writing (unsigned int)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) This is the total number of milliseconds spent by all writes (as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) measured from __make_request() to end_that_request_last()).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) Field 9 -- # of I/Os currently in progress (unsigned int)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) The only field that should go to zero. Incremented as requests are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) given to appropriate struct request_queue and decremented as they finish.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) Field 10 -- # of milliseconds spent doing I/Os (unsigned int)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) This field increases so long as field 9 is nonzero.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) Since 5.0 this field counts jiffies when at least one request was
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) started or completed. If request runs more than 2 jiffies then some
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) I/O time might be not accounted in case of concurrent requests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) Field 11 -- weighted # of milliseconds spent doing I/Os (unsigned int)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) This field is incremented at each I/O start, I/O completion, I/O
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) merge, or read of these stats by the number of I/Os in progress
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) (field 9) times the number of milliseconds spent doing I/O since the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) last update of this field. This can provide an easy measure of both
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) I/O completion time and the backlog that may be accumulating.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) Field 12 -- # of discards completed (unsigned long)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) This is the total number of discards completed successfully.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) Field 13 -- # of discards merged (unsigned long)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) See the description of field 2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) Field 14 -- # of sectors discarded (unsigned long)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) This is the total number of sectors discarded successfully.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) Field 15 -- # of milliseconds spent discarding (unsigned int)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) This is the total number of milliseconds spent by all discards (as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) measured from __make_request() to end_that_request_last()).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) Field 16 -- # of flush requests completed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) This is the total number of flush requests completed successfully.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) Block layer combines flush requests and executes at most one at a time.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) This counts flush requests executed by disk. Not tracked for partitions.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) Field 17 -- # of milliseconds spent flushing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) This is the total number of milliseconds spent by all flush requests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) To avoid introducing performance bottlenecks, no locks are held while
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) modifying these counters. This implies that minor inaccuracies may be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) introduced when changes collide, so (for instance) adding up all the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) read I/Os issued per partition should equal those made to the disks ...
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) but due to the lack of locking it may only be very close.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) In 2.6+, there are counters for each CPU, which make the lack of locking
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) almost a non-issue. When the statistics are read, the per-CPU counters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) are summed (possibly overflowing the unsigned long variable they are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) summed to) and the result given to the user. There is no convenient
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) user interface for accessing the per-CPU counters themselves.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) Since 4.19 request times are measured with nanoseconds precision and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) truncated to milliseconds before showing in this interface.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) Disks vs Partitions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) -------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) There were significant changes between 2.4 and 2.6+ in the I/O subsystem.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) As a result, some statistic information disappeared. The translation from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) a disk address relative to a partition to the disk address relative to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) the host disk happens much earlier. All merges and timings now happen
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) at the disk level rather than at both the disk and partition level as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) in 2.4. Consequently, you'll see a different statistics output on 2.6+ for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) partitions from that for disks. There are only *four* fields available
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) for partitions on 2.6+ machines. This is reflected in the examples above.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) Field 1 -- # of reads issued
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) This is the total number of reads issued to this partition.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) Field 2 -- # of sectors read
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) This is the total number of sectors requested to be read from this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) partition.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) Field 3 -- # of writes issued
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) This is the total number of writes issued to this partition.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) Field 4 -- # of sectors written
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) This is the total number of sectors requested to be written to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) this partition.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) Note that since the address is translated to a disk-relative one, and no
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) record of the partition-relative address is kept, the subsequent success
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) or failure of the read cannot be attributed to the partition. In other
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) words, the number of reads for partitions is counted slightly before time
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) of queuing for partitions, and at completion for whole disks. This is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) a subtle distinction that is probably uninteresting for most cases.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) More significant is the error induced by counting the numbers of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) reads/writes before merges for partitions and after for disks. Since a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) typical workload usually contains a lot of successive and adjacent requests,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) the number of reads/writes issued can be several times higher than the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) number of reads/writes completed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) In 2.6.25, the full statistic set is again available for partitions and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) disk and partition statistics are consistent again. Since we still don't
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) keep record of the partition-relative address, an operation is attributed to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) the partition which contains the first sector of the request after the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) eventual merges. As requests can be merged across partition, this could lead
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) to some (probably insignificant) inaccuracy.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) Additional notes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) ----------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) In 2.6+, sysfs is not mounted by default. If your distribution of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) Linux hasn't added it already, here's the line you'll want to add to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) your ``/etc/fstab``::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) none /sys sysfs defaults 0 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) In 2.6+, all disk statistics were removed from ``/proc/stat``. In 2.4, they
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) appear in both ``/proc/partitions`` and ``/proc/stat``, although the ones in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) ``/proc/stat`` take a very different format from those in ``/proc/partitions``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) (see proc(5), if your system has it.)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) -- ricklind@us.ibm.com