Orange Pi5 kernel

^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   1) ================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   2) RAID 4/5/6 cache
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   3) ================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   4) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   5) Raid 4/5/6 could include an extra disk for data cache besides normal RAID
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   6) disks. The role of RAID disks isn't changed with the cache disk. The cache disk
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   7) caches data to the RAID disks. The cache can be in write-through (supported
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   8) since 4.4) or write-back mode (supported since 4.10). mdadm (supported since
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   9) 3.4) has a new option '--write-journal' to create array with cache. Please
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  10) refer to mdadm manual for details. By default (RAID array starts), the cache is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  11) in write-through mode. A user can switch it to write-back mode by::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  12) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  13) 	echo "write-back" > /sys/block/md0/md/journal_mode
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  14) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  15) And switch it back to write-through mode by::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  16) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  17) 	echo "write-through" > /sys/block/md0/md/journal_mode
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  18) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  19) In both modes, all writes to the array will hit cache disk first. This means
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  20) the cache disk must be fast and sustainable.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  21) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  22) write-through mode
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  23) ==================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  24) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  25) This mode mainly fixes the 'write hole' issue. For RAID 4/5/6 array, an unclean
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  26) shutdown can cause data in some stripes to not be in consistent state, eg, data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  27) and parity don't match. The reason is that a stripe write involves several RAID
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  28) disks and it's possible the writes don't hit all RAID disks yet before the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  29) unclean shutdown. We call an array degraded if it has inconsistent data. MD
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  30) tries to resync the array to bring it back to normal state. But before the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  31) resync completes, any system crash will expose the chance of real data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  32) corruption in the RAID array. This problem is called 'write hole'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  33) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  34) The write-through cache will cache all data on cache disk first. After the data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  35) is safe on the cache disk, the data will be flushed onto RAID disks. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  36) two-step write will guarantee MD can recover correct data after unclean
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  37) shutdown even the array is degraded. Thus the cache can close the 'write hole'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  38) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  39) In write-through mode, MD reports IO completion to upper layer (usually
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  40) filesystems) after the data is safe on RAID disks, so cache disk failure
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  41) doesn't cause data loss. Of course cache disk failure means the array is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  42) exposed to 'write hole' again.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  43) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  44) In write-through mode, the cache disk isn't required to be big. Several
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  45) hundreds megabytes are enough.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  46) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  47) write-back mode
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  48) ===============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  49) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  50) write-back mode fixes the 'write hole' issue too, since all write data is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  51) cached on cache disk. But the main goal of 'write-back' cache is to speed up
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  52) write. If a write crosses all RAID disks of a stripe, we call it full-stripe
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  53) write. For non-full-stripe writes, MD must read old data before the new parity
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  54) can be calculated. These synchronous reads hurt write throughput. Some writes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  55) which are sequential but not dispatched in the same time will suffer from this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  56) overhead too. Write-back cache will aggregate the data and flush the data to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  57) RAID disks only after the data becomes a full stripe write. This will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  58) completely avoid the overhead, so it's very helpful for some workloads. A
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  59) typical workload which does sequential write followed by fsync is an example.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  60) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  61) In write-back mode, MD reports IO completion to upper layer (usually
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  62) filesystems) right after the data hits cache disk. The data is flushed to raid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  63) disks later after specific conditions met. So cache disk failure will cause
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  64) data loss.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  65) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  66) In write-back mode, MD also caches data in memory. The memory cache includes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  67) the same data stored on cache disk, so a power loss doesn't cause data loss.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  68) The memory cache size has performance impact for the array. It's recommended
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  69) the size is big. A user can configure the size by::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  70) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  71) 	echo "2048" > /sys/block/md0/md/stripe_cache_size
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  72) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  73) Too small cache disk will make the write aggregation less efficient in this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  74) mode depending on the workloads. It's recommended to use a cache disk with at
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  75) least several gigabytes size in write-back mode.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  76) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  77) The implementation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  78) ==================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  79) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  80) The write-through and write-back cache use the same disk format. The cache disk
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  81) is organized as a simple write log. The log consists of 'meta data' and 'data'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  82) pairs. The meta data describes the data. It also includes checksum and sequence
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  83) ID for recovery identification. Data can be IO data and parity data. Data is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  84) checksumed too. The checksum is stored in the meta data ahead of the data. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  85) checksum is an optimization because MD can write meta and data freely without
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  86) worry about the order. MD superblock has a field pointed to the valid meta data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  87) of log head.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  88) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  89) The log implementation is pretty straightforward. The difficult part is the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  90) order in which MD writes data to cache disk and RAID disks. Specifically, in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  91) write-through mode, MD calculates parity for IO data, writes both IO data and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  92) parity to the log, writes the data and parity to RAID disks after the data and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  93) parity is settled down in log and finally the IO is finished. Read just reads
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  94) from raid disks as usual.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  95) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  96) In write-back mode, MD writes IO data to the log and reports IO completion. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  97) data is also fully cached in memory at that time, which means read must query
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  98) memory cache. If some conditions are met, MD will flush the data to RAID disks.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  99) MD will calculate parity for the data and write parity into the log. After this
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) is finished, MD will write both data and parity into RAID disks, then MD can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) release the memory cache. The flush conditions could be stripe becomes a full
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) stripe write, free cache disk space is low or free in-kernel memory cache space
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) is low.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) After an unclean shutdown, MD does recovery. MD reads all meta data and data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) from the log. The sequence ID and checksum will help us detect corrupted meta
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) data and data. If MD finds a stripe with data and valid parities (1 parity for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) raid4/5 and 2 for raid6), MD will write the data and parities to RAID disks. If
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) parities are incompleted, they are discarded. If part of data is corrupted,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) they are discarded too. MD then loads valid data and writes them to RAID disks
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) in normal way.