^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) Notes on Filesystem Layout
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2) --------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4) These notes describe what mkcramfs generates. Kernel requirements are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) a bit looser, e.g. it doesn't care if the <file_data> items are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) swapped around (though it does care that directory entries (inodes) in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7) a given directory are contiguous, as this is used by readdir).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) All data is currently in host-endian format; neither mkcramfs nor the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) kernel ever do swabbing. (See section `Block Size' below.)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) <filesystem>:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13) <superblock>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) <directory_structure>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) <data>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17) <superblock>: struct cramfs_super (see cramfs_fs.h).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) <directory_structure>:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) For each file:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) struct cramfs_inode (see cramfs_fs.h).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) Filename. Not generally null-terminated, but it is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) null-padded to a multiple of 4 bytes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) The order of inode traversal is described as "width-first" (not to be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) confused with breadth-first); i.e. like depth-first but listing all of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) a directory's entries before recursing down its subdirectories: the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) same order as `ls -AUR' (but without the /^\..*:$/ directory header
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) lines); put another way, the same order as `find -type d -exec
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) ls -AU1 {} \;'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) Beginning in 2.4.7, directory entries are sorted. This optimization
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33) allows cramfs_lookup to return more quickly when a filename does not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) exist, speeds up user-space directory sorts, etc.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) <data>:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) One <file_data> for each file that's either a symlink or a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) regular file of non-zero st_size.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) <file_data>:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) nblocks * <block_pointer>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) (where nblocks = (st_size - 1) / blksize + 1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) nblocks * <block>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) padding to multiple of 4 bytes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) The i'th <block_pointer> for a file stores the byte offset of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) *end* of the i'th <block> (i.e. one past the last byte, which is the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) same as the start of the (i+1)'th <block> if there is one). The first
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) <block> immediately follows the last <block_pointer> for the file.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50) <block_pointer>s are each 32 bits long.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) When the CRAMFS_FLAG_EXT_BLOCK_POINTERS capability bit is set, each
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) <block_pointer>'s top bits may contain special flags as follows:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) CRAMFS_BLK_FLAG_UNCOMPRESSED (bit 31):
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) The block data is not compressed and should be copied verbatim.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) CRAMFS_BLK_FLAG_DIRECT_PTR (bit 30):
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) The <block_pointer> stores the actual block start offset and not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) its end, shifted right by 2 bits. The block must therefore be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) aligned to a 4-byte boundary. The block size is either blksize
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62) if CRAMFS_BLK_FLAG_UNCOMPRESSED is also specified, otherwise
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) the compressed data length is included in the first 2 bytes of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) the block data. This is used to allow discontiguous data layout
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) and specific data block alignments e.g. for XIP applications.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) The order of <file_data>'s is a depth-first descent of the directory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) tree, i.e. the same order as `find -size +0 \( -type f -o -type l \)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) -print'.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) <block>: The i'th <block> is the output of zlib's compress function
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) applied to the i'th blksize-sized chunk of the input data if the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) corresponding CRAMFS_BLK_FLAG_UNCOMPRESSED <block_ptr> bit is not set,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) otherwise it is the input data directly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) (For the last <block> of the file, the input may of course be smaller.)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) Each <block> may be a different size. (See <block_pointer> above.)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80) <block>s are merely byte-aligned, not generally u32-aligned.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) When CRAMFS_BLK_FLAG_DIRECT_PTR is specified then the corresponding
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) <block> may be located anywhere and not necessarily contiguous with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) the previous/next blocks. In that case it is minimally u32-aligned.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) If CRAMFS_BLK_FLAG_UNCOMPRESSED is also specified then the size is always
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) blksize except for the last block which is limited by the file length.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) If CRAMFS_BLK_FLAG_DIRECT_PTR is set and CRAMFS_BLK_FLAG_UNCOMPRESSED
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88) is not set then the first 2 bytes of the block contains the size of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) remaining block data as this cannot be determined from the placement of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) logically adjacent blocks.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) Holes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) -----
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96) This kernel supports cramfs holes (i.e. [efficient representation of]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) blocks in uncompressed data consisting entirely of NUL bytes), but by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) default mkcramfs doesn't test for & create holes, since cramfs in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) kernels up to at least 2.3.39 didn't support holes. Run mkcramfs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) with -z if you want it to create files that can have holes in them.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) Tools
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) -----
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) The cramfs user-space tools, including mkcramfs and cramfsck, are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) located at <http://sourceforge.net/projects/cramfs/>.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) Future Development
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) ==================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) Block Size
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) ----------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) (Block size in cramfs refers to the size of input data that is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) compressed at a time. It's intended to be somewhere around
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) PAGE_SIZE for cramfs_readpage's convenience.)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) The superblock ought to indicate the block size that the fs was
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) written for, since comments in <linux/pagemap.h> indicate that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) PAGE_SIZE may grow in future (if I interpret the comment
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) correctly).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) Currently, mkcramfs #define's PAGE_SIZE as 4096 and uses that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) for blksize, whereas Linux-2.3.39 uses its PAGE_SIZE, which in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) turn is defined as PAGE_SIZE (which can be as large as 32KB on arm).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) This discrepancy is a bug, though it's not clear which should be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) changed.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) One option is to change mkcramfs to take its PAGE_SIZE from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) <asm/page.h>. Personally I don't like this option, but it does
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) require the least amount of change: just change `#define
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) PAGE_SIZE (4096)' to `#include <asm/page.h>'. The disadvantage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) is that the generated cramfs cannot always be shared between different
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) kernels, not even necessarily kernels of the same architecture if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) PAGE_SIZE is subject to change between kernel versions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) (currently possible with arm and ia64).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) The remaining options try to make cramfs more sharable.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) One part of that is addressing endianness. The two options here are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) `always use little-endian' (like ext2fs) or `writer chooses
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) endianness; kernel adapts at runtime'. Little-endian wins because of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) code simplicity and little CPU overhead even on big-endian machines.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) The cost of swabbing is changing the code to use the le32_to_cpu
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) etc. macros as used by ext2fs. We don't need to swab the compressed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) data, only the superblock, inodes and block pointers.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) The other part of making cramfs more sharable is choosing a block
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) size. The options are:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) 1. Always 4096 bytes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) 2. Writer chooses blocksize; kernel adapts but rejects blocksize >
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) PAGE_SIZE.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) 3. Writer chooses blocksize; kernel adapts even to blocksize >
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) PAGE_SIZE.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) It's easy enough to change the kernel to use a smaller value than
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) PAGE_SIZE: just make cramfs_readpage read multiple blocks.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) The cost of option 1 is that kernels with a larger PAGE_SIZE
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) value don't get as good compression as they can.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) The cost of option 2 relative to option 1 is that the code uses
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) variables instead of #define'd constants. The gain is that people
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) with kernels having larger PAGE_SIZE can make use of that if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) they don't mind their cramfs being inaccessible to kernels with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) smaller PAGE_SIZE values.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) Option 3 is easy to implement if we don't mind being CPU-inefficient:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) e.g. get readpage to decompress to a buffer of size MAX_BLKSIZE (which
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) must be no larger than 32KB) and discard what it doesn't need.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) Getting readpage to read into all the covered pages is harder.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) The main advantage of option 3 over 1, 2, is better compression. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) cost is greater complexity. Probably not worth it, but I hope someone
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) will disagree. (If it is implemented, then I'll re-use that code in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) e2compr.)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) Another cost of 2 and 3 over 1 is making mkcramfs use a different
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) block size, but that just means adding and parsing a -b option.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) Inode Size
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) ----------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) Given that cramfs will probably be used for CDs etc. as well as just
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) silicon ROMs, it might make sense to expand the inode a little from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) its current 12 bytes. Inodes other than the root inode are followed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) by filename, so the expansion doesn't even have to be a multiple of 4
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) bytes.
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags