^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) .. _slub:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) ==========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4) Short users guide for SLUB
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) ==========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7) The basic philosophy of SLUB is very different from SLAB. SLAB
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8) requires rebuilding the kernel to activate debug options for all
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) slab caches. SLUB always includes full debugging but it is off by default.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10) SLUB can enable debugging only for selected slabs in order to avoid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) an impact on overall system performance which may make a bug more
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) difficult to find.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) In order to switch debugging on one can add an option ``slub_debug``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) to the kernel command line. That will enable full debugging for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) all slabs.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) Typically one would then use the ``slabinfo`` command to get statistical
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) data and perform operation on the slabs. By default ``slabinfo`` only lists
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) slabs that have data in them. See "slabinfo -h" for more options when
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21) running the command. ``slabinfo`` can be compiled with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) ::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) gcc -o slabinfo tools/vm/slabinfo.c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26) Some of the modes of operation of ``slabinfo`` require that slub debugging
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) be enabled on the command line. F.e. no tracking information will be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) available without debugging on and validation can only partially
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) be performed if debugging was not switched on.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31) Some more sophisticated uses of slub_debug:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) -------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) Parameters may be given to ``slub_debug``. If none is specified then full
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35) debugging is enabled. Format:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) slub_debug=<Debug-Options>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) Enable options for all slabs
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) slub_debug=<Debug-Options>,<slab name1>,<slab name2>,...
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) Enable options only for select slabs (no spaces
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42) after a comma)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) Multiple blocks of options for all slabs or selected slabs can be given, with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) blocks of options delimited by ';'. The last of "all slabs" blocks is applied
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46) to all slabs except those that match one of the "select slabs" block. Options
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) of the first "select slabs" blocks that matches the slab's name are applied.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) Possible debug options are::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52) Sorry SLAB legacy issues)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) Z Red zoning
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54) P Poisoning (object and padding)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) U User tracking (free and alloc)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) T Trace (please only use on single slabs)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) A Enable failslab filter mark for the cache
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58) O Switch debugging off for caches that would have
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) caused higher minimum slab orders
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) - Switch all debugging off (useful if the kernel is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) configured with CONFIG_SLUB_DEBUG_ON)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) F.e. in order to boot just with sanity checks and red zoning one would specify::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) slub_debug=FZ
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) Trying to find an issue in the dentry cache? Try::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69) slub_debug=,dentry
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) to only enable debugging on the dentry cache. You may use an asterisk at the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72) end of the slab name, in order to cover all slabs with the same prefix. For
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) example, here's how you can poison the dentry cache as well as all kmalloc
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) slabs::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76) slub_debug=P,kmalloc-*,dentry
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) Red zoning and tracking may realign the slab. We can just apply sanity checks
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) to the dentry cache with::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) slub_debug=F,dentry
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83) Debugging options may require the minimum possible slab order to increase as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) a result of storing the metadata (for example, caches with PAGE_SIZE object
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85) sizes). This has a higher liklihood of resulting in slab allocation errors
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) in low memory situations or if there's high fragmentation of memory. To
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) switch off debugging for such caches by default, use::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) slub_debug=O
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91) You can apply different options to different list of slab names, using blocks
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) of options. This will enable red zoning for dentry and user tracking for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93) kmalloc. All other slabs will not get any debugging enabled::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) slub_debug=Z,dentry;U,kmalloc-*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) You can also enable options (e.g. sanity checks and poisoning) for all caches
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) except some that are deemed too performance critical and don't need to be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99) debugged by specifying global debug options followed by a list of slab names
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) with "-" as options::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) slub_debug=FZ;-,zs_handle,zspage
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) The state of each debug option for a slab can be found in the respective files
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) under::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) /sys/kernel/slab/<slab name>/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) If the file contains 1, the option is enabled, 0 means disabled. The debug
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) options from the ``slub_debug`` parameter translate to the following files::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) F sanity_checks
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) Z red_zone
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) P poison
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) U store_user
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) T trace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) A failslab
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) Careful with tracing: It may spew out lots of information and never stop if
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) used on the wrong slab.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) Slab merging
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) If no debug options are specified then SLUB may merge similar slabs together
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) in order to reduce overhead and increase cache hotness of objects.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) ``slabinfo -a`` displays which slabs were merged together.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) Slab validation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) ===============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) SLUB can validate all object if the kernel was booted with slub_debug. In
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) order to do so you must have the ``slabinfo`` tool. Then you can do
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) ::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) slabinfo -v
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) which will test all objects. Output will be generated to the syslog.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) This also works in a more limited way if boot was without slab debug.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) In that case ``slabinfo -v`` simply tests all reachable objects. Usually
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) these are in the cpu slabs and the partial slabs. Full slabs are not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) tracked by SLUB in a non debug situation.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) Getting more performance
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) ========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) To some degree SLUB's performance is limited by the need to take the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) list_lock once in a while to deal with partial slabs. That overhead is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) governed by the order of the allocation for each slab. The allocations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) can be influenced by kernel parameters:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) .. slub_min_objects=x (default 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) .. slub_min_order=x (default 0)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) .. slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) ``slub_min_objects``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) allows to specify how many objects must at least fit into one
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) slab in order for the allocation order to be acceptable. In
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) general slub will be able to perform this number of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) allocations on a slab without consulting centralized resources
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) (list_lock) where contention may occur.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) ``slub_min_order``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) specifies a minimum order of slabs. A similar effect like
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) ``slub_min_objects``.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) ``slub_max_order``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) specified the order at which ``slub_min_objects`` should no
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) longer be checked. This is useful to avoid SLUB trying to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) generate super large order pages to fit ``slub_min_objects``
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) of a slab cache with large object sizes into one high order
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) page. Setting command line parameter
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) ``debug_guardpage_minorder=N`` (N > 0), forces setting
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) ``slub_max_order`` to 0, what cause minimum possible order of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) slabs allocation.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) SLUB Debug output
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) =================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) Here is a sample of slub debug output::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) ====================================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) BUG kmalloc-8: Right Redzone overwritten
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) --------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) Bytes b4 (0xc90f6d10): 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) Object (0xc90f6d20): 31 30 31 39 2e 30 30 35 1019.005
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) Redzone (0xc90f6d28): 00 cc cc cc .
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) Padding (0xc90f6d50): 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) [<c010523d>] dump_trace+0x63/0x1eb
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) [<c01053df>] show_trace_log_lvl+0x1a/0x2f
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) [<c010601d>] show_trace+0x12/0x14
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) [<c0106035>] dump_stack+0x16/0x18
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) [<c017e0fa>] object_err+0x143/0x14b
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) [<c017e2cc>] check_object+0x66/0x234
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) [<c017eb43>] __slab_free+0x239/0x384
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) [<c017f446>] kfree+0xa6/0xc6
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) [<c02e2335>] get_modalias+0xb9/0xf5
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) [<c02e23b7>] dmi_dev_uevent+0x27/0x3c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) [<c027866a>] dev_uevent+0x1ad/0x1da
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) [<c0205024>] kobject_uevent_env+0x20a/0x45b
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209) [<c020527f>] kobject_uevent+0xa/0xf
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) [<c02779f1>] store_uevent+0x4f/0x58
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) [<c027758e>] dev_attr_store+0x29/0x2f
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212) [<c01bec4f>] sysfs_write_file+0x16e/0x19c
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) [<c0183ba7>] vfs_write+0xd1/0x15a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) [<c01841d7>] sys_write+0x3d/0x72
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215) [<c0104112>] sysenter_past_esp+0x5f/0x99
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) [<b7f7b410>] 0xb7f7b410
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) =======================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) If SLUB encounters a corrupted object (full detection requires the kernel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) to be booted with slub_debug) then the following output will be dumped
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) into the syslog:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) 1. Description of the problem encountered
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) This will be a message in the system log starting with::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) ===============================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) BUG <slab cache affected>: <What went wrong>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231) -----------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233) INFO: <corruption start>-<corruption_end> <more info>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234) INFO: Slab <address> <slab information>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) INFO: Object <address> <object information>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236) INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237) cpu> pid=<pid of the process>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) pid=<pid of the process>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241) (Object allocation / free information is only available if SLAB_STORE_USER is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) set for the slab. slub_debug sets that option)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) 2. The object contents if an object was involved.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246) Various types of lines can follow the BUG SLUB line:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) Bytes b4 <address> : <bytes>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249) Shows a few bytes before the object where the problem was detected.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) Can be useful if the corruption does not stop with the start of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) object.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) Object <address> : <bytes>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) The bytes of the object. If the object is inactive then the bytes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) typically contain poison values. Any non-poison value shows a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256) corruption by a write after free.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258) Redzone <address> : <bytes>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) The Redzone following the object. The Redzone is used to detect
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260) writes after the object. All bytes should always have the same
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) value. If there is any deviation then it is due to a write after
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) the object boundary.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) (Redzone information is only available if SLAB_RED_ZONE is set.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) slub_debug sets that option)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) Padding <address> : <bytes>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268) Unused data to fill up the space in order to get the next object
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269) properly aligned. In the debug case we make sure that there are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) at least 4 bytes of padding. This allows the detection of writes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271) before the object.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273) 3. A stackdump
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) The stackdump describes the location where the error was detected. The cause
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276) of the corruption is may be more likely found by looking at the function that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277) allocated or freed the object.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279) 4. Report on how the problem was dealt with in order to ensure the continued
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280) operation of the system.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282) These are messages in the system log beginning with::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284) FIX <slab cache affected>: <corrective action taken>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286) In the above sample SLUB found that the Redzone of an active object has
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287) been overwritten. Here a string of 8 characters was written into a slab that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288) has the length of 8 characters. However, a 8 character string needs a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289) terminating 0. That zero has overwritten the first byte of the Redzone field.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) After reporting the details of the issue encountered the FIX SLUB message
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291) tells us that SLUB has restored the Redzone to its proper value and then
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292) system operations continue.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294) Emergency operations
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295) ====================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297) Minimal debugging (sanity checks alone) can be enabled by booting with::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299) slub_debug=F
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301) This will be generally be enough to enable the resiliency features of slub
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302) which will keep the system running even if a bad kernel component will
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303) keep corrupting objects. This may be important for production systems.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) Performance will be impacted by the sanity checks and there will be a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305) continual stream of error messages to the syslog but no additional memory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306) will be used (unlike full debugging).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308) No guarantees. The kernel component still needs to be fixed. Performance
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309) may be optimized further by locating the slab that experiences corruption
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310) and enabling debugging only for that cache
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312) I.e.::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314) slub_debug=F,dentry
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) If the corruption occurs by writing after the end of the object then it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317) may be advisable to enable a Redzone to avoid corrupting the beginning
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318) of other objects::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320) slub_debug=FZ,dentry
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322) Extended slabinfo mode and plotting
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323) ===================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325) The ``slabinfo`` tool has a special 'extended' ('-X') mode that includes:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326) - Slabcache Totals
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327) - Slabs sorted by size (up to -N <num> slabs, default 1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328) - Slabs sorted by loss (up to -N <num> slabs, default 1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330) Additionally, in this mode ``slabinfo`` does not dynamically scale
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331) sizes (G/M/K) and reports everything in bytes (this functionality is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332) also available to other slabinfo modes via '-B' option) which makes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333) reporting more precise and accurate. Moreover, in some sense the `-X'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334) mode also simplifies the analysis of slabs' behaviour, because its
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335) output can be plotted using the ``slabinfo-gnuplot.sh`` script. So it
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336) pushes the analysis from looking through the numbers (tons of numbers)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) to something easier -- visual analysis.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339) To generate plots:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341) a) collect slabinfo extended records, for example::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343) while [ 1 ]; do slabinfo -X >> FOO_STATS; sleep 1; done
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345) b) pass stats file(-s) to ``slabinfo-gnuplot.sh`` script::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347) slabinfo-gnuplot.sh FOO_STATS [FOO_STATS2 .. FOO_STATSN]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349) The ``slabinfo-gnuplot.sh`` script will pre-processes the collected records
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) and generates 3 png files (and 3 pre-processing cache files) per STATS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351) file:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352) - Slabcache Totals: FOO_STATS-totals.png
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353) - Slabs sorted by size: FOO_STATS-slabs-by-size.png
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354) - Slabs sorted by loss: FOO_STATS-slabs-by-loss.png
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) Another use case, when ``slabinfo-gnuplot.sh`` can be useful, is when you
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357) need to compare slabs' behaviour "prior to" and "after" some code
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358) modification. To help you out there, ``slabinfo-gnuplot.sh`` script
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359) can 'merge' the `Slabcache Totals` sections from different
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360) measurements. To visually compare N plots:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362) a) Collect as many STATS1, STATS2, .. STATSN files as you need::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364) while [ 1 ]; do slabinfo -X >> STATS<X>; sleep 1; done
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366) b) Pre-process those STATS files::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368) slabinfo-gnuplot.sh STATS1 STATS2 .. STATSN
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370) c) Execute ``slabinfo-gnuplot.sh`` in '-t' mode, passing all of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) generated pre-processed \*-totals::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373) slabinfo-gnuplot.sh -t STATS1-totals STATS2-totals .. STATSN-totals
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375) This will produce a single plot (png file).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377) Plots, expectedly, can be large so some fluctuations or small spikes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378) can go unnoticed. To deal with that, ``slabinfo-gnuplot.sh`` has two
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379) options to 'zoom-in'/'zoom-out':
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 380)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 381) a) ``-s %d,%d`` -- overwrites the default image width and height
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 382) b) ``-r %d,%d`` -- specifies a range of samples to use (for example,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 383) in ``slabinfo -X >> FOO_STATS; sleep 1;`` case, using a ``-r
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 384) 40,60`` range will plot only samples collected between 40th and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 385) 60th seconds).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 386)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 387) Christoph Lameter, May 30, 2007
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 388) Sergey Senozhatsky, October 23, 2015
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags