Orange Pi5 kernel

^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   1) .. SPDX-License-Identifier: GPL-2.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   2) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   3) .. include:: <isonum.txt>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   4) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   5) =========================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   6) Rockchip Image Signal Processor (rkisp1)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   7) =========================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   8) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   9) Introduction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  10) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  11) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  12) This file documents the driver for the Rockchip ISP1 that is part of RK3288
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  13) and RK3399 SoCs. The driver is located under drivers/staging/media/rkisp1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  14) and uses the Media-Controller API.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  15) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  16) Topology
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  17) ========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  18) .. _rkisp1_topology_graph:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  19) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  20) .. kernel-figure:: rkisp1.dot
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  21)     :alt:   Diagram of the default media pipeline topology
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  22)     :align: center
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  23) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  24) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  25) The driver has 4 video devices:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  26) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  27) - rkisp1_mainpath: capture device for retrieving images, usually in higher
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  28)   resolution.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  29) - rkisp1_selfpath: capture device for retrieving images.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  30) - rkisp1_stats: a metadata capture device that sends statistics.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  31) - rkisp1_params: a metadata output device that receives parameters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  32)   configurations from userspace.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  33) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  34) The driver has 3 subdevices:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  35) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  36) - rkisp1_resizer_mainpath: used to resize and downsample frames for the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  37)   mainpath capture device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  38) - rkisp1_resizer_selfpath: used to resize and downsample frames for the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  39)   selfpath capture device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  40) - rkisp1_isp: is connected to the sensor and is responsible for all the isp
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  41)   operations.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  42) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  43) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  44) rkisp1_mainpath, rkisp1_selfpath - Frames Capture Video Nodes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  45) -------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  46) Those are the `mainpath` and `selfpath` capture devices to capture frames.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  47) Those entities are the DMA engines that write the frames to memory.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  48) The selfpath video device can capture YUV/RGB formats. Its input is YUV encoded
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  49) stream and it is able to convert it to RGB. The selfpath is not able to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  50) capture bayer formats.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  51) The mainpath can capture both bayer and YUV formats but it is not able to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  52) capture RGB formats.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  53) Both capture videos support
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  54) the ``V4L2_CAP_IO_MC`` :ref:`capability <device-capabilities>`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  55) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  56) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  57) rkisp1_resizer_mainpath, rkisp1_resizer_selfpath - Resizers Subdevices Nodes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  58) ----------------------------------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  59) Those are resizer entities for the mainpath and the selfpath. Those entities
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  60) can scale the frames up and down and also change the YUV sampling (for example
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  61) YUV4:2:2 -> YUV4:2:0). They also have cropping capability on the sink pad.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  62) The resizers entities can only operate on YUV:4:2:2 format
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  63) (MEDIA_BUS_FMT_YUYV8_2X8).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  64) The mainpath capture device supports capturing video in bayer formats. In that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  65) case the resizer of the mainpath is set to 'bypass' mode - it just forward the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  66) frame without operating on it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  67) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  68) rkisp1_isp - Image Signal Processing Subdevice Node
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  69) ---------------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  70) This is the isp entity. It is connected to the sensor on sink pad 0 and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  71) receives the frames using the CSI-2 protocol. It is responsible of configuring
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  72) the CSI-2 protocol. It has a cropping capability on sink pad 0 that is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  73) connected to the sensor and on source pad 2 connected to the resizer entities.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  74) Cropping on sink pad 0 defines the image region from the sensor.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  75) Cropping on source pad 2 defines the region for the Image Stabilizer (IS).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  76) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  77) .. _rkisp1_stats:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  78) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  79) rkisp1_stats - Statistics Video Node
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  80) ------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  81) The statistics video node outputs the 3A (auto focus, auto exposure and auto
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  82) white balance) statistics, and also histogram statistics for the frames that
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  83) are being processed by the rkisp1 to userspace applications.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  84) Using these data, applications can implement algorithms and re-parameterize
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  85) the driver through the rkisp_params node to improve image quality during a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  86) video stream.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  87) The buffer format is defined by struct :c:type:`rkisp1_stat_buffer`, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  88) userspace should set
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  89) :ref:`V4L2_META_FMT_RK_ISP1_STAT_3A <v4l2-meta-fmt-stat-rkisp1>` as the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  90) dataformat.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  91) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  92) .. _rkisp1_params:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  93) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  94) rkisp1_params - Parameters Video Node
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  95) -------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  96) The rkisp1_params video node receives a set of parameters from userspace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  97) to be applied to the hardware during a video stream, allowing userspace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  98) to dynamically modify values such as black level, cross talk corrections
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  99) and others.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) The buffer format is defined by struct :c:type:`rkisp1_params_cfg`, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) userspace should set
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) :ref:`V4L2_META_FMT_RK_ISP1_PARAMS <v4l2-meta-fmt-params-rkisp1>` as the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) dataformat.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) Capturing Video Frames Example
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) ==============================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) In the following example, the sensor connected to pad 0 of 'rkisp1_isp' is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) imx219.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) The following commands can be used to capture video from the selfpath video
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) node with dimension 900x800 planar format YUV 4:2:2. It uses all cropping
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) capabilities possible, (see explanation right below)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) .. code-block:: bash
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) 	# set the links
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) 	"media-ctl" "-d" "platform:rkisp1" "-r"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) 	"media-ctl" "-d" "platform:rkisp1" "-l" "'imx219 4-0010':0 -> 'rkisp1_isp':0 [1]"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) 	"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_selfpath':0 [1]"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) 	"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_mainpath':0 [0]"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) 	# set format for imx219 4-0010:0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) 	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1640x1232]'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) 	# set format for rkisp1_isp pads:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) 	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":0 [fmt:SRGGB10_1X10/1640x1232 crop: (0,0)/1600x1200]'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) 	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":2 [fmt:YUYV8_2X8/1600x1200 crop: (0,0)/1500x1100]'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) 	# set format for rkisp1_resizer_selfpath pads:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) 	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":0 [fmt:YUYV8_2X8/1500x1100 crop: (300,400)/1400x1000]'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) 	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":1 [fmt:YUYV8_2X8/900x800]'
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) 	# set format for rkisp1_selfpath:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) 	"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "width=900,height=800,"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) 	"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "pixelformat=422P"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) 	# start streaming:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) 	v4l2-ctl "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "--stream-mmap" "--stream-count" "10"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) In the above example the sensor is configured to bayer format:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) `SRGGB10_1X10/1640x1232`. The rkisp1_isp:0 pad should be configured to the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) same mbus format and dimensions as the sensor, otherwise streaming will fail
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) with 'EPIPE' error. So it is also configured to `SRGGB10_1X10/1640x1232`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) In addition, the rkisp1_isp:0 pad is configured to cropping `(0,0)/1600x1200`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) The cropping dimensions are automatically propagated to be the format of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) isp source pad `rkisp1_isp:2`. Another cropping operation is configured on
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) the isp source pad: `(0,0)/1500x1100`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) The resizer's sink pad `rkisp1_resizer_selfpath` should be configured to format
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155) `YUYV8_2X8/1500x1100` in order to match the format on the other side of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) link. In addition a cropping `(300,400)/1400x1000` is configured on it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) The source pad of the resizer, `rkisp1_resizer_selfpath:1` is configured to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159) format `YUYV8_2X8/900x800`. That means that the resizer first crop a window
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) of `(300,400)/1400x100` from the received frame and then scales this window
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) to dimension `900x800`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163) Note that the above example does not uses the stats-params control loop.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) Therefore the capture frames will not go through the 3A algorithms and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) probably won't have a good quality, and can even look dark and greenish.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) Configuring Quantization
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) ========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) The driver supports limited and full range quantization on YUV formats,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) where limited is the default.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) To switch between one or the other, userspace should use the Colorspace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) Conversion API (CSC) for subdevices on source pad 2 of the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) isp (`rkisp1_isp:2`). The quantization configured on this pad is the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) quantization of the captured video frames on the mainpath and selfpath
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) video nodes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) Note that the resizer and capture entities will always report
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) ``V4L2_QUANTIZATION_DEFAULT`` even if the quantization is configured to full
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) range on `rkisp1_isp:2`. So in order to get the configured quantization,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) application should get it from pad `rkisp1_isp:2`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181)