^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 1) .. SPDX-License-Identifier: GPL-2.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 2)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 3) .. include:: <isonum.txt>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 4)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 5) ===============================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 6) Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 7) ===============================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 8)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 9) Copyright |copy| 2018 Intel Corporation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 10)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 11) Introduction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 12) ============
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 13)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 14) This file documents the Intel IPU3 (3rd generation Image Processing Unit)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 15) Imaging Unit drivers located under drivers/media/pci/intel/ipu3 (CIO2) as well
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 16) as under drivers/staging/media/ipu3 (ImgU).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 17)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 18) The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 19) platforms (U/Y processor lines) is made up of two parts namely the Imaging Unit
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 20) (ImgU) and the CIO2 device (MIPI CSI2 receiver).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 21)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 22) The CIO2 device receives the raw Bayer data from the sensors and outputs the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 23) frames in a format that is specific to the IPU3 (for consumption by the IPU3
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 24) ImgU). The CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 25) and is enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 26)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 27) The Imaging Unit (ImgU) is responsible for processing images captured
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 28) by the IPU3 CIO2 device. The ImgU driver sources can be found under
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 29) drivers/staging/media/ipu3 directory. The driver is enabled through the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 30) CONFIG_VIDEO_IPU3_IMGU config option.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 31)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 32) The two driver modules are named ipu3_csi2 and ipu3_imgu, respectively.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 33)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 34) The drivers has been tested on Kaby Lake platforms (U/Y processor lines).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 35)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 36) Both of the drivers implement V4L2, Media Controller and V4L2 sub-device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 37) interfaces. The IPU3 CIO2 driver supports camera sensors connected to the CIO2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 38) MIPI CSI-2 interfaces through V4L2 sub-device sensor drivers.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 39)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 40) CIO2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 41) ====
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 42)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 43) The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 44) interface to the user space. There is a video node for each CSI-2 receiver,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 45) with a single media controller interface for the entire device.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 46)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 47) The CIO2 contains four independent capture channel, each with its own MIPI CSI-2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 48) receiver and DMA engine. Each channel is modelled as a V4L2 sub-device exposed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 49) to userspace as a V4L2 sub-device node and has two pads:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 50)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 51) .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 52)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 53) .. flat-table::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 54)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 55) * - pad
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 56) - direction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 57) - purpose
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 58)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 59) * - 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 60) - sink
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 61) - MIPI CSI-2 input, connected to the sensor subdev
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 62)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 63) * - 1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 64) - source
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 65) - Raw video capture, connected to the V4L2 video interface
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 66)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 67) The V4L2 video interfaces model the DMA engines. They are exposed to userspace
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 68) as V4L2 video device nodes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 69)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 70) Capturing frames in raw Bayer format
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 71) ------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 72)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 73) CIO2 MIPI CSI2 receiver is used to capture frames (in packed raw Bayer format)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 74) from the raw sensors connected to the CSI2 ports. The captured frames are used
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 75) as input to the ImgU driver.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 76)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 77) Image processing using IPU3 ImgU requires tools such as raw2pnm [#f1]_, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 78) yavta [#f2]_ due to the following unique requirements and / or features specific
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 79) to IPU3.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 80)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 81) -- The IPU3 CSI2 receiver outputs the captured frames from the sensor in packed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 82) raw Bayer format that is specific to IPU3.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 83)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 84) -- Multiple video nodes have to be operated simultaneously.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 85)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 86) Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 87) 2592x1944 image capture.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 88)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 89) Using the media contorller APIs, the ov5670 sensor is configured to send
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 90) frames in packed raw Bayer format to IPU3 CSI2 receiver.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 91)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 92) .. code-block:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 93)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 94) # This example assumes /dev/media0 as the CIO2 media device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 95) export MDEV=/dev/media0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 96)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 97) # and that ov5670 sensor is connected to i2c bus 10 with address 0x36
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 98) export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036")
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 99)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) # Establish the link for the media devices using media-ctl [#f3]_
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) # Set the format for the media devices
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) Once the media pipeline is configured, desired sensor specific settings
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) (such as exposure and gain settings) can be set, using the yavta tool.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) e.g
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) .. code-block:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) yavta -w 0x009e0903 444 $SDEV
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) yavta -w 0x009e0913 1024 $SDEV
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) yavta -w 0x009e0911 2046 $SDEV
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) Once the desired sensor settings are set, frame captures can be done as below.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) e.g
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) .. code-block:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) -f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0")
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) With the above command, 10 frames are captured at 2592x1944 resolution, with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) sGRBG10 format and output as IPU3_SGRBG10 format.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) The captured frames are available as /tmp/frame-#.bin files.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) ImgU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) ====
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) subdev interface to the user space.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) Each V4L2 subdev represents a pipe, which can support a maximum of 2 streams.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) This helps to support advanced camera features like Continuous View Finder (CVF)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) and Snapshot During Video(SDV).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) The ImgU contains two independent pipes, each modelled as a V4L2 sub-device
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) exposed to userspace as a V4L2 sub-device node.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) Each pipe has two sink pads and three source pads for the following purpose:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) .. flat-table::
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152) * - pad
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153) - direction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154) - purpose
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156) * - 0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157) - sink
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158) - Input raw video stream
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160) * - 1
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161) - sink
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162) - Processing parameters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) * - 2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) - source
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) - Output processed video stream
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) * - 3
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) - source
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) - Output viewfinder video stream
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) * - 4
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) - source
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) - 3A statistics
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) Each pad is connected to a corresponding V4L2 video interface, exposed to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) userspace as a V4L2 video device node.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) Device operation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) ----------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) With ImgU, once the input video node ("ipu3-imgu 0/1":0, in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) <entity>:<pad-number> format) is queued with buffer (in packed raw Bayer
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) format), ImgU starts processing the buffer and produces the video output in YUV
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) format and statistics output on respective output nodes. The driver is expected
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) to have buffers ready for all of parameter, output and statistics nodes, when
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) input video node is queued with buffer.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) At a minimum, all of input, main output, 3A statistics and viewfinder
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) video nodes should be enabled for IPU3 to start image processing.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) Each ImgU V4L2 subdev has the following set of video nodes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) input, output and viewfinder video nodes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) ----------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) The frames (in packed raw Bayer format specific to the IPU3) received by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) input video node is processed by the IPU3 Imaging Unit and are output to 2 video
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) nodes, with each targeting a different purpose (main output and viewfinder
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) output).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) Details onand the Bayer format specific to the IPU3 can be found in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) :ref:`v4l2-pix-fmt-ipu3-sbggr10`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) Only the multi-planar API is supported. More details can be found at
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) :ref:`planar-apis`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) Parameters video node
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) ---------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) The parameters video node receives the ImgU algorithm parameters that are used
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) to configure how the ImgU algorithms process the image.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) Details on processing parameters specific to the IPU3 can be found in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) :ref:`v4l2-meta-fmt-params`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) 3A statistics video node
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220) ------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) 3A statistics video node is used by the ImgU driver to output the 3A (auto
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) focus, auto exposure and auto white balance) statistics for the frames that are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224) being processed by the ImgU to user space applications. User space applications
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) can use this statistics data to compute the desired algorithm parameters for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226) the ImgU.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228) Configuring the Intel IPU3
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) ==========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231) The IPU3 ImgU pipelines can be configured using the Media Controller, defined at
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) :ref:`media_controller`.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234) Running mode and firmware binary selection
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) ------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237) ImgU works based on firmware, currently the ImgU firmware support run 2 pipes in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) time-sharing with single input frame data. Each pipe can run at certain mode -
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) "VIDEO" or "STILL", "VIDEO" mode is commonly used for video frames capture, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240) "STILL" is used for still frame capture. However, you can also select "VIDEO" to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241) capture still frames if you want to capture images with less system load and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) power. For "STILL" mode, ImgU will try to use smaller BDS factor and output
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) larger bayer frame for further YUV processing than "VIDEO" mode to get high
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) quality images. Besides, "STILL" mode need XNR3 to do noise reduction, hence
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245) "STILL" mode will need more power and memory bandwidth than "VIDEO" mode. TNR
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246) will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is running at
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) “VIDEO” mode by default, the user can use v4l2 control V4L2_CID_INTEL_IPU3_MODE
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) (currently defined in drivers/staging/media/ipu3/include/intel-ipu3.h) to query
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249) and set the running mode. For user, there is no difference for buffer queueing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) between the "VIDEO" and "STILL" mode, mandatory input and main output node
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) should be enabled and buffers need be queued, the statistics and the view-finder
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252) queues are optional.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) The firmware binary will be selected according to current running mode, such log
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) "using binary if_to_osys_striped " or "using binary if_to_osys_primary_striped"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256) could be observed if you enable the ImgU dynamic debug, the binary
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) if_to_osys_striped is selected for "VIDEO" and the binary
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258) "if_to_osys_primary_striped" is selected for "STILL".
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) Processing the image in raw Bayer format
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) ----------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) Configuring ImgU V4L2 subdev for image processing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) The ImgU V4L2 subdevs have to be configured with media controller APIs to have
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268) all the video nodes setup correctly.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) Let us take "ipu3-imgu 0" subdev as an example.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272) .. code-block:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274) media-ctl -d $MDEV -r
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275) media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276) media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277) media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278) media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280) Also the pipe mode of the corresponding V4L2 subdev should be set as desired
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281) (e.g 0 for video mode or 1 for still mode) through the control id 0x009819a1 as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282) below.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284) .. code-block:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286) yavta -w "0x009819A1 1" /dev/v4l-subdev7
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288) Certain hardware blocks in ImgU pipeline can change the frame resolution by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289) cropping or scaling, these hardware blocks include Input Feeder(IF), Bayer Down
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) Scaler (BDS) and Geometric Distortion Correction (GDC).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291) There is also a block which can change the frame resolution - YUV Scaler, it is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292) only applicable to the secondary output.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294) RAW Bayer frames go through these ImgU pipeline hardware blocks and the final
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295) processed image output to the DDR memory.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297) .. kernel-figure:: ipu3_rcb.svg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298) :alt: ipu3 resolution blocks image
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300) IPU3 resolution change hardware blocks
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302) **Input Feeder**
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) Input Feeder gets the Bayer frame data from the sensor, it can enable cropping
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305) of lines and columns from the frame and then store pixels into device's internal
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306) pixel buffer which are ready to readout by following blocks.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308) **Bayer Down Scaler**
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310) Bayer Down Scaler is capable of performing image scaling in Bayer domain, the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311) downscale factor can be configured from 1X to 1/4X in each axis with
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312) configuration steps of 0.03125 (1/32).
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314) **Geometric Distortion Correction**
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) Geometric Distortion Correction is used to performe correction of distortions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317) and image filtering. It needs some extra filter and envelop padding pixels to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318) work, so the input resolution of GDC should be larger than the output
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319) resolution.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321) **YUV Scaler**
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323) YUV Scaler which similar with BDS, but it is mainly do image down scaling in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324) YUV domain, it can support up to 1/12X down scaling, but it can not be applied
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325) to the main output.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327) The ImgU V4L2 subdev has to be configured with the supported resolutions in all
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328) the above hardware blocks, for a given input resolution.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329) For a given supported resolution for an input frame, the Input Feeder, Bayer
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330) Down Scaler and GDC blocks should be configured with the supported resolutions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331) as each hardware block has its own alignment requirement.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333) You must configure the output resolution of the hardware blocks smartly to meet
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334) the hardware requirement along with keeping the maximum field of view. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335) intermediate resolutions can be generated by specific tool -
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) https://github.com/intel/intel-ipu3-pipecfg
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339) This tool can be used to generate intermediate resolutions. More information can
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340) be obtained by looking at the following IPU3 ImgU configuration table.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342) https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344) Under baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345) directory, graph_settings_ov5670.xml can be used as an example.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347) The following steps prepare the ImgU pipeline for the image processing.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349) 1. The ImgU V4L2 subdev data format should be set by using the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352) 2. The ImgU V4L2 subdev cropping should be set by using the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353) VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354) using the input feeder height and width.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) 3. The ImgU V4L2 subdev composing should be set by using the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357) VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the target,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358) using the BDS height and width.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360) For the ov5670 example, for an input frame with a resolution of 2592x1944
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361) (which is input to the ImgU subdev pad 0), the corresponding resolutions
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362) for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363) respectively.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365) Once this is done, the received raw Bayer frames can be input to the ImgU
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366) V4L2 subdev as below, using the open source application v4l2n [#f1]_.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368) For an image captured with 2592x1944 [#f4]_ resolution, with desired output
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369) resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370) v4l2n command can be used. This helps process the raw Bayer frames and produces
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) the desired results for the main output image and the viewfinder output, in NV12
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372) format.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374) .. code-block:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376) v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377) --fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378) --reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379) --output=/tmp/frames.out --open=/dev/video5 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 380) --fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 381) --reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 382) --output=/tmp/frames.vf --open=/dev/video6 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 383) --fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 384) --reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 385) --output=/tmp/frames.3A --fmt=type:META_CAPTURE,? \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 386) --reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 387)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 388) You can also use yavta [#f2]_ command to do same thing as above:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 389)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 390) .. code-block:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 391)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 392) yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 393) --file=frame-#.out-f NV12 /dev/video5 & \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 394) yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 395) --file=frame-#.vf -f NV12 /dev/video6 & \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 396) yavta --data-prefix -Bmeta-capture -c10 -n5 -I \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 397) --file=frame-#.3a /dev/video7 & \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 398) yavta --data-prefix -Boutput-mplane -c10 -n5 -I -s2592x1944 \
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 399) --file=/tmp/frame-in.cio2 -f IPU3_SGRBG10 /dev/video4
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 400)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 401) where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 402) input, output, viewfinder and 3A statistics video nodes respectively.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 403)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 404) Converting the raw Bayer image into YUV domain
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 405) ----------------------------------------------
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 406)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 407) The processed images after the above step, can be converted to YUV domain
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 408) as below.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 409)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 410) Main output frames
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 411) ~~~~~~~~~~~~~~~~~~
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 412)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 413) .. code-block:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 414)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 415) raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 416)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 417) where 2560x1920 is output resolution, NV12 is the video format, followed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 418) by input frame and output PNM file.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 419)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 420) Viewfinder output frames
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 421) ~~~~~~~~~~~~~~~~~~~~~~~~
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 422)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 423) .. code-block:: none
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 424)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 425) raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 426)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 427) where 2560x1920 is output resolution, NV12 is the video format, followed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 428) by input frame and output PNM file.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 429)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 430) Example user space code for IPU3
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 431) ================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 432)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 433) User space code that configures and uses IPU3 is available here.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 434)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 435) https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 436)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 437) The source can be located under hal/intel directory.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 438)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 439) Overview of IPU3 pipeline
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 440) =========================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 441)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 442) IPU3 pipeline has a number of image processing stages, each of which takes a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 443) set of parameters as input. The major stages of pipelines are shown here:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 444)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 445) .. kernel-render:: DOT
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 446) :alt: IPU3 ImgU Pipeline
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 447) :caption: IPU3 ImgU Pipeline Diagram
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 448)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 449) digraph "IPU3 ImgU" {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 450) node [shape=box]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 451) splines="ortho"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 452) rankdir="LR"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 453)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 454) a [label="Raw pixels"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 455) b [label="Bayer Downscaling"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 456) c [label="Optical Black Correction"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 457) d [label="Linearization"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 458) e [label="Lens Shading Correction"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 459) f [label="White Balance / Exposure / Focus Apply"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 460) g [label="Bayer Noise Reduction"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 461) h [label="ANR"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 462) i [label="Demosaicing"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 463) j [label="Color Correction Matrix"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 464) k [label="Gamma correction"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 465) l [label="Color Space Conversion"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 466) m [label="Chroma Down Scaling"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 467) n [label="Chromatic Noise Reduction"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 468) o [label="Total Color Correction"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 469) p [label="XNR3"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 470) q [label="TNR"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 471) r [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 472) s [label="YUV Downscaling"]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 473) t [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 474)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 475) { rank=same; a -> b -> c -> d -> e -> f -> g -> h -> i }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 476) { rank=same; j -> k -> l -> m -> n -> o -> p -> q -> s -> t}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 477)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 478) a -> j [style=invis, weight=10]
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 479) i -> j
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 480) q -> r
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 481) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 482)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 483) The table below presents a description of the above algorithms.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 484)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 485) ======================== =======================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 486) Name Description
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 487) ======================== =======================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 488) Optical Black Correction Optical Black Correction block subtracts a pre-defined
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 489) value from the respective pixel values to obtain better
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 490) image quality.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 491) Defined in struct ipu3_uapi_obgrid_param.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 492) Linearization This algo block uses linearization parameters to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 493) address non-linearity sensor effects. The Lookup table
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 494) table is defined in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 495) struct ipu3_uapi_isp_lin_vmem_params.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 496) SHD Lens shading correction is used to correct spatial
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 497) non-uniformity of the pixel response due to optical
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 498) lens shading. This is done by applying a different gain
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 499) for each pixel. The gain, black level etc are
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 500) configured in struct ipu3_uapi_shd_config_static.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 501) BNR Bayer noise reduction block removes image noise by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 502) applying a bilateral filter.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 503) See struct ipu3_uapi_bnr_static_config for details.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 504) ANR Advanced Noise Reduction is a block based algorithm
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 505) that performs noise reduction in the Bayer domain. The
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 506) convolution matrix etc can be found in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 507) struct ipu3_uapi_anr_config.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 508) DM Demosaicing converts raw sensor data in Bayer format
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 509) into RGB (Red, Green, Blue) presentation. Then add
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 510) outputs of estimation of Y channel for following stream
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 511) processing by Firmware. The struct is defined as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 512) struct ipu3_uapi_dm_config.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 513) Color Correction Color Correction algo transforms sensor specific color
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 514) space to the standard "sRGB" color space. This is done
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 515) by applying 3x3 matrix defined in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 516) struct ipu3_uapi_ccm_mat_config.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 517) Gamma correction Gamma correction struct ipu3_uapi_gamma_config is a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 518) basic non-linear tone mapping correction that is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 519) applied per pixel for each pixel component.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 520) CSC Color space conversion transforms each pixel from the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 521) RGB primary presentation to YUV (Y: brightness,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 522) UV: Luminance) presentation. This is done by applying
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 523) a 3x3 matrix defined in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 524) struct ipu3_uapi_csc_mat_config
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 525) CDS Chroma down sampling
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 526) After the CSC is performed, the Chroma Down Sampling
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 527) is applied for a UV plane down sampling by a factor
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 528) of 2 in each direction for YUV 4:2:0 using a 4x2
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 529) configurable filter struct ipu3_uapi_cds_params.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 530) CHNR Chroma noise reduction
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 531) This block processes only the chrominance pixels and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 532) performs noise reduction by cleaning the high
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 533) frequency noise.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 534) See struct struct ipu3_uapi_yuvp1_chnr_config.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 535) TCC Total color correction as defined in struct
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 536) struct ipu3_uapi_yuvp2_tcc_static_config.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 537) XNR3 eXtreme Noise Reduction V3 is the third revision of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 538) noise reduction algorithm used to improve image
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 539) quality. This removes the low frequency noise in the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 540) captured image. Two related structs are being defined,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 541) struct ipu3_uapi_isp_xnr3_params for ISP data memory
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 542) and struct ipu3_uapi_isp_xnr3_vmem_params for vector
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 543) memory.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 544) TNR Temporal Noise Reduction block compares successive
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 545) frames in time to remove anomalies / noise in pixel
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 546) values. struct ipu3_uapi_isp_tnr3_vmem_params and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 547) struct ipu3_uapi_isp_tnr3_params are defined for ISP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 548) vector and data memory respectively.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 549) ======================== =======================================================
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 550)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 551) Other often encountered acronyms not listed in above table:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 552)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 553) ACC
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 554) Accelerator cluster
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 555) AWB_FR
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 556) Auto white balance filter response statistics
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 557) BDS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 558) Bayer downscaler parameters
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 559) CCM
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 560) Color correction matrix coefficients
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 561) IEFd
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 562) Image enhancement filter directed
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 563) Obgrid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 564) Optical black level compensation
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 565) OSYS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 566) Output system configuration
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 567) ROI
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 568) Region of interest
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 569) YDS
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 570) Y down sampling
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 571) YTM
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 572) Y-tone mapping
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 573)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 574) A few stages of the pipeline will be executed by firmware running on the ISP
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 575) processor, while many others will use a set of fixed hardware blocks also
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 576) called accelerator cluster (ACC) to crunch pixel data and produce statistics.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 577)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 578) ACC parameters of individual algorithms, as defined by
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 579) struct ipu3_uapi_acc_param, can be chosen to be applied by the user
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 580) space through struct struct ipu3_uapi_flags embedded in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 581) struct ipu3_uapi_params structure. For parameters that are configured as
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 582) not enabled by the user space, the corresponding structs are ignored by the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 583) driver, in which case the existing configuration of the algorithm will be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 584) preserved.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 585)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 586) References
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 587) ==========
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 588)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 589) .. [#f5] drivers/staging/media/ipu3/include/intel-ipu3.h
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 590)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 591) .. [#f1] https://github.com/intel/nvt
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 592)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 593) .. [#f2] http://git.ideasonboard.org/yavta.git
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 594)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 595) .. [#f3] http://git.ideasonboard.org/?p=media-ctl.git;a=summary
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 596)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 597) .. [#f4] ImgU limitation requires an additional 16x16 for all input resolutions
Orange Pi5 kernel
Deprecated Linux kernel 5.10.110 for OrangePi 5/5B/5+ boards
3 Commits
0 Branches
0 Tags