/dev/video[01]?

Oskar_Enoksson · March 29, 2024, 8:20am

I’m trying to make the Radxa 4K MIPI/CSI camera capture video on my Rock 5A, but have problems with using anything else than /dev/video11. Can someone give me a hint?

I’m running Armbian which I build with the new “vendor” kernel 6.1. I did add the 4K camera device tree overlay node successfully. I can capture video from /dev/video11 (the raw video device if I understand right) but trying with /dev/video[01] (mainpath/selfpath) always fails with “VIDIOC_STREAMON returned -1 (Operation not permitted)”. Question is: is there something I can do about this, or is it just that the hardware blocks used by mainline/selfpath not yet supported even in the “vendor” branch of 6.1? Has anyone been able to use these selfpath/mainpath streams on any rk3588 platform?

Big thanks! ‘’’
enok@rock-5a:~$ VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture Multiplanar:
Width/Height : 1920/1280
Pixel Format Field : None
Number of planes : 1
Flags :
Colorspace : Default
Transfer Function : Default
YCbCr/HSV Encoding: Default
Quantization : Default
Plane 0 :
Bytes per Line : 2560
Size Image : 2764800
VIDIOC_REQBUFS returned 0 (Success)
VIDIOC_QUERYBUF VIDIOC_QUERYBUF VIDIOC_QUERYBUF VIDIOC_QUERYBUF VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_STREAMON cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: cap dqbuf: 0 seq: cap dqbuf: 1 seq: cap dqbuf: 2 seq: cap dqbuf: 3 seq: Output from v4l2-ctl video capture commands follow:
sudo v4l2-ctl --verbose -d11 --set-fmt-video=width=1920,height=1280,pixelformat=‘RG10’ --stream-to=out.rgbp --stream-mmap=4 --stream-count=60
: ‘RG10’ (10-bit Bayer RGRG/GBGB)
returned 0 (Success)
returned 0 (Success)
returned 0 (Success)
returned 0 (Success)
returned 0 (Success)
0 bytesused: 2764800 ts: 424222.531128 (ts-monotonic, ts-src-eof)
1 bytesused: 2764800 ts: 424222.564458 delta: 33.330 ms (ts-monotonic, ts-src-eof)
2 bytesused: 2764800 ts: 424222.597792 delta: 33.334 ms (ts-monotonic, ts-src-eof)
3 bytesused: 2764800 ts: 424222.631126 delta: 33.334 ms (ts-monotonic, ts-src-eof)
4 bytesused: 2764800 ts: 424222.664459 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
5 bytesused: 2764800 ts: 424222.697793 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
6 bytesused: 2764800 ts: 424222.731126 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
7 bytesused: 2764800 ts: 424222.764461 delta: 33.335 ms fps: 30.00 (ts-monotonic, ts-src-eof)
8 bytesused: 2764800 ts: 424222.797795 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
9 bytesused: 2764800 ts: 424222.831128 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
10 bytesused: 2764800 ts: 424222.864462 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
11 bytesused: 2764800 ts: 424222.897795 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
12 bytesused: 2764800 ts: 424222.931129 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
13 bytesused: 2764800 ts: 424222.964462 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
14 bytesused: 2764800 ts: 424222.997796 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
15 bytesused: 2764800 ts: 424223.031130 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
16 bytesused: 2764800 ts: 424223.064463 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
17 bytesused: 2764800 ts: 424223.097797 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
18 bytesused: 2764800 ts: 424223.131131 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
19 bytesused: 2764800 ts: 424223.164464 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
20 bytesused: 2764800 ts: 424223.197798 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
21 bytesused: 2764800 ts: 424223.231131 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
22 bytesused: 2764800 ts: 424223.264466 delta: 33.335 ms fps: 30.00 (ts-monotonic, ts-src-eof)
23 bytesused: 2764800 ts: 424223.297799 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
24 bytesused: 2764800 ts: 424223.331132 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
25 bytesused: 2764800 ts: 424223.364466 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
26 bytesused: 2764800 ts: 424223.397800 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
27 bytesused: 2764800 ts: 424223.431133 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
28 bytesused: 2764800 ts: 424223.464467 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
29 bytesused: 2764800 ts: 424223.497800 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
30 bytesused: 2764800 ts: 424223.531135 delta: 33.335 ms fps: 30.00 (ts-monotonic, ts-src-eof)
31 bytesused: 2764800 ts: 424223.564468 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
32 bytesused: 2764800 ts: 424223.597802 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
33 bytesused: 2764800 ts: 424223.631135 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
34 bytesused: 2764800 ts: 424223.664469 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
35 bytesused: 2764800 ts: 424223.697802 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
36 bytesused: 2764800 ts: 424223.731136 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
37 bytesused: 2764800 ts: 424223.764470 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
38 bytesused: 2764800 ts: 424223.797803 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
39 bytesused: 2764800 ts: 424223.831137 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
40 bytesused: 2764800 ts: 424223.864471 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
41 bytesused: 2764800 ts: 424223.897804 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
42 bytesused: 2764800 ts: 424223.931138 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
43 bytesused: 2764800 ts: 424223.964472 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
44 bytesused: 2764800 ts: 424223.997805 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
45 bytesused: 2764800 ts: 424224.031138 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
46 bytesused: 2764800 ts: 424224.064472 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
47 bytesused: 2764800 ts: 424224.097805 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
48 bytesused: 2764800 ts: 424224.131139 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
49 bytesused: 2764800 ts: 424224.164474 delta: 33.335 ms fps: 30.00 (ts-monotonic, ts-src-eof)
50 bytesused: 2764800 ts: 424224.197806 delta: 33.332 ms fps: 30.00 (ts-monotonic, ts-src-eof)
51 bytesused: 2764800 ts: 424224.231141 delta: 33.335 ms fps: 30.00 (ts-monotonic, ts-src-eof)
52 bytesused: 2764800 ts: 424224.264474 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
53 bytesused: 2764800 ts: 424224.297808 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
54 bytesused: 2764800 ts: 424224.331142 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
55 bytesused: 2764800 ts: 424224.364476 delta: 33.334 ms fps: 30.00 (ts-monotonic, ts-src-eof)
56 bytesused: 2764800 ts: 424224.397883 delta: 33.407 ms fps: 30.00 (ts-monotonic, ts-src-eof)
57 bytesused: 2764800 ts: 424224.431215 delta: 33.332 ms fps: 30.00 (ts-monotonic, ts-src-eof)
58 bytesused: 2764800 ts: 424224.464466 delta: 33.251 ms fps: 30.00 (ts-monotonic, ts-src-eof)
59 bytesused: 2764800 ts: 424224.497881 delta: 33.415 ms fps: 30.00 (ts-monotonic, ts-src-eof)

enok@rock-5a:~$ sudo v4l2-ctl --verbose -d1 --set-fmt-video=width=1920,height=1280,pixelformat=‘RGBP’ --stream-to=out.rgbp --stream-mmap=4 --stream-count=60
VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture Multiplanar:
Width/Height : 1920/1280
Pixel Format : ‘RGBP’ (16-bit RGB 5-6-5)
Field : None
Number of planes : 1
Flags :
Colorspace : Default
Transfer Function : Default
YCbCr/HSV Encoding: Default
Quantization : Full Range
Plane 0 :
Bytes per Line : 3840
Size Image : 4915200
VIDIOC_REQBUFS returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_STREAMON returned -1 (Operation not permitted)
enok@rock-5a:
‘’’

Oskar_Enoksson · March 31, 2024, 1:17pm

I do get frequent system hangups when experimenting with different parameters using v4l2-ctl and media-ctl. Is it just my v4l skills that are lacking, or is the video capture and ISP functionality in the vendor kernel 6.1 not fully functional/stable? Is there any other recommended kernel version?

Message from syslogd@rock-5a at Mar 31 06:02:35 …
kernel:[ 193.089358] Internal error: synchronous external abort: 0000000096000010 [#1] SMP

Maybe I just need to study the documentation more. But where can I find updated documents on rkisp, rkcif and how to use media-ctl or v4l2-ctl, preferably not in Chinese? I only find old documents referring to kernel 4.4 and previous chip families.

I also tried asking in forums, discord etc. Is the lack of answers perhaps an indication that very few people use the MIPI/CSI camera interface for RK3588? Or that implementation is not ready enough to be useful? Please let me know in that case, and save me from spending my time on a path that is a dead end. Thanks.

Oskar_Enoksson · March 31, 2024, 9:37pm

This image from the rk3588 TRM seems to indicate that the captured video stream could be sent to ISP0/1. Further down this “toisp” is very briefly mentioned:

So there are two ISPs? I find no further information in the TRM about ISP0 or ISP1. There are sections on FEC (FishEye Correction) and IE (Image Enhancement). Are those blocks part of ISP0/1? I can’t make sense from the TRM here. If anyone has a clue please help. Here is the graph generated by the command ‘media-ctl -d0 --print-dot’ on my Rock 5a with an Radxa 4K camera connected:

And here is the graph ‘media-ctl -d1 --print-dot’

How does this correspond to the block diagram above (in the TRM)? How am i supposed to use all these /dev/video* blocks? Is there any documentation?

avaf · March 31, 2024, 9:38pm

Parsed data go to ISP0/1 for real-time processing, that is video11 and video31.

rock@rock5b:~$ v4l2-ctl --verbose -d11 --all
VIDIOC_QUERYCAP: ok
Driver Info:
	Driver name      : rkisp_v6
	Card type        : rkisp_mainpath
	Bus info         : platform:rkisp0-vir0
	Driver version   : 2.0.0
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkisp0-vir0
	Model            : rkisp0
	Serial           : 
	Bus info         : 
	Media version    : 5.10.110
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.110
Interface Info:
	ID               : 0x03000007
	Type             : V4L Video
Entity Info:
	ID               : 0x00000006 (6)
	Name             : rkisp_mainpath
	Function         : V4L2 I/O
	Pad 0x01000009   : 0: Sink
	  Link 0x0200000a: from remote pad 0x1000004 of entity 'rkisp-isp-subdev' (Unknown V4L2 Sub-Device): Data, Enabled
Priority: 2
Format Video Capture Multiplanar:
	Width/Height      : 1920/1080
	Pixel Format      : 'NM12' (Y/CbCr 4:2:0 (N-C))
	Field             : None
	Number of planes  : 2
	Flags             : 
	Colorspace        : sRGB
	Transfer Function : Rec. 709
	YCbCr/HSV Encoding: Rec. 709
	Quantization      : Full Range
	Plane 0           :
	   Bytes per Line : 1920
	   Size Image     : 2073600
	Plane 1           :
	   Bytes per Line : 1920
	   Size Image     : 1036800
Selection Video Capture: crop, Left 0, Top 0, Width 3840, Height 2160, Flags: 
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 3840, Height 2160, Flags: 
Selection Video Output: crop, Left 0, Top 0, Width 3840, Height 2160, Flags: 
Selection Video Output: crop_bounds, Left 0, Top 0, Width 3840, Height 2160, Flags: 

Image Processing Controls

                     pixel_rate 0x009f0902 (int64)  : min=0 max=1000000000 step=1 default=1000000000 value=356800000 flags=read-only, volatile
rock@rock5b:~$

Goetterfunke · March 31, 2024, 9:38pm

Hi There, does this help in any way? These are for usb-dp etc. Camera is Video11 after enabling dts overlay.

ock@rock5b:~$ v4l2-ctl --list-devices

rk_hdmirx (fdee0000.hdmirx-controller):
/dev/video20
rkisp-statistics (platform: rkisp):
/dev/video18
/dev/video19
rkcif-mipi-lvds2 (platform:rkcif):
/dev/media0
rkcif (platform:rkcif-mipi-lvds2):
/dev/video0
/dev/video1
/dev/video2
/dev/video3
/dev/video4
/dev/video5
/dev/video6
/dev/video7
/dev/video8
/dev/video9
/dev/video10

rkisp_mainpath (platform:rkisp0-vir0):
/dev/video11
/dev/video12
/dev/video13
/dev/video14
/dev/video15
/dev/video16
/dev/video17
/dev/media1

USB 2.0 Camera: USB Camera (usb-fc880000.usb-1):
/dev/video21
/dev/video22
/dev/video23
/dev/video24
/dev/media2

Oskar_Enoksson · March 31, 2024, 9:50pm

@avaf: On my system it looks differently. The “rkisp_mainpath” device is /dev/video0, not /dev/video11 as in your printout. My /dev/video11 seems to be related to another “rkcif” driver, not “rkisp”. And there is no /dev/video31. Here are my devices:

enok@rock-5a:~$ v4l2-ctl --list-devices
rkisp-statistics (platform: rkisp):
	/dev/video7
	/dev/video8

rkcif (platform:rkcif-mipi-lvds2):
	/dev/video9
	/dev/video10
	/dev/video11
	/dev/video12
	/dev/video13
	/dev/video14
	/dev/video15
	/dev/video16
	/dev/video17
	/dev/video18
	/dev/video19
	/dev/media1

rkisp_mainpath (platform:rkisp0-vir0):
	/dev/video0
	/dev/video1
	/dev/video2
	/dev/video3
	/dev/video4
	/dev/video5
	/dev/video6
	/dev/media0

And /dev/video11 detailes:

enok@rock-5a:~$ v4l2-ctl -d11 --all
Driver Info:
	Driver name      : rkcif
	Card type        : rkcif
	Bus info         : platform:rkcif-mipi-lvds2
	Driver version   : 6.1.43
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkcif
	Model            : rkcif-mipi-lvds2
	Serial           : 
	Bus info         : platform:rkcif-mipi-lvds2
	Media version    : 6.1.43
	Hardware revision: 0x00000000 (0)
	Driver version   : 6.1.43
Interface Info:
	ID               : 0x0300000a
	Type             : V4L Video
Entity Info:
	ID               : 0x00000009 (9)
	Name             : stream_cif_mipi_id2
	Function         : V4L2 I/O
	Pad 0x0100000c   : 0: Sink
	  Link 0x02000047: from remote pad 0x100002f of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
	  Link 0x0200005d: from remote pad 0x1000030 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
	  Link 0x02000073: from remote pad 0x1000031 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data, Enabled
	  Link 0x02000089: from remote pad 0x1000032 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
	  Link 0x0200009f: from remote pad 0x1000033 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
	  Link 0x020000b5: from remote pad 0x1000034 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
	  Link 0x020000cb: from remote pad 0x1000035 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
	  Link 0x020000e1: from remote pad 0x1000036 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
	  Link 0x020000f7: from remote pad 0x1000037 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
	  Link 0x0200010d: from remote pad 0x1000038 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
	  Link 0x02000123: from remote pad 0x1000039 of entity 'rockchip-mipi-csi2' (Unknown V4L2 Sub-Device): Data
Priority: 2
Format Video Capture Multiplanar:
	Width/Height      : 1920/1080
	Pixel Format      : 'RG10' (10-bit Bayer RGRG/GBGB)
	Field             : None
	Number of planes  : 1
	Flags             : 
	Colorspace        : Default
	Transfer Function : Default
	YCbCr/HSV Encoding: Default
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 4864
	   Size Image     : 10506240
Selection Video Capture: crop, Left 12, Top 16, Width 3840, Height 2160, Flags: 
Selection Video Capture: crop_bounds, Left 12, Top 16, Width 3840, Height 2160, Flags: 
Selection Video Output: crop, Left 12, Top 16, Width 3840, Height 2160, Flags: 
Selection Video Output: crop_bounds, Left 12, Top 16, Width 3840, Height 2160, Flags:

avaf · March 31, 2024, 9:56pm

Rock 5B has hdmi input (video0), Rock 5A does not (?), i think.

Edit: actually is video20, same as @Goetterfunke

rock@rock5b:~$ v4l2-ctl --list-devices

rk_hdmirx (fdee0000.hdmirx-controller):
/dev/video20

rkisp-statistics (platform: rkisp):
/dev/video18
/dev/video19

rkcif-mipi-lvds2 (platform:rkcif):
/dev/media0

rkcif (platform:rkcif-mipi-lvds2):
/dev/video0
/dev/video1
/dev/video2
/dev/video3
/dev/video4
/dev/video5
/dev/video6
/dev/video7
/dev/video8
/dev/video9
/dev/video10

rkisp_mainpath (platform:rkisp0-vir0):
/dev/video11
/dev/video12
/dev/video13
/dev/video14
/dev/video15
/dev/video16
/dev/video17
/dev/media1

Dummy video device (0x0000) (platform:v4l2loopback-000):
/dev/video21

Oskar_Enoksson · March 31, 2024, 10:01pm

That’s right. No HDMI Rx on 5A. My camera is on MIPI/CSI input.

So is the /dev/video11 in my system (name=“stream_cif_mipi_id2”) actually my CSI stream after passing through the ISP?

Goetterfunke · March 31, 2024, 9:59pm

Hi there did you enable dts? rsetup -> Overlays?

avaf · March 31, 2024, 10:00pm

Looks like your kernel 6.1 is borked

Goetterfunke · March 31, 2024, 10:01pm

Ohh, hes on armbian. It’s working with official image for me

Oskar_Enoksson · March 31, 2024, 10:05pm

It’s armbian “vendor branch” kernel 6.1. I did add overlay for the Radxa 4k IM415 MIPI/CSI camera. But perhaps some more overlays are needed?

BTW THANKS EVERYBODY FOR HELPING!!!

Allright @goetterfunke I’ll try the official image tomorrow and compare the results.

Goetterfunke · March 31, 2024, 10:08pm

Yes, I think you also need some parts in the kernel? AFAIK dts is working together with kernel somehow. Perfect! Just reading your edit: Good luck!

Edit: did you check dmesg for errors registering devices or something? I don’t know if its worth debugging but maybe you can find something related to your custom dts there…

Oskar_Enoksson · March 31, 2024, 10:17pm

I have indeed lots of errors in dmesg, will look into that. I also saw in some doc that debug verbosity can be increased by writing to differen /sys/module/video_*/parameters/debug and /sys/class/video4linux/video*/dev_debug files. Also something to look deeper into.

Thanks once again! Will post here if I figure it out.

Oskar_Enoksson · April 4, 2024, 8:43am

I did experiment with the “official image” (b18) for Rock 5A. I did software update after boot, enabled the 4K camera overlay and rebooted.

The devices /dev/video* and /dev/media* are the same as I got with Armbian vendor kernel (6.1), just the numbering differs, i.e. they are permuted. There is however one new /dev/v4l-subdev4 in b18 whereas 6.1 only had /dev/v4l-subdev[0123] .

I can capture video frames from the MIPI/CSI connected 4K camera from “main-path” and “self-path” devices with b18, which I couldn’t with 6.1. The graph generated from the media-ctl -d0 --print-dot command reveals that there is an additional path from /dev/v4l-subdev4 to the rkisp block.

/home/enok/a.official.dot.png

So why is this connection path from rkcif to rkisp not available under my 6.1 vendor kernel? Some missing kernel config or a bug in some newer version of the code? I looked through the kernel config and found nothing obvious. Can anyone guess?

configs.zip (102.4 KB)

The following diagram from the outdated RKISP driver user manual shows the video stream paths as they are supposed to be pretty clearly, although I don’t know if it applies to the rk3588

Oskar_Enoksson · April 6, 2024, 10:42pm

If anyone is following this thread perhaps also struggling with MIPI/ISP video capture: I think the 6.1 vendor kernel in Armbian is simply broken when it comes to MIPI camera capture, ISP or something related, at least for Rock 5a.

I found another interesting source in English on how video capture and the associated data streams are supposed to work for the RK3588 here. The mainline kernel docs also have some info, although it looks less up to date.

So no solution yet.

avaf · April 6, 2024, 10:50pm

Have you double-checked the DTS?

Oskar_Enoksson · April 6, 2024, 11:03pm

Not in detail no. I’ll try to decide if I want to try to understand the details and attempt to debug the 6.1 drivers and/or dts, or if I should perhaps do something else and just hope for someone more knowledgeable to eventually implement rk3588(s) ISP and video capture support in mainline instead.

avaf · April 6, 2024, 11:22pm

It can be something missing in the camera overlay. I don’t use overlay for this reason.
I think one way to find more info about this is to decode the running DT and compare the fragments to 5.10.

Here is the correct fragment ROCK 5B:

// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
/*
 * Copyright (c) 2022 Rockchip Electronics Co., Ltd.
 *
 */

/ {
	compatible = "radxa,rock-5b", "rockchip,rk3588";
};

&i2c3 {
	status = "okay";

	imx415: imx415@1a {
		status = "okay";
		compatible = "sony,imx415";
		reg = <0x1a>;
		clocks = <&cru CLK_MIPI_CAMARAOUT_M3>;
		clock-names = "xvclk";
		pinctrl-names = "default";
		pinctrl-0 = <&mipim0_camera3_clk>;
		power-domains = <&power RK3588_PD_VI>;
		pwdn-gpios = <&gpio1 RK_PB0 GPIO_ACTIVE_HIGH>;
		reset-gpios = <&gpio4 RK_PA0 GPIO_ACTIVE_LOW>;
		rockchip,camera-module-index = <0>;
		rockchip,camera-module-facing = "back";
		rockchip,camera-module-name = "CMK-OT2022-PX1";
		rockchip,camera-module-lens-name = "IR0147-50IRC-8M-F20";
		rockchip,camera-hdr-mode = <0>;
		rockchip,camera-vflip = <0>;
		rockchip,camera-hflip = <1>;
		port {
			imx415_out0: endpoint {
				remote-endpoint = <&mipidphy0_in_ucam0>;
				data-lanes = <1 2 3 4>;
			};
		};
	};
};

&csi2_dphy0_hw {
	status = "okay";
};

&csi2_dphy0 {
	status = "okay";

	ports {
		#address-cells = <1>;
		#size-cells = <0>;

		port@0 {
			reg = <0>;
			#address-cells = <1>;
			#size-cells = <0>;

			mipidphy0_in_ucam0: endpoint@1 {
				reg = <1>;
				remote-endpoint = <&imx415_out0>;
				data-lanes = <1 2 3 4>;
			};
		};

		port@1 {
			reg = <1>;
			#address-cells = <1>;
			#size-cells = <0>;

			csidphy0_out: endpoint@0 {
				reg = <0>;
				remote-endpoint = <&mipi2_csi2_input>;
			};
		};
	};
};

&mipi2_csi2 {
	status = "okay";

	ports {
		#address-cells = <1>;
		#size-cells = <0>;

		port@0 {
			reg = <0>;
			#address-cells = <1>;
			#size-cells = <0>;

			mipi2_csi2_input: endpoint@1 {
				reg = <1>;
				remote-endpoint = <&csidphy0_out>;
			};
		};

		port@1 {
			reg = <1>;
			#address-cells = <1>;
			#size-cells = <0>;

			mipi2_csi2_output: endpoint@0 {
				reg = <0>;
				remote-endpoint = <&cif_mipi2_in0>;
			};
		};
	};
};

&rkcif {
	status = "okay";
};

&rkcif_mipi_lvds2 {
	status = "okay";

	port {
		cif_mipi2_in0: endpoint {
			remote-endpoint = <&mipi2_csi2_output>;
		};
	};
};

&rkcif_mipi_lvds2_sditf {
	status = "okay";

	port {
		mipi_lvds2_sditf: endpoint {
			remote-endpoint = <&isp0_vir0>;
		};
	};
};

&rkcif_mmu {
	status = "okay";
};

&rkisp0 {
	status = "okay";
};

&isp0_mmu {
	status = "okay";
};

&rkisp0_vir0 {
	status = "okay";

	port {
		#address-cells = <1>;
		#size-cells = <0>;

		isp0_vir0: endpoint@0 {
			reg = <0>;
			remote-endpoint = <&mipi_lvds2_sditf>;
		};
	};
};

Check for this link ??:

&rkcif_mipi_lvds2 {
	status = "okay";

	port {
		cif_mipi2_in0: endpoint {
			remote-endpoint = <&mipi2_csi2_output>;
		};
	};
};

Decode the running DT that works on you 5.10 and compare to 6.1:

dtc -I fs -O dts /sys/firmware/devicetree/base > running_5.10.dts
dtc -I fs -O dts /sys/firmware/devicetree/base > running_6.1.dts

Oskar_Enoksson · April 8, 2024, 9:09pm

The camera overlay looks ok to me. The resulting 5.10 and 6.1 camera nodes look virtually identical. See below for zipped dts.

Maybe it’s in the rk3588s.dtsi file some error has been introduced? The rk3588s.dtsi is included from rkk3588.dtsi. I guess the idea is that everything in rk3588s is also in rk3588. But maybe that doesn’t hold true for the MIPI/CSI devices?

According to the datasheet rk3588 supports 2 dcphy and up to 4 dphy CSI inputs with total 8 lanes, while the rk3588s supports 2 dcphy and up to 3 dphy CSI inputs with total 4 lanes.

Here is dmesg output showing the failure to set up rkcif-mipi-lvds2.

[  +5.496351] rkcifhw fdce0000.rkcif: Adding to iommu group 16
[  +0.002283] rkcifhw fdce0000.rkcif: No reserved memory region assign to CIF
[  +0.000848] rkcif rkcif-mipi-lvds2: Adding to iommu group 16
[  +0.000095] rkcif rkcif-mipi-lvds2: rkcif driver version: v00.02.00
[  +0.000394] rkcif rkcif-mipi-lvds2: attach to cif hw node
[  +0.000033] rkcif rkcif-mipi-lvds2: rkcif wait line 0
[  +0.000032] : terminal subdev does not exist
[  +0.000023] : terminal subdev does not exist
[  +0.000020] : terminal subdev does not exist
[  +0.000020] : terminal subdev does not exist
[  +0.000019] : get_remote_sensor: video pad[0] is null
[  +0.000018] : rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[  +0.000025] : rkcif_scale_set_fmt: req(80, 60) src out(0, 0)
[  +0.000029] : get_remote_sensor: video pad[0] is null
[  +0.000017] : rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[  +0.000020] : rkcif_scale_set_fmt: req(80, 60) src out(0, 0)
[  +0.000025] : get_remote_sensor: video pad[0] is null
[  +0.000017] : rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[  +0.000020] : rkcif_scale_set_fmt: req(80, 60) src out(0, 0)
[  +0.000025] : get_remote_sensor: video pad[0] is null
[  +0.000017] : rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[  +0.000021] : rkcif_scale_set_fmt: req(80, 60) src out(0, 0)
[  +0.007855] rkcif-mipi-lvds2: get_remote_sensor: remote pad is null
[  +0.000022] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[  +0.000014] stream_cif_mipi_id0: update sensor info failed -19
[  +0.001055] rkcif-mipi-lvds2: get_remote_sensor: remote pad is null
[  +0.000022] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[1] get remote sensor_sd failed!
[  +0.000013] stream_cif_mipi_id1: update sensor info failed -19
[  +0.000128] rkcif-mipi-lvds2: get_remote_sensor: remote pad is null
[  +0.000021] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[2] get remote sensor_sd failed!
[  +0.000014] stream_cif_mipi_id2: update sensor info failed -19
[  +0.002936] rkcif-mipi-lvds2: get_remote_sensor: remote pad is null
[  +0.000020] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[  +0.000012] rkcif_scale_ch0: update sensor info failed -19
[  +0.000478] rkcif rkcif-mipi-lvds2: Entity type for entity rockchip-mipi-csi2 was not initialized!
[  +0.000030] rockchip-mipi-csi2: Async registered subdev
[  +0.000025] imx415 3-001a: Consider updating driver imx415 to match on endpoints
[  +0.000017] rockchip-csi2-dphy csi2-dphy0: dphy0 matches m00_b_imx415 3-001a:bus type 5
[  +0.001187] stream_cif_mipi_id3: Failed to get runtime pm, -13
[  +0.001098] rkcif-mipi-lvds2: Async subdev notifier completed
[  +0.000029] rkcif rkcif-mipi-lvds2: No memory-region-thunderboot specified

The 5.10 official image b18 and the 6.1 armbian vendor dts are here (both with 4k camera overlays added):
fdt.dts.zip (77.0 KB)

In any case, it seems that the problem is that the device /dev/v4l-subdev4 which would have corresponded to rkcif-mipi-lvds2 is never created. Still don’t know why.

Here is newer version 2.0 of that rockchip camera developer guide document BTW. (Why is documentation found in so many different places?)