If anyone is following this thread perhaps also struggling with MIPI/ISP video capture: I think the 6.1 vendor kernel in Armbian is simply broken when it comes to MIPI camera capture, ISP or something related, at least for Rock 5a.
I found another interesting source in English on how video capture and the associated data streams are supposed to work for the RK3588 here. The mainline kernel docs also have some info, although it looks less up to date.
So no solution yet.
Have you double-checked the DTS?
Not in detail no. I’ll try to decide if I want to try to understand the details and attempt to debug the 6.1 drivers and/or dts, or if I should perhaps do something else and just hope for someone more knowledgeable to eventually implement rk3588(s) ISP and video capture support in mainline instead.
It can be something missing in the camera overlay. I don’t use overlay for this reason.
I think one way to find more info about this is to decode the running DT and compare the fragments to 5.10.
Here is the correct fragment ROCK 5B:
// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
/*
* Copyright (c) 2022 Rockchip Electronics Co., Ltd.
*
*/
/ {
compatible = "radxa,rock-5b", "rockchip,rk3588";
};
&i2c3 {
status = "okay";
imx415: imx415@1a {
status = "okay";
compatible = "sony,imx415";
reg = <0x1a>;
clocks = <&cru CLK_MIPI_CAMARAOUT_M3>;
clock-names = "xvclk";
pinctrl-names = "default";
pinctrl-0 = <&mipim0_camera3_clk>;
power-domains = <&power RK3588_PD_VI>;
pwdn-gpios = <&gpio1 RK_PB0 GPIO_ACTIVE_HIGH>;
reset-gpios = <&gpio4 RK_PA0 GPIO_ACTIVE_LOW>;
rockchip,camera-module-index = <0>;
rockchip,camera-module-facing = "back";
rockchip,camera-module-name = "CMK-OT2022-PX1";
rockchip,camera-module-lens-name = "IR0147-50IRC-8M-F20";
rockchip,camera-hdr-mode = <0>;
rockchip,camera-vflip = <0>;
rockchip,camera-hflip = <1>;
port {
imx415_out0: endpoint {
remote-endpoint = <&mipidphy0_in_ucam0>;
data-lanes = <1 2 3 4>;
};
};
};
};
&csi2_dphy0_hw {
status = "okay";
};
&csi2_dphy0 {
status = "okay";
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
#address-cells = <1>;
#size-cells = <0>;
mipidphy0_in_ucam0: endpoint@1 {
reg = <1>;
remote-endpoint = <&imx415_out0>;
data-lanes = <1 2 3 4>;
};
};
port@1 {
reg = <1>;
#address-cells = <1>;
#size-cells = <0>;
csidphy0_out: endpoint@0 {
reg = <0>;
remote-endpoint = <&mipi2_csi2_input>;
};
};
};
};
&mipi2_csi2 {
status = "okay";
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
#address-cells = <1>;
#size-cells = <0>;
mipi2_csi2_input: endpoint@1 {
reg = <1>;
remote-endpoint = <&csidphy0_out>;
};
};
port@1 {
reg = <1>;
#address-cells = <1>;
#size-cells = <0>;
mipi2_csi2_output: endpoint@0 {
reg = <0>;
remote-endpoint = <&cif_mipi2_in0>;
};
};
};
};
&rkcif {
status = "okay";
};
&rkcif_mipi_lvds2 {
status = "okay";
port {
cif_mipi2_in0: endpoint {
remote-endpoint = <&mipi2_csi2_output>;
};
};
};
&rkcif_mipi_lvds2_sditf {
status = "okay";
port {
mipi_lvds2_sditf: endpoint {
remote-endpoint = <&isp0_vir0>;
};
};
};
&rkcif_mmu {
status = "okay";
};
&rkisp0 {
status = "okay";
};
&isp0_mmu {
status = "okay";
};
&rkisp0_vir0 {
status = "okay";
port {
#address-cells = <1>;
#size-cells = <0>;
isp0_vir0: endpoint@0 {
reg = <0>;
remote-endpoint = <&mipi_lvds2_sditf>;
};
};
};
Check for this link ??:
&rkcif_mipi_lvds2 {
status = "okay";
port {
cif_mipi2_in0: endpoint {
remote-endpoint = <&mipi2_csi2_output>;
};
};
};
Decode the running DT that works on you 5.10 and compare to 6.1:
dtc -I fs -O dts /sys/firmware/devicetree/base > running_5.10.dts
dtc -I fs -O dts /sys/firmware/devicetree/base > running_6.1.dts
The camera overlay looks ok to me. The resulting 5.10 and 6.1 camera nodes look virtually identical. See below for zipped dts.
Maybe it’s in the rk3588s.dtsi file some error has been introduced? The rk3588s.dtsi is included from rkk3588.dtsi. I guess the idea is that everything in rk3588s is also in rk3588. But maybe that doesn’t hold true for the MIPI/CSI devices?
According to the datasheet rk3588 supports 2 dcphy and up to 4 dphy CSI inputs with total 8 lanes, while the rk3588s supports 2 dcphy and up to 3 dphy CSI inputs with total 4 lanes.
Here is dmesg output showing the failure to set up rkcif-mipi-lvds2.
[ +5.496351] rkcifhw fdce0000.rkcif: Adding to iommu group 16
[ +0.002283] rkcifhw fdce0000.rkcif: No reserved memory region assign to CIF
[ +0.000848] rkcif rkcif-mipi-lvds2: Adding to iommu group 16
[ +0.000095] rkcif rkcif-mipi-lvds2: rkcif driver version: v00.02.00
[ +0.000394] rkcif rkcif-mipi-lvds2: attach to cif hw node
[ +0.000033] rkcif rkcif-mipi-lvds2: rkcif wait line 0
[ +0.000032] : terminal subdev does not exist
[ +0.000023] : terminal subdev does not exist
[ +0.000020] : terminal subdev does not exist
[ +0.000020] : terminal subdev does not exist
[ +0.000019] : get_remote_sensor: video pad[0] is null
[ +0.000018] : rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[ +0.000025] : rkcif_scale_set_fmt: req(80, 60) src out(0, 0)
[ +0.000029] : get_remote_sensor: video pad[0] is null
[ +0.000017] : rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[ +0.000020] : rkcif_scale_set_fmt: req(80, 60) src out(0, 0)
[ +0.000025] : get_remote_sensor: video pad[0] is null
[ +0.000017] : rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[ +0.000020] : rkcif_scale_set_fmt: req(80, 60) src out(0, 0)
[ +0.000025] : get_remote_sensor: video pad[0] is null
[ +0.000017] : rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[ +0.000021] : rkcif_scale_set_fmt: req(80, 60) src out(0, 0)
[ +0.007855] rkcif-mipi-lvds2: get_remote_sensor: remote pad is null
[ +0.000022] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[ +0.000014] stream_cif_mipi_id0: update sensor info failed -19
[ +0.001055] rkcif-mipi-lvds2: get_remote_sensor: remote pad is null
[ +0.000022] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[1] get remote sensor_sd failed!
[ +0.000013] stream_cif_mipi_id1: update sensor info failed -19
[ +0.000128] rkcif-mipi-lvds2: get_remote_sensor: remote pad is null
[ +0.000021] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[2] get remote sensor_sd failed!
[ +0.000014] stream_cif_mipi_id2: update sensor info failed -19
[ +0.002936] rkcif-mipi-lvds2: get_remote_sensor: remote pad is null
[ +0.000020] rkcif-mipi-lvds2: rkcif_update_sensor_info: stream[0] get remote sensor_sd failed!
[ +0.000012] rkcif_scale_ch0: update sensor info failed -19
[ +0.000478] rkcif rkcif-mipi-lvds2: Entity type for entity rockchip-mipi-csi2 was not initialized!
[ +0.000030] rockchip-mipi-csi2: Async registered subdev
[ +0.000025] imx415 3-001a: Consider updating driver imx415 to match on endpoints
[ +0.000017] rockchip-csi2-dphy csi2-dphy0: dphy0 matches m00_b_imx415 3-001a:bus type 5
[ +0.001187] stream_cif_mipi_id3: Failed to get runtime pm, -13
[ +0.001098] rkcif-mipi-lvds2: Async subdev notifier completed
[ +0.000029] rkcif rkcif-mipi-lvds2: No memory-region-thunderboot specified
The 5.10 official image b18 and the 6.1 armbian vendor dts are here (both with 4k camera overlays added):
fdt.dts.zip (77.0 KB)
In any case, it seems that the problem is that the device /dev/v4l-subdev4 which would have corresponded to rkcif-mipi-lvds2 is never created. Still don’t know why.
Here is newer version 2.0 of that rockchip camera developer guide document BTW. (Why is documentation found in so many different places?)
I don’t see anything wrong with the dts.
Unfortunately, I can’t help much, my whole setup is based on 5.10 due to using libmali, i can’t just switch kernel as you can with Armbian.
What i know is these guys are actively working on the rockchip media and perhaps they have up-to-date documentation (or they can talk to rockchip directly) and can give you some inside information.
@nyanmisaka @amazingfate
I think you are right.
If you want to debug the kernel: (https://github.com/armbian/linux-rockchip/blob/rk-6.1-rkr1/drivers/media/platform/rockchip/cif/capture.c#L6405)
I will follow the conversation…
Thanks. Yes amazingfate usually answers on discord armbian-rockchip channel. I’ll try to contact him there too.
Update: since this CIF and ISP driver/sdk software is Rockchips work their staff should be most suited for solving it. I sent an email to someone there.
@avaf @Oskar_Enoksson did you find a solution to the problem? I’m facing the same on a different SBC, however, my overlay for cameras works on 5.1.160 kernel but fails on both 6.1.y kernel from friendlyelec and also 6.1.57 armbian kernel…
I have reserved my ready-to-fire image with kernel 6.1 to Rock 5C but no sign of the board being delivered any time soon. I am waiting for a few SD card to arrive soon and then i will try on Rock 5B. I think OrangePi has kernel 6.6 , have you tried that kernel version?
I still have the same problem. The camera overlay works for me though, it is just the video processing drivers that don’t work (with armbian 6.1 vendor kernel that is, the 5.10 vendor kernel works).
I opened issues here and here but no solution yet.
Note that you must have “vendor” kernel and the only existing versions are 5.10 and 6.1 for all I know. The mainline 6.1 or 6.6 or any other kernel can’t use the MIPI/CSI camera as far as I know because the functionality is not yet implemented there. Collabora has it on their TODO list but I hear that they are not as focused on rockchip development lately.
Please, disregard this info.
thx for the info, at least I’m not alone 
FYI
I don’t know if you have found a solution to the issue with kernel 6.1 but it is working on rock-5c.
rock@rock-5c:~$ ./capture -d /dev/video11 -f NV12 -K 400 -c 500 -s 1920x1080
HDMI-A-1 -> connected
rga_api version 1.10.0_[9]
frameSize: 6220800, srcWidth: 1920, srcHeight: 1080
rock@rock-5c:~$ ls *.png
screenshot_1920x1080.png
rock@rock-5c:~$ ./capture -d /dev/video11 -f NV12 -c 2000 -s 1920x1080
HDMI-A-1 -> connected
rga_api version 1.10.0_[9]
rock@rock-5c:~$ uname -ra
Linux rock-5c 6.1.43-10-rk2312 #eb3b42978 SMP Fri Jun 7 02:38:46 UTC 2024 aarch64 GNU/Linux
rock@rock-5c:~$ v4l2-ctl -d /dev/video11 --get-fmt-video
Format Video Capture Multiplanar:
Width/Height : 1920/1080
Pixel Format : 'NV12' (Y/UV 4:2:0)
Field : None
Number of planes : 1
Flags :
Colorspace : Default
Transfer Function : Default
YCbCr/HSV Encoding: Default
Quantization : Full Range
Plane 0 :
Bytes per Line : 1920
Size Image : 3110400
rock@rock-5c:~$it’s been a while, however I just retried this on 6.1.99 armbian kernel today and found the issue to still be there… @Oskar_Enoksson did you ultimately find a fix?
@avaf just to be on the same side - this test captures from HDMI-IN, right? In @Oskar_Enoksson and my case we want to capture from a MIPI CAM…
I also tried it again a month ago, on latest kernel 6.1, same issue. I’m also trying to capture from a MIPI camera on the Rock5A, and trying to take advantage of the hardware rescaling ISP block in the CPU.
I think @avaf captures video on the device which is directly connected to the camera, i.e. without passing it through the ISP. I’m able to do that too, with the cameras native resolution and output format.
A few things do work, but it’s very shaky. Fiddling with v4l often hangs the system requiring reboot.
Yes. I mean it works on HDMI-IN as well.
The Kernel 6.1.x is running on Rock 5C, not Rock 5A. I think there are some minor differences regarding the MIPI connection on 5A. My advice is to use vendor kernel 6.1.43, find the correct DTS and then move on to newer kernel.
FYI
Kernel 6.1.x on Rock 5B is using CAMERAOUT_M3 / mipim0_camera3
Kernel 6.1.x on Rock 5C is using CAMERAOUT_M2 / mipim0_camera2
I think Rock 5A is more like Rock 5B than 5C.
@Oskar_Enoksson yes I verified that capturing the native raw data works fine for me as well, but the connection to the ISP is simply not there, so the /dev/v4l-subdev4 is not there…
@avaf I’m on the NanoPC-T6 actually, but it is exactly the same problem that Oskar has described… my setup is just either one single 4-lane OV4689 on csi0, or two 4-lane OV4689 on csi0 and csi1 - DeviceTree works on 5.10.160 fine, does not on 6.1.99
@avaf not sure I understand you right, but I am running Armbians “6.1.99-vendor-rk35xx” kernel on my Rock-5A (I understand you have Rock-5C and that there are differences, but Armbian has some sort of “unsupported inofficial support” for Rock-5A with a working devicetree). If you talk about mainline kernel I think RKISP support is one of the things that is unfortunately still not implemented there.
But does anyone know: is the idea that the desired connections between the video blocks of this CPU can be changed/configured in the devicetree? If so maybe that is the missing key to success?
I meant Radxa official OS. (6.1.43)
Interestingly, people with Rock 5A are silent about kernel 6.1 and the camera.
If i recall correctly, i had this issue when i tried rockchip 6.1.57 with the wrong camera engine.
FYI:
According to Rock 5A overlay, it uses CAMARAOUT_M2 and camera2_clk, same as Rock 5C (CSI2), did you try with this configuration?