Connect Arducam HQ Camera to ROCK5B

Raphael · June 22, 2023, 9:35pm

Most of the cameras come with M12 lenses, which can be changed.

Monstrofil · July 10, 2023, 6:32pm

First colour-correct’ish image captured on imx477 & RK3588s device.

https://i.imgur.com/AUeI4bI.jpg

Zipdox · July 10, 2023, 6:58pm

Holy crap? Did you write drivers?

Monstrofil · July 10, 2023, 7:14pm

Hardware adapter, slightly modified rpi driver, the tools to make isp work (rkaiq_3A_server), the tools to calibrate isp (rkaiq_tool_server, rkisp tuner, /holy crap, that was really hard to make them work together/), some Chinese documentation, couple of lamps with different
temperatures and yea, here we go with something we can call a “photo”.

Still trying to figure out why is it so dark tho, I’m shining with 15w led lamp directly on that calibration target within 40 cm.

Zipdox · July 10, 2023, 8:22pm

Will you be documenting this and sharing the code?

Monstrofil · July 12, 2023, 4:00pm

Sure.

But I just noticed that exposure and both analog and digital gains are not controlled by ISP, which is… weird and I don’t know how to fix that.

nickliu973 · July 12, 2023, 7:01pm

Hi, @Monstrofil,

This progress is fantastic.
Not sure what’s your driver looks like.
In my scratch driver, I have to use the v4l2-ctl command to config the IMX477 instead of ISP, e.g.,

sudo v4l2-ctl -d /dev/v4l-subdev2 --list-ctrls-menus # get the control 
sudo v4l2-ctl -d /dev/v4l-subdev2 -c auto_exposure=0 # enable auto exposure
sudo v4l2-ctl -d /dev/v4l-subdev2 -c exposure=300 # set exposure to 300

You may need to replace v4l-subdev2 with the device ID in your setup.

Bests:)

Monstrofil · July 12, 2023, 8:24pm

@nickliu973 mine is almost identical to https://github.com/raspberrypi/linux/blob/rpi-5.10.y/drivers/media/i2c/imx477.c

I see that you turn on (?) auto_exposure parameter, mine does not have such option, and it seems by datasheet that imx477 does not have any onboard ISP.

sudo v4l2-ctl -d /dev/v4l-subdev2 -c auto_exposure=0 # enable auto exposure

And later you set exposure manually, I know about that option, but that means that exposure is not controlled by ISP in your setup, right?

sudo v4l2-ctl -d /dev/v4l-subdev2 -c exposure=300 # set exposure to 300

According to the articles, making automatic exposure work should be as easy as running rkaiq_3A_server in background, but in my case I don’t see any attempts of exposure control at all and no errors in dmesg or 3a tool logs

Zipdox · July 12, 2023, 10:30pm

What hardware adapter did you use? The one I made? Does it work, or did you make your own?

Monstrofil · July 13, 2023, 2:26pm

Different one, because I’m using another board.
https://oshwlab.com/shalal545/mipi-adapters

PCB 5/5.

Does anybody know where I can find description of ioctl’s used by rkisp?

github.com

armbian/linux-rockchip/blob/rk-5.10-rkr4/drivers/media/i2c/imx415.c#L1692


	struct imx415 *imx415 = to_imx415(sd);
	struct rkmodule_hdr_cfg *hdr;
	struct rkmodule_channel_info *ch_info;
	u32 i, h, w, stream;
	long ret = 0;
	const struct imx415_mode *mode;
	u64 pixel_rate = 0;
	struct rkmodule_csi_dphy_param *dphy_param;


	switch (cmd) {
	case PREISP_CMD_SET_HDRAE_EXP:
		if (imx415->cur_mode->hdr_mode == HDR_X2)
			ret = imx415_set_hdrae(imx415, arg);
		else if (imx415->cur_mode->hdr_mode == HDR_X3)
			ret = imx415_set_hdrae_3frame(imx415, arg);
		break;
	case RKMODULE_GET_MODULE_INFO:
		imx415_get_module_inf(imx415, (struct rkmodule_inf *)arg);
		break;
	case RKMODULE_GET_HDR_CFG:
		hdr = (struct rkmodule_hdr_cfg *)arg;

PREISP_CMD_SET_HDRAE_EXP
RKMODULE_SET_HDR_CFG
RKMODULE_SET_HDR_CFG

I thought that they are not important in terms of AE configuration because v4l has standart V4L2_CID_EXPOSURE control, but now I see that rkisp does not call V4L2_CID_EXPOSURE, but makes some ioctl calls.

nickliu973 · July 13, 2023, 4:02pm

@Monstrofil, my driver basically adopted from the Radxa OV5647 and IMX477 driver.

I remember the camera driver has to use some rockchip API to be probed by kernel(?) Like the code for imx477_probe

ret = of_property_read_u32(node, RKMODULE_CAMERA_MODULE_INDEX,
				   &imx477->module_index);
ret |= of_property_read_string(node, RKMODULE_CAMERA_MODULE_FACING,
				       &imx477->module_facing);
ret |= of_property_read_string(node, RKMODULE_CAMERA_MODULE_NAME,
				       &imx477->module_name);
ret |= of_property_read_string(node, RKMODULE_CAMERA_LENS_NAME,
				       &imx477->len_name);

Sorry for the script of enable auto_exposure, I just want to show some example configuration.
This is what I used in my driver to setup the HDRAE

static long imx477_compat_ioctl32(struct v4l2_subdev *sd,
				  unsigned int cmd, unsigned long arg)
{
	void __user *up = compat_ptr(arg);
	struct rkmodule_inf *inf;
	struct rkmodule_awb_cfg *cfg;
	struct rkmodule_hdr_cfg *hdr;
	struct preisp_hdrae_exp_s *hdrae;
	long ret;
	u32 stream = 0;

	switch (cmd) {
	case RKMODULE_GET_MODULE_INFO:
		inf = kzalloc(sizeof(*inf), GFP_KERNEL);
		if (!inf) {
			ret = -ENOMEM;
			return ret;
		}

		ret = imx477_ioctl(sd, cmd, inf);
		if (!ret)
			ret = copy_to_user(up, inf, sizeof(*inf));
		kfree(inf);
		break;
	case RKMODULE_AWB_CFG:
		cfg = kzalloc(sizeof(*cfg), GFP_KERNEL);
		if (!cfg) {
			ret = -ENOMEM;
			return ret;
		}

		ret = copy_from_user(cfg, up, sizeof(*cfg));
		if (!ret)
			ret = imx477_ioctl(sd, cmd, cfg);
		kfree(cfg);
		break;
	case RKMODULE_GET_HDR_CFG:
		hdr = kzalloc(sizeof(*hdr), GFP_KERNEL);
		if (!hdr) {
			ret = -ENOMEM;
			return ret;
		}

		ret = imx477_ioctl(sd, cmd, hdr);
		if (!ret)
			ret = copy_to_user(up, hdr, sizeof(*hdr));
		kfree(hdr);
		break;
	case RKMODULE_SET_HDR_CFG:
		hdr = kzalloc(sizeof(*hdr), GFP_KERNEL);
		if (!hdr) {
			ret = -ENOMEM;
			return ret;
		}

		ret = copy_from_user(hdr, up, sizeof(*hdr));
		if (!ret)
			ret = imx477_ioctl(sd, cmd, hdr);
		kfree(hdr);
		break;
	case PREISP_CMD_SET_HDRAE_EXP:
		hdrae = kzalloc(sizeof(*hdrae), GFP_KERNEL);
		if (!hdrae) {
			ret = -ENOMEM;
			return ret;
		}

		ret = copy_from_user(hdrae, up, sizeof(*hdrae));
		if (!ret)
			ret = imx477_ioctl(sd, cmd, hdrae);
		kfree(hdrae);
		break;
	case RKMODULE_SET_QUICK_STREAM:
		ret = copy_from_user(&stream, up, sizeof(u32));
		if (!ret)
			ret = imx477_ioctl(sd, cmd, &stream);
		break;
	default:
		ret = -ENOIOCTLCMD;
		break;
	}

	return ret;
}

BTW, would you mind sharing the rkisp tuner? I tried to search it online but failed.
I would really appreciate if you could help me out.

Monstrofil · July 13, 2023, 7:07pm

Here is ISP tuner that I used.
https://dl.khadas.com/.test/isp.zip

Just to confirm, your IMX477 DOES NOT change exposure, analogue and digital gains automatically?

Meanwhile, I found debug switch and turned it on:

export persist_camera_engine_log=0xffffffffff

Now I see some errors and warnings.

XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:ioctl return ok on fd(5), cmd:0x80885659
CAMHW:D:camId: 0, frameId: 30: dequeue the event on dev: /dev/v4l-subdev3
ANALYZER:D:camId:0, sequence(30), expDelay(4), id(33), maxId(33)
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(GRP_OTHER), flags 1
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(GRP1), flags 101
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(GRP_AWB), flags 41001
ANALYZER:D:camId: 0, group(GRP1): id[33] push msg(SOF_INFO), msg delayCnt(0), map size is 6
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(GRP0), flags 41a01
ANALYZER:W:camId:0 group(GRP1): id[28] map size is 6, erase 1, element, missing conditions: AWB_PROC_RES,
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(AF), flags 42a01
ANALYZER:D:>>> Framenum=30, id=34, Cur sgain=0.000000,stime=0.000000,mgain=0.000000,mtime=0.000000,lgain=0.000000,ltime=0.000000
ANALYZER:D:>>> Framenum=30, id=34, nxt sgain=0.000000,stime=0.000000,mgain=0.000000,mtime=0.000000,lgain=0.000000,ltime=0.000000
ANALYZER:D:analyze the id(34), sequence(30), mLastAnalyzedId(33)
ANALYZER:D:camId: 0, group(GRP0): id[33] push msg(SOF_INFO), msg delayCnt(0), map size is 6
ANALYZER:D:camId: 0, group(GRP_OTHER): id[33] push msg(SOF_INFO), msg delayCnt(0), map size is 1
ANALYZER:E:no free cp buffer!

But I’m out of ideas. Camera driver works fine and returns correct data, but AE is just not working. Would be glad if someone with working AE would capture the log with debug enabled and attach here.

The only idea that I have now is to purchase imx415 sensor, create adapter for it and check how it works, because it seems to be the only imx sensor that stock ubuntu image has iqfile for and also I saw multiple chineese articles with working isp with it.

p.s. I also see that 3A tool makes a lot of ioctl calls to the camera when scene changes it’s brightness, maybe that is another direction to dive into…

[ 2658.797121] ov13850: imx477_update_image_pad_format
[ 2658.797124] ov13850: imx477_get_format_code
[ 2658.797139] ov13850: imx477_g_frame_interval
[ 2658.797147] ov13850: imx477_ioctl cmd=-2146412855
[ 2658.797150] RKMODULE_GET_NR_SWITCH_THRESHOLD
[ 2658.797154] ov13850: imx477_ioctl => -515
[ 2658.797164] ov13850: imx477_ioctl cmd=1074026182
[ 2658.797168] RKMODULE_SET_CONVERSION_GAIN
[ 2658.797171] ov13850: imx477_ioctl => -515
[ 2658.797177] ov13850: imx477_ioctl cmd=1074812616
[ 2658.797180] RKMODULE_SET_DPCC_CFG
[ 2658.797183] ov13850: imx477_ioctl => -515
[ 2658.897111] ov13850: imx477_get_fmt pad=0
[ 2658.897119] ov13850: imx477_update_image_pad_format
[ 2658.897122] ov13850: imx477_get_format_code
[ 2658.897136] ov13850: imx477_g_frame_interval
[ 2658.897145] ov13850: imx477_ioctl cmd=-2146412855
[ 2658.897148] RKMODULE_GET_NR_SWITCH_THRESHOLD
[ 2658.897151] ov13850: imx477_ioctl => -515
[ 2658.897161] ov13850: imx477_ioctl cmd=1074026182
[ 2658.897164] RKMODULE_SET_CONVERSION_GAIN
[ 2658.897167] ov13850: imx477_ioctl => -515
[ 2658.897174] ov13850: imx477_ioctl cmd=1074812616
[ 2658.897176] RKMODULE_SET_DPCC_CFG
[ 2658.897179] ov13850: imx477_ioctl => -515```

nickliu973 · July 13, 2023, 7:38pm

Thanks, @Monstrofil.
That link is very helpful and I can get back to hq camera asap.

I didn’t have a chance to test the auto exposure/gain, etc, as I was stuck at the ISP tuning like 5 or 6 month ago.
Attached is my scratch driver. imx477(1).zip (10.4 KB) It only support 1080p right now. Please feel free to let me know if there is any bug in there.

My last test renamed one of those IQ file to imx477.
If I remember correctly, I could see the auto changing of brightness (with some latency) or maybe it’s just artifact of incorrect ISP configuration.

I don’t know the csi intereface looks like in Orange PI.
I’m using cool-pi 4b board for this camera which does not need adapter
Cool pi cm5 with PCIE port may also worth to be considered (do double check the csi port, I didn’t remember exactly).

Bests:)

Zipdox · July 13, 2023, 11:18pm

So you’re using the original RPi HQ camera with 2-lane CSI?

Monstrofil · July 14, 2023, 12:51pm

@nickliu973 thanks for the driver. I tried to drop it in my kernel real quick and it seems that is works a bit different, but overall situation is pretty similar, gain/exposure (top right window) is not changed via ISP.
https://i.imgur.com/aPyuAvQ.png

Here is my driver. Would be great if you can try it once to check whether my issue is board-specific.
imx477.zip (13.6 KB)

@Zipdox, yes, stock RPi HQ camera with 2 lane CSI.

UPD: I found some suspicious lines in sources and I start thinking that exporuse and gains actually never worked properly.

github.com

khadas/android_external_camera_engine_rkaiq/blob/khadas-edge2-android12/hwi/isp3x/CamHwIsp3x.cpp#L353C18-L353C18


#if 0 //TODO: isp21 params has no exposure info field

nickliu973 · July 14, 2023, 4:13pm

Good morning, @Monstrofil
Thanks for sharing the information and link.

I did a quick test of the OV5648 (PI camera v1) + coolpi-4b + ubuntu 20.04 + 5.10.110 kernel (a bit customized):

Without running rkaiq_3A_server in debug mode, after enable the auto_exposure using v4l2-ctl, I could see the exposure time changing with the light using the v4l2-ctl.
However, if I enable the 3A log and run rkaiq_3A_server in debug mode, the exposure time is fixed to a constant value even if the auto_exposure is enabled.

Re-setup the IMX477 may need a bit more time (I have left the code on the shelf for a while). But the frame of my scratch driver is based on the OV5648.
The script I used for streaming camera is:

gst-launch-1.0 v4l2src device=/dev/video11 ! \
video/x-raw,format=NV12,width=640,height=480, framerate=30/1 ! \
videoconvert ! \
ximagesink

I also ran into the problem getting the YUV and raw image with the rkisp tuner.
Would you mind helping me with this? (are you using the Android OS?)
The error messages is

[aiqtool][Setup]:Linux,Create domain socket success.
[aiqtool][Setup]:connect domain server failed. Error
[aiqtool][ConnectAiq]:domain connect failed
[aiqtool][RkAiqSocketClientINETReceive]:########################################################
[aiqtool][RkAiqSocketClientINETReceive]:#### Forward to AIQ failed! please check AIQ status.####
[aiqtool][RkAiqSocketClientINETReceive]:########################################################

[aiqtool][DoCaptureOnlineRaw]:NO_HDR read_frame
[aiqtool][device_dqbuf]:device_dqbuf begin
[aiqtool][device_dqbuf]:device_dqbuf end
[aiqtool][device_dqbuf]:VIDIOC_DQBUF error 16, Device or resource busy
[aiqtool][operator()]:NO_HDR Capture finished,stop get ispStats.
[aiqtool][DoCaptureOnlineRaw]:DoCapture 52 ms 416 us
[aiqtool][DoCaptureOnlineRaw]:DoCapture exit!!!!!
[aiqtool][device_streamoff]:VIDIOC_STREAMOFF error 16, Device or resource busy
[aiqtool][uninit_device]:munmap error 22, Invalid argument
[aiqtool][uninit_device]:munmap error 22, Invalid argument
[aiqtool][uninit_device]:munmap error 22, Invalid argument
[aiqtool][uninit_device]:munmap error 22, Invalid argument
[aiqtool][HandlerOnLineMessage]:ProcID DATA_ID_CAPTURE_ONLINE_RAW_START out
[aiqtool][HandlerOnLineMessage]:CMD_TYPE_CAPTURE out

Bests:)

Monstrofil · July 14, 2023, 6:50pm

Finally I managed to make everything (except some minor things like lens correction) work.

Some imprortant things were missing:

rkisp does not control digital gain, only analogue, e.g. imx577 splits received value in the driver:
https://github.com/armbian/linux-rockchip/blob/rk-5.10-rkr4/drivers/media/i2c/imx577.c#L2075
iq files that isp tuner produces are quite unusable, better to start with copy-pasting something existing and tuning it afterwards

Final image, this time with automatic esposure and gain.
https://i.imgur.com/tZhSlPM.jpg

Driver and iqfile (AIQ 8.8(!!)) attached.
imx477_working.zip (72.1 KB)

Basically, you need two things running:
rkaiq_3A_server
rkaiq_tool_server (the one from isp.zip).

Here is how my log looks like on Ubuntu 22:

root@orangepi5:~# ps auxf | grep 3A
root         989  0.1  0.1 313484 11920 ?        Sl   02:30   0:00 /usr/bin/rkaiq_3A_server
root        3547  0.0  0.0  19680  2028 pts/0    S+   02:35   0:00          \_ grep 3A
root@orangepi5:~#
root@orangepi5:~#
root@orangepi5:~# ./rkaiq_tool_server-aarch64-20221025_112250_RK3588
[aiqtool][main]:#### AIQ tool server 20221019_174349 ####
[aiqtool][main]:iqfile cmd_parser.get  /oem/etc/iqfiles
[aiqtool][main]:g_mode cmd_parser.get  -1
[aiqtool][main]:g_width cmd_parser.get  1920
[aiqtool][main]:g_height cmd_parser.get  1080
[aiqtool][main]:g_device_id cmd_parser.get  0
[aiqtool][GetMediaInfo]:access /dev/media0
[aiqtool][GetCifSubDevs]:cif media index 0, media info array id  0
[aiqtool][GetLensSubDevs]:isp media index 0, media info array id  0
[aiqtool][GetMediaInfo]:access /dev/media1
[aiqtool][GetIspSubDevs]:isp media index 1, media info array id  0
[aiqtool][GetIspSubDevs]:model(rkisp0): isp_info(0): isp-subdev entity name: /dev/v4l-subdev3
[aiqtool][DumpMediaInfo]:DumpMediaInfo:
[aiqtool][DumpMediaInfo]:        sensor_name :    m00_f_imx477 7-001a
[aiqtool][DumpMediaInfo]:#### cif:
[aiqtool][DumpMediaInfo]:        model_idx :         0
[aiqtool][DumpMediaInfo]:        linked_sensor :     1
[aiqtool][DumpMediaInfo]:        media_dev_path :    /dev/media0
[aiqtool][DumpMediaInfo]:        mipi_id0 :               /dev/video0
[aiqtool][DumpMediaInfo]:        mipi_id1 :               /dev/video1
[aiqtool][DumpMediaInfo]:        mipi_id2 :               /dev/video2
[aiqtool][DumpMediaInfo]:        mipi_id3 :               /dev/video3
[aiqtool][DumpMediaInfo]:        rkcif_tools_id0 :   /dev/video10
[aiqtool][DumpMediaInfo]:        mipi_dphy_rx_path :
[aiqtool][DumpMediaInfo]:        mipi_csi2_sd_path : /dev/v4l-subdev0
[aiqtool][DumpMediaInfo]:        lvds_sd_path :
[aiqtool][DumpMediaInfo]:        mipi_luma_path :
[aiqtool][DumpMediaInfo]:#### isp:
[aiqtool][DumpMediaInfo]:        model_idx :         1
[aiqtool][DumpMediaInfo]:        linked_sensor :     0
[aiqtool][DumpMediaInfo]:        media_dev_path :    /dev/media1
[aiqtool][DumpMediaInfo]:        isp_dev_path :      /dev/v4l-subdev3
[aiqtool][DumpMediaInfo]:        csi_dev_path :
[aiqtool][DumpMediaInfo]:        mpfbc_dev_path :
[aiqtool][DumpMediaInfo]:        main_path :         /dev/video11
[aiqtool][DumpMediaInfo]:        self_path :         /dev/video12
[aiqtool][DumpMediaInfo]:        fbc_path :          /dev/video13
[aiqtool][DumpMediaInfo]:        rawwr0_path :
[aiqtool][DumpMediaInfo]:        rawwr1_path :
[aiqtool][DumpMediaInfo]:        rawwr2_path :
[aiqtool][DumpMediaInfo]:        rawwr3_path :
[aiqtool][DumpMediaInfo]:        dma_path :
[aiqtool][DumpMediaInfo]:        rawrd0_m_path :     /dev/video15
[aiqtool][DumpMediaInfo]:        rawrd1_l_path :     /dev/video17
[aiqtool][DumpMediaInfo]:        rawrd2_s_path :     /dev/video16
[aiqtool][DumpMediaInfo]:        stats_path :        /dev/video18
[aiqtool][DumpMediaInfo]:        input_params_path : /dev/video19
[aiqtool][DumpMediaInfo]:        mipi_luma_path :
[aiqtool][DumpMediaInfo]:        mipi_dphy_rx_path :
[aiqtool][main]:================== -1 =====================
[aiqtool][main]:ToolServer using old socket node
[aiqtool][Setup]:Linux,Create domain socket success.
[aiqtool][Setup]:Credentials from SO_PEERCRED: pid=989, euid=0, egid=0
[aiqtool][main]:#### ToolServer connect AIQ success ####
[aiqtool][Process]:TCPServer::Process
[aiqtool][Accepted]:TCPServer::Accepted

Based on “#### Forward to AIQ failed! please check AIQ status.####” I can assume that you have 3A script not running.

Hz_Lin · July 15, 2023, 3:07am

@Raphael you mentioned that veye is partnering with radxa.
Actually the modules offered by veye suits my need most. I am more interested in laboratory applications, so a monochrome global-shutter camera like MV-MIPI-IMX296M is amazing! Given that it has been adapted for firefly rk3588 boards, porting it to a radxa board should be a pretty low hanging fruit? I would love to have one of those run on a rock 3 or rock 5 to do some basic monitoring. Is there interest on radxa / veye side to make this work? This combo can replace some industrial camera like the Firefly S we currently use today (costs $234).
If there is no plan to make such adapter and driver commercially, is there any instruction for how to get started?

avaf · July 15, 2023, 1:46pm

@Zipdox

Did you test your design?

@Monstrofil

Nice job. Can you give us more detail about the quality of the image and FPS / window size you get with your current setup? Can you also compare your results with imx477 vs the OV13855 .

Zipdox · July 15, 2023, 9:11pm

No I did not test it, as it doesn’t have level shifters. The last thing I wanna do is fry my camera or the SBC.

But I’m working on a new design. I want to make it an FPC that plugs into the camera. I’m probably gonna need a little help with routing though.
https://oshwlab.com/zipdox/arducam-hq-camera-to-rock5b-adapter