Connect Arducam HQ Camera to ROCK5B

Monstrofil · July 13, 2023, 7:07pm

Here is ISP tuner that I used.
https://dl.khadas.com/.test/isp.zip

Just to confirm, your IMX477 DOES NOT change exposure, analogue and digital gains automatically?

Meanwhile, I found debug switch and turned it on:

export persist_camera_engine_log=0xffffffffff

Now I see some errors and warnings.

XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:device(/dev/video18) dequeue buffer failed since not activated
XCORE:W:dequeue buffer failed
XCORE:D:ioctl return ok on fd(5), cmd:0x80885659
CAMHW:D:camId: 0, frameId: 30: dequeue the event on dev: /dev/v4l-subdev3
ANALYZER:D:camId:0, sequence(30), expDelay(4), id(33), maxId(33)
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(GRP_OTHER), flags 1
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(GRP1), flags 101
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(GRP_AWB), flags 41001
ANALYZER:D:camId: 0, group(GRP1): id[33] push msg(SOF_INFO), msg delayCnt(0), map size is 6
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(GRP0), flags 41a01
ANALYZER:W:camId:0 group(GRP1): id[28] map size is 6, erase 1, element, missing conditions: AWB_PROC_RES,
ANALYZER:D:Handle message(SOF_INFO) id[33] on group(AF), flags 42a01
ANALYZER:D:>>> Framenum=30, id=34, Cur sgain=0.000000,stime=0.000000,mgain=0.000000,mtime=0.000000,lgain=0.000000,ltime=0.000000
ANALYZER:D:>>> Framenum=30, id=34, nxt sgain=0.000000,stime=0.000000,mgain=0.000000,mtime=0.000000,lgain=0.000000,ltime=0.000000
ANALYZER:D:analyze the id(34), sequence(30), mLastAnalyzedId(33)
ANALYZER:D:camId: 0, group(GRP0): id[33] push msg(SOF_INFO), msg delayCnt(0), map size is 6
ANALYZER:D:camId: 0, group(GRP_OTHER): id[33] push msg(SOF_INFO), msg delayCnt(0), map size is 1
ANALYZER:E:no free cp buffer!

But I’m out of ideas. Camera driver works fine and returns correct data, but AE is just not working. Would be glad if someone with working AE would capture the log with debug enabled and attach here.

The only idea that I have now is to purchase imx415 sensor, create adapter for it and check how it works, because it seems to be the only imx sensor that stock ubuntu image has iqfile for and also I saw multiple chineese articles with working isp with it.

p.s. I also see that 3A tool makes a lot of ioctl calls to the camera when scene changes it’s brightness, maybe that is another direction to dive into…

[ 2658.797121] ov13850: imx477_update_image_pad_format
[ 2658.797124] ov13850: imx477_get_format_code
[ 2658.797139] ov13850: imx477_g_frame_interval
[ 2658.797147] ov13850: imx477_ioctl cmd=-2146412855
[ 2658.797150] RKMODULE_GET_NR_SWITCH_THRESHOLD
[ 2658.797154] ov13850: imx477_ioctl => -515
[ 2658.797164] ov13850: imx477_ioctl cmd=1074026182
[ 2658.797168] RKMODULE_SET_CONVERSION_GAIN
[ 2658.797171] ov13850: imx477_ioctl => -515
[ 2658.797177] ov13850: imx477_ioctl cmd=1074812616
[ 2658.797180] RKMODULE_SET_DPCC_CFG
[ 2658.797183] ov13850: imx477_ioctl => -515
[ 2658.897111] ov13850: imx477_get_fmt pad=0
[ 2658.897119] ov13850: imx477_update_image_pad_format
[ 2658.897122] ov13850: imx477_get_format_code
[ 2658.897136] ov13850: imx477_g_frame_interval
[ 2658.897145] ov13850: imx477_ioctl cmd=-2146412855
[ 2658.897148] RKMODULE_GET_NR_SWITCH_THRESHOLD
[ 2658.897151] ov13850: imx477_ioctl => -515
[ 2658.897161] ov13850: imx477_ioctl cmd=1074026182
[ 2658.897164] RKMODULE_SET_CONVERSION_GAIN
[ 2658.897167] ov13850: imx477_ioctl => -515
[ 2658.897174] ov13850: imx477_ioctl cmd=1074812616
[ 2658.897176] RKMODULE_SET_DPCC_CFG
[ 2658.897179] ov13850: imx477_ioctl => -515```

nickliu973 · July 13, 2023, 7:38pm

Thanks, @Monstrofil.
That link is very helpful and I can get back to hq camera asap.

I didn’t have a chance to test the auto exposure/gain, etc, as I was stuck at the ISP tuning like 5 or 6 month ago.
Attached is my scratch driver. imx477(1).zip (10.4 KB) It only support 1080p right now. Please feel free to let me know if there is any bug in there.

My last test renamed one of those IQ file to imx477.
If I remember correctly, I could see the auto changing of brightness (with some latency) or maybe it’s just artifact of incorrect ISP configuration.

I don’t know the csi intereface looks like in Orange PI.
I’m using cool-pi 4b board for this camera which does not need adapter
Cool pi cm5 with PCIE port may also worth to be considered (do double check the csi port, I didn’t remember exactly).

Bests:)

Zipdox · July 13, 2023, 11:18pm

So you’re using the original RPi HQ camera with 2-lane CSI?

Monstrofil · July 14, 2023, 12:51pm

@nickliu973 thanks for the driver. I tried to drop it in my kernel real quick and it seems that is works a bit different, but overall situation is pretty similar, gain/exposure (top right window) is not changed via ISP.
https://i.imgur.com/aPyuAvQ.png

Here is my driver. Would be great if you can try it once to check whether my issue is board-specific.
imx477.zip (13.6 KB)

@Zipdox, yes, stock RPi HQ camera with 2 lane CSI.

UPD: I found some suspicious lines in sources and I start thinking that exporuse and gains actually never worked properly.

github.com

khadas/android_external_camera_engine_rkaiq/blob/khadas-edge2-android12/hwi/isp3x/CamHwIsp3x.cpp#L353C18-L353C18


#if 0 //TODO: isp21 params has no exposure info field

nickliu973 · July 14, 2023, 4:13pm

Good morning, @Monstrofil
Thanks for sharing the information and link.

I did a quick test of the OV5648 (PI camera v1) + coolpi-4b + ubuntu 20.04 + 5.10.110 kernel (a bit customized):

Without running rkaiq_3A_server in debug mode, after enable the auto_exposure using v4l2-ctl, I could see the exposure time changing with the light using the v4l2-ctl.
However, if I enable the 3A log and run rkaiq_3A_server in debug mode, the exposure time is fixed to a constant value even if the auto_exposure is enabled.

Re-setup the IMX477 may need a bit more time (I have left the code on the shelf for a while). But the frame of my scratch driver is based on the OV5648.
The script I used for streaming camera is:

gst-launch-1.0 v4l2src device=/dev/video11 ! \
video/x-raw,format=NV12,width=640,height=480, framerate=30/1 ! \
videoconvert ! \
ximagesink

I also ran into the problem getting the YUV and raw image with the rkisp tuner.
Would you mind helping me with this? (are you using the Android OS?)
The error messages is

[aiqtool][Setup]:Linux,Create domain socket success.
[aiqtool][Setup]:connect domain server failed. Error
[aiqtool][ConnectAiq]:domain connect failed
[aiqtool][RkAiqSocketClientINETReceive]:########################################################
[aiqtool][RkAiqSocketClientINETReceive]:#### Forward to AIQ failed! please check AIQ status.####
[aiqtool][RkAiqSocketClientINETReceive]:########################################################

[aiqtool][DoCaptureOnlineRaw]:NO_HDR read_frame
[aiqtool][device_dqbuf]:device_dqbuf begin
[aiqtool][device_dqbuf]:device_dqbuf end
[aiqtool][device_dqbuf]:VIDIOC_DQBUF error 16, Device or resource busy
[aiqtool][operator()]:NO_HDR Capture finished,stop get ispStats.
[aiqtool][DoCaptureOnlineRaw]:DoCapture 52 ms 416 us
[aiqtool][DoCaptureOnlineRaw]:DoCapture exit!!!!!
[aiqtool][device_streamoff]:VIDIOC_STREAMOFF error 16, Device or resource busy
[aiqtool][uninit_device]:munmap error 22, Invalid argument
[aiqtool][uninit_device]:munmap error 22, Invalid argument
[aiqtool][uninit_device]:munmap error 22, Invalid argument
[aiqtool][uninit_device]:munmap error 22, Invalid argument
[aiqtool][HandlerOnLineMessage]:ProcID DATA_ID_CAPTURE_ONLINE_RAW_START out
[aiqtool][HandlerOnLineMessage]:CMD_TYPE_CAPTURE out

Bests:)

Monstrofil · July 14, 2023, 6:50pm

Finally I managed to make everything (except some minor things like lens correction) work.

Some imprortant things were missing:

rkisp does not control digital gain, only analogue, e.g. imx577 splits received value in the driver:
https://github.com/armbian/linux-rockchip/blob/rk-5.10-rkr4/drivers/media/i2c/imx577.c#L2075
iq files that isp tuner produces are quite unusable, better to start with copy-pasting something existing and tuning it afterwards

Final image, this time with automatic esposure and gain.
https://i.imgur.com/tZhSlPM.jpg

Driver and iqfile (AIQ 8.8(!!)) attached.
imx477_working.zip (72.1 KB)

Basically, you need two things running:
rkaiq_3A_server
rkaiq_tool_server (the one from isp.zip).

Here is how my log looks like on Ubuntu 22:

root@orangepi5:~# ps auxf | grep 3A
root         989  0.1  0.1 313484 11920 ?        Sl   02:30   0:00 /usr/bin/rkaiq_3A_server
root        3547  0.0  0.0  19680  2028 pts/0    S+   02:35   0:00          \_ grep 3A
root@orangepi5:~#
root@orangepi5:~#
root@orangepi5:~# ./rkaiq_tool_server-aarch64-20221025_112250_RK3588
[aiqtool][main]:#### AIQ tool server 20221019_174349 ####
[aiqtool][main]:iqfile cmd_parser.get  /oem/etc/iqfiles
[aiqtool][main]:g_mode cmd_parser.get  -1
[aiqtool][main]:g_width cmd_parser.get  1920
[aiqtool][main]:g_height cmd_parser.get  1080
[aiqtool][main]:g_device_id cmd_parser.get  0
[aiqtool][GetMediaInfo]:access /dev/media0
[aiqtool][GetCifSubDevs]:cif media index 0, media info array id  0
[aiqtool][GetLensSubDevs]:isp media index 0, media info array id  0
[aiqtool][GetMediaInfo]:access /dev/media1
[aiqtool][GetIspSubDevs]:isp media index 1, media info array id  0
[aiqtool][GetIspSubDevs]:model(rkisp0): isp_info(0): isp-subdev entity name: /dev/v4l-subdev3
[aiqtool][DumpMediaInfo]:DumpMediaInfo:
[aiqtool][DumpMediaInfo]:        sensor_name :    m00_f_imx477 7-001a
[aiqtool][DumpMediaInfo]:#### cif:
[aiqtool][DumpMediaInfo]:        model_idx :         0
[aiqtool][DumpMediaInfo]:        linked_sensor :     1
[aiqtool][DumpMediaInfo]:        media_dev_path :    /dev/media0
[aiqtool][DumpMediaInfo]:        mipi_id0 :               /dev/video0
[aiqtool][DumpMediaInfo]:        mipi_id1 :               /dev/video1
[aiqtool][DumpMediaInfo]:        mipi_id2 :               /dev/video2
[aiqtool][DumpMediaInfo]:        mipi_id3 :               /dev/video3
[aiqtool][DumpMediaInfo]:        rkcif_tools_id0 :   /dev/video10
[aiqtool][DumpMediaInfo]:        mipi_dphy_rx_path :
[aiqtool][DumpMediaInfo]:        mipi_csi2_sd_path : /dev/v4l-subdev0
[aiqtool][DumpMediaInfo]:        lvds_sd_path :
[aiqtool][DumpMediaInfo]:        mipi_luma_path :
[aiqtool][DumpMediaInfo]:#### isp:
[aiqtool][DumpMediaInfo]:        model_idx :         1
[aiqtool][DumpMediaInfo]:        linked_sensor :     0
[aiqtool][DumpMediaInfo]:        media_dev_path :    /dev/media1
[aiqtool][DumpMediaInfo]:        isp_dev_path :      /dev/v4l-subdev3
[aiqtool][DumpMediaInfo]:        csi_dev_path :
[aiqtool][DumpMediaInfo]:        mpfbc_dev_path :
[aiqtool][DumpMediaInfo]:        main_path :         /dev/video11
[aiqtool][DumpMediaInfo]:        self_path :         /dev/video12
[aiqtool][DumpMediaInfo]:        fbc_path :          /dev/video13
[aiqtool][DumpMediaInfo]:        rawwr0_path :
[aiqtool][DumpMediaInfo]:        rawwr1_path :
[aiqtool][DumpMediaInfo]:        rawwr2_path :
[aiqtool][DumpMediaInfo]:        rawwr3_path :
[aiqtool][DumpMediaInfo]:        dma_path :
[aiqtool][DumpMediaInfo]:        rawrd0_m_path :     /dev/video15
[aiqtool][DumpMediaInfo]:        rawrd1_l_path :     /dev/video17
[aiqtool][DumpMediaInfo]:        rawrd2_s_path :     /dev/video16
[aiqtool][DumpMediaInfo]:        stats_path :        /dev/video18
[aiqtool][DumpMediaInfo]:        input_params_path : /dev/video19
[aiqtool][DumpMediaInfo]:        mipi_luma_path :
[aiqtool][DumpMediaInfo]:        mipi_dphy_rx_path :
[aiqtool][main]:================== -1 =====================
[aiqtool][main]:ToolServer using old socket node
[aiqtool][Setup]:Linux,Create domain socket success.
[aiqtool][Setup]:Credentials from SO_PEERCRED: pid=989, euid=0, egid=0
[aiqtool][main]:#### ToolServer connect AIQ success ####
[aiqtool][Process]:TCPServer::Process
[aiqtool][Accepted]:TCPServer::Accepted

Based on “#### Forward to AIQ failed! please check AIQ status.####” I can assume that you have 3A script not running.

Hz_Lin · July 15, 2023, 3:07am

@Raphael you mentioned that veye is partnering with radxa.
Actually the modules offered by veye suits my need most. I am more interested in laboratory applications, so a monochrome global-shutter camera like MV-MIPI-IMX296M is amazing! Given that it has been adapted for firefly rk3588 boards, porting it to a radxa board should be a pretty low hanging fruit? I would love to have one of those run on a rock 3 or rock 5 to do some basic monitoring. Is there interest on radxa / veye side to make this work? This combo can replace some industrial camera like the Firefly S we currently use today (costs $234).
If there is no plan to make such adapter and driver commercially, is there any instruction for how to get started?

avaf · July 15, 2023, 1:46pm

@Zipdox

Did you test your design?

@Monstrofil

Nice job. Can you give us more detail about the quality of the image and FPS / window size you get with your current setup? Can you also compare your results with imx477 vs the OV13855 .

Zipdox · July 15, 2023, 9:11pm

No I did not test it, as it doesn’t have level shifters. The last thing I wanna do is fry my camera or the SBC.

But I’m working on a new design. I want to make it an FPC that plugs into the camera. I’m probably gonna need a little help with routing though.
https://oshwlab.com/zipdox/arducam-hq-camera-to-rock5b-adapter

Monstrofil · July 25, 2023, 8:15pm

4048x3040, 10 fps, 12bbp which seems to be maximum of what this sensor can do with 2 mipi lanes.

Unfortunately I cannot compare them as I don’t have any ov sensors laying around, but I can make some photos later. Probably “later” whould be when I finish my project of 360 camera creation, which I hope I can do in 1-2 month.

avaf · July 25, 2023, 8:51pm

No need to rush, thanks.
I compared ov13855 vs imx415 6 months ago but the OV driver was/is in such bad shape and has no available documentation. I was curious to know a direct comparison on the same board. I would be happy with a screenshot with gstreamer when you have time.

Raphael · July 28, 2023, 5:46pm

There has been some time since I last commented here, but I can finally say, the first prototypes have arrived and I was able to port the Arducam driver also to the Rock5A, I’ll share a picture which shows a Arducam Pivariety camera, tuning is not perfect yet and there might be a lot that needs to be improved but … here is a proof of concept (it’s also a little out of focus, didn’t have my glasses on while playing with it)

avaf · July 28, 2023, 6:07pm

Out of curiosity, what are those avionics control-level things drawn on the screen?

Raphael · July 28, 2023, 8:09pm

It’s an open source digital video transmission system for drones. Which I develop for:

cnoah · September 22, 2023, 7:44am

My rk3399 with imx477 images were above. Anybody know which i’m missing?

incognito · September 18, 2023, 11:19am

https://forums.raspberrypi.com/viewtopic.php?t=278828 is this helpful at all?

Also, what board would be rk3399? That’s not Rock5

Goetterfunke · September 19, 2023, 7:41pm

AFAIK you need a file like this for imx415 to have correct color correction etc. /etc/iqfiles/imx415_RADXA-CAMERA-4K_DEFAULT.json

Raphael · September 20, 2023, 12:11am

you need somebody to tune the sensor for your board, it isn’t universally

Raphael · September 20, 2023, 12:13am

A little hint, currently I have prototypes for the IMX462 and IMX708 on my Rock5A and Rock5B working.
Currently discussing when/how availability will be handled, and after that I’ll merge the drivers/overlays into the Radxa kernels.
Primary they are made for rkr3.4 but I have also made them work on 4.1.

Monstrofil · September 21, 2023, 7:24pm

You are probably using original lenses which have blue tint on the filter and I swapped those lenses with m12 lenses which have orange tint, that explains colour issues. Sorry that I did not mention that earlier, I’ll take a look if I still have original lens for test.