Best option for YOLOv8 object detection?

avaf · May 12, 2025, 11:41am

Jagan · May 12, 2025, 11:50am

Code built.

How to run it, with camera.

$ ./ffrknn-sdl2-jagan -x 1920 -y 1080 -m /mnt/model/yolov8-v2.3.0/yolov8.rknn -f v4l2 -i 11 
Model: /mnt/model/yolov8-v2.3.0/yolov8.rknn - size: 4328139.
sdk version: 2.3.0 (c949ad889d@2024-11-07T11:35:33) driver version: 0.9.6
model input num: 1, output num: 9
model: 640x640x3
[video4linux2,v4l2 @ 0x55ad1377d0] Cannot open video device 11: No such file or directory
Cannot open input file '11'

avaf · May 12, 2025, 12:06pm

Example:
./ff-rknn -f v4l2 -p nv12 -s 1920x1080 -i /dev/video11 -x 1920 -y 1080 -b 28 -a 40 -m ./model/RK3588/yolov5s-640-640.rknn

You should note ffmpeg-rockchip is substantially changed/improved, if i recall correctly, the drm hack for rk3568 did not work on rk3588, you need to review that part of the hack and adjust that.

Jagan · May 12, 2025, 1:01pm

$ ./ffrknn-sdl2-jagan -f v4l2 -p nv12 -s 1920x1080 -i /dev/video11 -x 1920 -y 1080 -b 28 -a 40 -m /mnt/model/yolov8-v2.3.0/yolov8.rknn 
Model: /mnt/model/yolov8-v2.3.0/yolov8.rknn - size: 4328139.
sdk version: 2.3.0 (c949ad889d@2024-11-07T11:35:33) driver version: 0.9.6
model input num: 1, output num: 9
model: 640x640x3
[video4linux2,v4l2 @ 0x55bb7d57d0] Not a video capture device.
Cannot open input file '/dev/video11'

I have camera at /dev/video11.

I build the ffmpeg-rockchip from master as per the docs - https://github.com/nyanmisaka/ffmpeg-rockchip/wiki/Compilation

Do we need to apply any more changes on top of it for rk3588 fix?

This is my kernel

Linux version 6.1.43-gf6777ff27006 (tops@instance-20241224-145244) (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Wed Feb 26 07:53:24 UTC 2025

avaf · May 12, 2025, 1:50pm

github.com

nyanmisaka/ffmpeg-rockchip/blob/master/libavdevice/v4l2.c#L187


if (v4l2_ioctl(fd, VIDIOC_QUERYCAP, &cap) < 0) {
    err = AVERROR(errno);
    av_log(ctx, AV_LOG_ERROR, "ioctl(VIDIOC_QUERYCAP): %s\n",
           av_err2str(err));
    goto fail;
}


av_log(ctx, AV_LOG_VERBOSE, "fd:%d capabilities:%x\n",
       fd, cap.capabilities);


if (!(cap.capabilities & (V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_CAPTURE_MPLANE))) {
    av_log(ctx, AV_LOG_ERROR, "Not a video capture device.\n");
    err = AVERROR(ENODEV);
    goto fail;
}
s->multi_planer = ((cap.capabilities & (V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_CAPTURE_MPLANE)) == V4L2_CAP_VIDEO_CAPTURE_MPLANE) ? 1 : 0;


if (!(cap.capabilities & V4L2_CAP_STREAMING)) {
    av_log(ctx, AV_LOG_ERROR,
           "The device does not support the streaming I/O method.\n");
    err = AVERROR(ENOSYS);

Jagan · May 12, 2025, 2:02pm

I’m using the same code, Do we need to change capabilities?

Jagan · May 12, 2025, 3:24pm

Meanwhile, I tried to MP4 video input.

$ ./ffrknn-sdl2-jagan -i model/pexels-george.mp4 -x 1920 -y 1080 -b 28 -a 40 -m model/RK3588/yolov5s-640-640.rknn 
Model: model/RK3588/yolov5s-640-640.rknn - size: 8477315.
sdk version: 2.3.0 (c949ad889d@2024-11-07T11:35:33) driver version: 0.9.6
model input num: 1, output num: 3
model: 640x640x3
INFO: SDL: compiled with=2.26.5 linked against=2.26.5
INFO: Event thread running...
rga_api version 1.10.1_[4]
err act[0,0]
Error srcRect
[RgaBlit,776]Error srcRect 

handl-fd-vir-phy-hnd-format[0, -1, 0x7f5ef6f010, (nil), 0, 0]
rect[0, 0, 0, 0, 0, 0, 2560, 0]

Is this related to drm hack you mentioned above?

avaf · May 12, 2025, 6:51pm

I was referring to this code: [https://github.com/avafinger/ff-rknn/blob/main/ff-rknn.c#L382]
Try running ff-rknn to see if you get the same or similar error.

Looks like RGA is complaining.
Get more info with:

dmesg|grep -i rga

And see what parameters are being passed to rga, maybe you need hStride/wStride to be aligned.

Jagan · May 13, 2025, 8:01am

$ ./ff-rknn -i model/pexels-george.mp4 -x 1920 -y 1080 -l 0 -t 0 -m model/yolov8.rknn 
Model: model/yolov8.rknn - size: 5625667.
sdk version: 1.6.0 (9a7b5d24c@2023-12-13T17:31:11) driver version: 0.9.6
model input num: 1, output num: 3
model: 640x640x3
INFO: SDL: compiled with=2.26.5 linked against=2.26.5
[h264 @ 0x55a23c0f20] decode_slice_header error
[h264 @ 0x55a23c0f20] no frame!
Error sending a packet for decoding
Avg FPS: 0.0
$ sudo dmesg | grep -i rga
[   12.897951] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up rga30-supply from device tree
[   12.897980] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up rga30-supply property in node /power-management@fd8d8000/power-controller failed
[   12.898447] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up rga31-supply from device tree
[   12.898470] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up rga31-supply property in node /power-management@fd8d8000/power-controller failed
[   13.244934] rga3_core0 fdb60000.rga: Adding to iommu group 2
[   13.245088] rga: rga3_core0, irq = 46, match scheduler
[   13.245202] rga: rga3_core0 hardware loaded successfully, hw_version:3.0.76831.
[   13.245255] rga: rga3_core0 probe successfully
[   13.245494] rga3_core1 fdb70000.rga: Adding to iommu group 3
[   13.245573] rga: rga3_core1, irq = 47, match scheduler
[   13.245673] rga: rga3_core1 hardware loaded successfully, hw_version:3.0.76831.
[   13.245698] rga: rga3_core1 probe successfully
[   13.245917] rga: rga2, irq = 107, match scheduler
[   13.246023] rga: rga2 hardware loaded successfully, hw_version:3.2.63318.
[   13.246056] rga: rga2 probe successfully
[   13.246153] rga_iommu: IOMMU binding successfully, default mapping core[0x1]
[   13.246257] rga: Module initialized. v1.3.1

avaf · May 13, 2025, 1:12pm

In previous ffmpeg versions, the default was HW decoder.

Try:

av_dict_set(&opts, “hwaccel”, “rkmpp”, 0);

And

rect[0, 0, 0, 0, 0, 0, 2560, 0]

Looks like frame height and width are ‘0’

Update 2:
if you are using DRM (ff-rknn) you may need to add:

av_dict_set(&opts, “hwaccel_output_format”, “drm_prime”, 0);
av_dict_set(&opts, “afbc”, “1”, 0);

Narasimha_Vadlamudi · May 15, 2025, 12:09pm

i add the above lines , again same issue happen like
./ff-rknn -i model/pexels-george.mp4 -x 1920 -y 1080 -l 0 -t 0 -m model/yolov8.rknn
Model: model/yolov8.rknn - size: 5625667.
sdk version: 1.6.0 (9a7b5d24c@2023-12-13T17:31:11) driver version: 0.9.6
model input num: 1, output num: 3
model: 640x640x3
INFO: SDL: compiled with=2.26.5 linked against=2.26.5
[h264 @ 0x55a23c0f20] decode_slice_header error
[h264 @ 0x55a23c0f20] no frame!
Error sending a packet for decoding
Avg FPS: 0.0
any other way?

avaf · May 15, 2025, 1:13pm

    // av_dict_set(&opts, "hwaccel", "rkmpp", 0);
    // av_dict_set(&opts, "hwaccel_output_format", "drm_prime", 0);
    // av_dict_set(&opts, "afbc", "1", 0);
    // av_dict_set(&opts, "vcodec", "av1_rkmpp", 0);

You need to force the hardware decoder to make it a DRM_PRIME frame explicitly, the above options did not work.
Change the code to code below and link to your ffmpeg:

#if 0
    if (codecpar->codec_id != AV_CODEC_ID_H264) {
        av_log(0, AV_LOG_ERROR, "H264 support only!\n");
        avformat_close_input(&input_ctx);
        return -1;
    }
#endif

    if (codecpar->codec_id == AV_CODEC_ID_H264) {
        codec = avcodec_find_decoder_by_name("h264_rkmpp");
    } else {
        if (codecpar->codec_id == AV_CODEC_ID_HEVC) {
            codec = avcodec_find_decoder_by_name("hevc_rkmpp");
        } else {
            if (codecpar->codec_id == AV_CODEC_ID_AV1) {
                codec = avcodec_find_decoder_by_name("av1_rkmpp");
            } else {
                if (codecpar->codec_id == AV_CODEC_ID_VP9) {
                    codec = avcodec_find_decoder_by_name("vp9_rkmpp");
                } else {
                    if (codecpar->codec_id == AV_CODEC_ID_VP8) {
                        codec = avcodec_find_decoder_by_name("vp8_rkmpp");
                    } else {
                        if (codecpar->codec_id == AV_CODEC_ID_MPEG4) {
                            codec = avcodec_find_decoder_by_name("mpeg4_rkmpp");
                        } else {
                            if (codecpar->codec_id == AV_CODEC_ID_MPEG2VIDEO) {
                                codec = avcodec_find_decoder_by_name("mpeg2_rkmpp");
                            } else {
                                if (codecpar->codec_id == AV_CODEC_ID_MPEG1VIDEO) {
                                    codec = avcodec_find_decoder_by_name("mpeg1_rkmpp");
                                } else {
                                    codec = avcodec_find_decoder(codecpar->codec_id);
                                }
                            }
                        }
                    }
                }
            }
        }
    }
    if (!codec) {
        av_log(0, AV_LOG_ERROR, "Codec not found!\n");
        avformat_close_input(&input_ctx);
        return -1;
    }

    codec_ctx = avcodec_alloc_context3(codec);
    if (!codec_ctx) {
        av_log(0, AV_LOG_ERROR, "Could not allocate video codec context!\n");
        avformat_close_input(&input_ctx);
        return -1;
    }

More optimizations can be done, and a few fixes to improve, such as:

NV12 texture instead of RGB24
RAWVIDEO (feeding from MIPI camera) needs to be in dma buffer or convert NV12 to DRM_PRIME (using filter)

Jagan · May 15, 2025, 3:04pm

I don’t find these codes in ffmpeg-rockchip/master ? master seems to be 6.1 and there is no previous version tags either.

avaf · May 15, 2025, 6:19pm

By the previous version, i mean in the kernel 5.10 era.

Read carefully how to build @nyanmisaka’s ffmpeg
.

ffmpeg-rockchip-new/ffmpeg -hide_banner -decoders | grep rkmpp
 V..... av1_rkmpp            Rockchip MPP (Media Process Platform) AV1 decoder (codec av1)
 V..... h263_rkmpp           Rockchip MPP (Media Process Platform) H263 decoder (codec h263)
 V..... h264_rkmpp           Rockchip MPP (Media Process Platform) H264 decoder (codec h264)
 V..... hevc_rkmpp           Rockchip MPP (Media Process Platform) HEVC decoder (codec hevc)
 V..... mpeg1_rkmpp          Rockchip MPP (Media Process Platform) MPEG1VIDEO decoder (codec mpeg1video)
 V..... mpeg2_rkmpp          Rockchip MPP (Media Process Platform) MPEG2VIDEO decoder (codec mpeg2video)
 V..... mpeg4_rkmpp          Rockchip MPP (Media Process Platform) MPEG4 decoder (codec mpeg4)
 V..... vp8_rkmpp            Rockchip MPP (Media Process Platform) VP8 decoder (codec vp8)
 V..... vp9_rkmpp            Rockchip MPP (Media Process Platform) VP9 decoder (codec vp9)

I don’t know which board you are using, some people reported problems with /dev/video11 in kernel 6.1 (Rock 5A). I don’t know if the issue persists. I don’t have Rock 5A. What i can tell you is Rock 5B and 5C work fine with the camera in kernel 6.1

v4l2-ctl --list-devices
rkisp-statistics (platform: rkisp):
	/dev/video18
	/dev/video19

rkcif (platform:rkcif-mipi-lvds1):
	/dev/video0
	/dev/video1
	/dev/video2
	/dev/video3
	/dev/video4
	/dev/video5
	/dev/video6
	/dev/video7
	/dev/video8
	/dev/video9
	/dev/video10
	/dev/media0

rkisp_mainpath (platform:rkisp1-vir0):
	/dev/video11
	/dev/video12
	/dev/video13
	/dev/video14
	/dev/video15
	/dev/video16
	/dev/video17
	/dev/media1

One thing to note:
Boards with more than 4 GB need a specific mpp patch, but you can find some posts about it in the forum.