IMX415 + NPU demo on ROCK 5B

Jagan · February 6, 2024, 10:03am

Made necessary changes, and able to built ff-rknn. Look like imx415 (MIPI-based) not possible to test isn’t it? Tried rkisp_mainpath video11 but it is not working. Can you share vid-3.mp4 file for testing? let me know the change for mipi camera.

Jagan · February 6, 2024, 11:04am

tops@ncm6b:~/ff-rknn$ sudo ./ff-rknn -f v4l2 -p h264 -s 1920x1080 -i /dev/video11 -m model/RK3588/yolov5s-640-640.rknn -x 960 -y 540
Model: model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
[video4linux2,v4l2 @ 0xaaaafb2ce070] Not a video capture device.
Cannot open input file '/dev/video11'

Log for mainpath video file, imx415

avaf · February 6, 2024, 12:02pm

You need the FFmpeg with MIPI support, see my previous comment.

Jagan · February 6, 2024, 2:07pm

Yes I did build the FFmpeg with v4l2. - https://github.com/hbiyik/FFmpeg.git

$./configure --enable-libv4l2

$ ffmpeg -f v4l2 -pixel_format nv12 -framerate 30 -video_size 1920x1080 -i /dev/video11 out.h264
ffmpeg version 65f90322bf Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --enable-libv4l2
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
[video4linux2,v4l2 @ 0xaaaaeea58350] ioctl(VIDIOC_G_INPUT): Inappropriate ioctl for device
[video4linux2,v4l2 @ 0xaaaaeea58350] ioctl(VIDIOC_G_PARM): Inappropriate ioctl for device, using framerate 30/1
Input #0, video4linux2,v4l2, from '/dev/video11':
  Duration: N/A, start: 3740.028140, bitrate: 829 kb/s
  Stream #0:0: Video: rawvideo (NV12 / 0x3231564E), nv12, 1920x1080, 829 kb/s, 0.03 fps, 0.03 tbr, 1000k tbn
File 'out.h264' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0xaaaaeea69770] Could not find a valid device
[h264_v4l2m2m @ 0xaaaaeea69770] can't configure encoder
[vost#0:0/h264_v4l2m2m @ 0xaaaaeea694a0] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

avaf · February 6, 2024, 2:41pm

This is an FFmpeg issue. Most likely you need to disable h264_v4l2m2m and a few others, note the codec used for MIPI will be RAW, so you need to adjust the main loop, to get packet (raw) only and not frames (there is no decoding in this case).

Regarding the vid-3.mp4, i don’t think i can share it without paying the author. But you find it on pexels, or similar videos like this: https://www.pexels.com/video/people-crossing-the-road-10697014/
and https://www.pexels.com/@georgemorina/gallery/

I have created ff-rknn-v4l2 to read from the camera. I am thinking to share the code on gitlab.
The only issue is the latency, 5s… a bit annoying…

Jagan · February 6, 2024, 6:04pm

Can you please share your FFmpeg configure steps. I tried to use ./configure --enable-libv4l2 --disable-v4l2-m2m

avaf · February 7, 2024, 11:17pm

Try this config:

Jagan · February 10, 2024, 1:39pm

Look like I have weird problems when i try to build ffmpeg in target.

$ ./configure --prefix=/usr --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --disable-libxvid --disable-libx264 --disable-libx265 --enable-rkmpp --enable-nonfree --enable-gpl --enable-version3 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-libxml2 --enable-librtmp --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --disable-shared --enable-decoder='aac,ac3,flac' --extra-cflags=-I/usr/src/linux-headers-5.10.110-15-rockchip/include --disable-cuvid 
ERROR: libdrm not found using pkg-config

even if install the apt libdrm-dev it is reproducing again as it look for libdrm in --extra-cflags=-I/usr/src/linux-headers-5.10.110-15-rockchip/include which is technically not there.

When i skop --extra-cflags option it is continuing and asking many packages like libfreetype-dev libmp3lame-dev libopus-dev libpulse-dev librtmp-dev libv4l-dev libvorbis-dev libxml2-dev

Do we have a right procedure to build the things in target for this FFmpeg. I’m using official bullseye_xfce_b39.img

Let me know I’m fully on to it.

avaf · February 10, 2024, 5:16pm

Because you enabled them, it asked for the packages. You should install or disable the packages (disabling them is not a good idea).

Two guys are actively working on FFmpeg, please ask them for the best parameters.
I don’t use the official image.

Obs:
You can suppress --extra-cflags

Jagan · February 15, 2024, 9:49am

Okay. I will check these options. Apart from ffmpeg, I did check your previous sdl2-cam where I can use the default rknn.

$ sdl2-cam 1 11 1 model/RK3588/yolov5s-640-640.rknn 640 640
post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Loading mode...
sdk version: 1.2.0 (1867aec5b@2022-01-14T15:16:40) driver version: 0.8.2
model input num: 1, output num: 3
resize_buf: 0x7f8945f010
texture_dst_buf: 0x7f8937d010
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Hot keys:
	ESC - quit the program
	p - pause video
Camera res: 640x480 [ 342 ]
rga_api version 1.7.3_[6]
loadLabelName ./model/coco_80_labels_list.txt

When I change the rknn that I built with rknn_model_zoo it seem failing after post processing. the same rknn was able to run with rknn_yolov5_demo

$ sdl2-cam 1 11 1 model/RK3588/yolov5.rknn 640 640
post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Loading mode...
terminate called after throwing an instance of 'jsonxx::json_parse_error'
  what():  unexpected token
Aborted

Is there any difference for rknn to use during stream vs running with input image? please share how to convert for stream?

avaf · February 17, 2024, 10:45pm

Looks like something wrong with your camera setup.

Jagan · February 19, 2024, 2:16pm

I don’t think so, I hardly not changing anything just use the another rknn which I built and tested from rknn_model_zoo

open@6tops:~/od$ ls
model  rknn_model_zoo  yolov5.rknn  yolov5s-640-640.rknn
open@6tops:~/od$ ls model/
coco_80_labels_list.txt
open@6tops:~/od$ sudo sdl2-cam 1 11 1 yolov5s-640-640.rknn 640 640
sudo: unable to resolve host 6tops: Name or service not known
post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Loading mode...
sdk version: 1.2.0 (1867aec5b@2022-01-14T15:16:40) driver version: 0.8.2
model input num: 1, output num: 3
resize_buf: 0x7f6c044010
texture_dst_buf: 0x7f6bf62010
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Hot keys:
	ESC - quit the program
	p - pause video
Camera res: 640x480 [ 342 ]
rga_api version 1.7.3_[6]
loadLabelName ./model/coco_80_labels_list.txt
Camera is closing...
Camera closed.
SDL cleanup...
SDL released!
resize_buf: 0x7f6c044010
texture_dst_buf: 0x7f6bf62010
free(texture_dst_buf)
free(resize_buf)
Buffers released and exit now, done!
open@6tops:~/od$ sudo sdl2-cam 1 11 1 yolov5.rknn 640 640
sudo: unable to resolve host 6tops: Name or service not known
post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Loading mode...
terminate called after throwing an instance of 'jsonxx::json_parse_error'
  what():  unexpected token
Aborted

avaf · February 19, 2024, 4:49pm

That code does not use json, so i don’t know where the error comes from.

Anyway, here is the code example of how to pull frames from the camera using only FFmpeg and SDL2.
There is an issue, high latency. 5s on average.

Possibly you need to re-write the main loop logic to fix it. (like using threads to pull the frames)

Requirements:

FFmpeg with MIPI support (NV12)
SDL2

ff-rknn-v4l2.c.zip (6.7 KB)

Jagan · February 20, 2024, 8:53am

zip missed include files

#include "postprocess.h"
#include "rknn_api.h"

avaf · February 20, 2024, 11:15am

Adjust the command line to build it inside the github project.

github.com

avafinger/ff-rknn/blob/main/postprocess.h

#ifndef _RKNN_ZERO_COPY_DEMO_POSTPROCESS_H_
#define _RKNN_ZERO_COPY_DEMO_POSTPROCESS_H_

#include <stdint.h>
#include <vector>

#define OBJ_NAME_MAX_SIZE 16
#define OBJ_NUMB_MAX_SIZE 64
#define OBJ_CLASS_NUM     80
#define NMS_THRESH        0.45
#define BOX_THRESH        0.25
#define PROP_BOX_SIZE     (5+OBJ_CLASS_NUM)

typedef struct _BOX_RECT
{
    int left;
    int right;
    int top;
    int bottom;
} BOX_RECT;

This file has been truncated. show original

github.com

avafinger/ff-rknn/blob/main/rknn_api.h

/****************************************************************************
*
*    Copyright (c) 2017 - 2022 by Rockchip Corp.  All rights reserved.
*
*    The material in this file is confidential and contains trade secrets
*    of Rockchip Corporation. This is proprietary information owned by
*    Rockchip Corporation. No part of this work may be disclosed,
*    reproduced, copied, transmitted, or used in any way for any purpose,
*    without the express written permission of Rockchip Corporation.
*
*****************************************************************************/


#ifndef _RKNN_API_H
#define _RKNN_API_H

#ifdef __cplusplus
extern "C" {
#endif

This file has been truncated. show original

These files depend on the RKNN SDK version.

Jagan · February 21, 2024, 1:39pm

Do you want to use this project again? or only these two files? and build instructions on the README is what we need build correct?

https://github.com/avafinger/ff-rknn/tree/main

avaf · February 24, 2024, 4:40pm

Save the file inside the project tree and the build is something like this:

g++ -O2 --permissive -o ff-rknn-v4l2 ff-rknn-v4l2.c postprocess.cc -D_FILE_OFFSET_BITS=64 -D REENTRANT -I./ -I/usr/include/drm -I/usr/include -I/usr/include/SDL2 -lSDL2 -lavformat -lavcodec -lavdevice -lavutil -lswscale -lswresample -lz -lm -lpthread -ldrm -lrockchip_mpp -llzma -lrtmp -lssl -lcrypto -lbz2 -lxml2 -lX11 -lxcb -lXv -lXext -lv4l2 -lrga -lva -lxcb-shm -lxcb-xfixes -lxcb-render -lxcb-shape -lxcb -lXau -lXdmcp -lxcb -lxcb-shape -lXmu -lXext -lXi -lv4l2 -lasound -lpulse -lsndio -lGL -lpulse -lGL -lGLESv2 -lsndio -lfreetype -lvorbis -lvorbisenc -lopus -lrknnrt -lmp3lame -L ./lib1.5

That depends on your environment…

Jagan · February 25, 2024, 1:54pm

Nothing is moving as expected, so I did try from scratch. My root has only X11, XFCE, libMali.

I have pushed all your reference files to repo

https://github.com/edgeble/ff-rknn

Installed FFMPEG (as per nyanmisaka/ffmpeg-rockchip)
https://github.com/edgeble/ff-rknn/blob/main/ffmpeg.BUID

open@6tops:~/ff-rknn$ ffmpeg -filters | grep rkrga
ffmpeg version 6b13265 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
 ... overlay_rkrga     VV->V      Rockchip RGA (2D Raster Graphic Acceleration) video compositor
 ... scale_rkrga       V->V       Rockchip RGA (2D Raster Graphic Acceleration) video resizer and format converter
 ... vpp_rkrga         V->V       Rockchip RGA (2D Raster Graphic Acceleration) video post-process (scale/crop/transpose)
open@6tops:~/ff-rknn$ ffmpeg -encoders | grep rkmppv
ffmpeg version 6b13265 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
open@6tops:~/ff-rknn$ ffmpeg -decoders | grep rkmppv
ffmpeg version 6b13265 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100

Installed ff-rknn-v4l2 with few changes from your files to satisfy build
https://github.com/edgeble/ff-rknn/commit/b4ffeb4b881dc842f504cd84ee4cd54625bd839a
Try to run with pixel file you pointed
https://www.pexels.com/video/people-crossing-the-road-10697014/

With mp4 file:

./ff-rknn-v4l2 -i /mnt/object.mp4 -x 960 -y 540 -l 0 -t 0 -m model/RK3588/yolov5s-640-640.rknn
Model: model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
Raw video support only!

With MIPI:

./ff-rknn-v4l2 -f v4l2 -p h264 -s 1920x1080 -i /dev/video11 -m model/RK3588/yolov5s-640-640.rknn -x 960 -y 540
Model: model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
[video4linux2,v4l2 @ 0xaaaad74b4070] Not a video capture device.
Cannot open input file '/dev/video11'

v4l2 on the same MIPI is working as expected.

$ v4l2-ctl -d /dev/video11 --set-fmt-video=width=1920,|  Cannot open /dev/ttyUSB0!  |stream-mmap=4 --stream-skip=100 --stream-to=11.yuv --stream-count=1 --stream-poll                                                                  
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps                               
<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<

Please let me know what is wrong, I’m on it.

avaf · February 25, 2024, 2:33pm

Link FFmpeg statically and it will work!

Jagan · February 25, 2024, 2:38pm

You mean, build the ffmpeg statically?