tops@ncm6b:~/ff-rknn$ sudo ./ff-rknn -f v4l2 -p h264 -s 1920x1080 -i /dev/video11 -m model/RK3588/yolov5s-640-640.rknn -x 960 -y 540
Model: model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
[video4linux2,v4l2 @ 0xaaaafb2ce070] Not a video capture device.
Cannot open input file '/dev/video11'
Log for mainpath video file, imx415
You need the FFmpeg with MIPI support, see my previous comment.
Yes I did build the FFmpeg with v4l2. - https://github.com/hbiyik/FFmpeg.git
$./configure --enable-libv4l2
$ ffmpeg -f v4l2 -pixel_format nv12 -framerate 30 -video_size 1920x1080 -i /dev/video11 out.h264
ffmpeg version 65f90322bf Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --enable-libv4l2
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
[video4linux2,v4l2 @ 0xaaaaeea58350] ioctl(VIDIOC_G_INPUT): Inappropriate ioctl for device
[video4linux2,v4l2 @ 0xaaaaeea58350] ioctl(VIDIOC_G_PARM): Inappropriate ioctl for device, using framerate 30/1
Input #0, video4linux2,v4l2, from '/dev/video11':
Duration: N/A, start: 3740.028140, bitrate: 829 kb/s
Stream #0:0: Video: rawvideo (NV12 / 0x3231564E), nv12, 1920x1080, 829 kb/s, 0.03 fps, 0.03 tbr, 1000k tbn
File 'out.h264' already exists. Overwrite? [y/N] y
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0xaaaaeea69770] Could not find a valid device
[h264_v4l2m2m @ 0xaaaaeea69770] can't configure encoder
[vost#0:0/h264_v4l2m2m @ 0xaaaaeea694a0] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!
This is an FFmpeg issue. Most likely you need to disable h264_v4l2m2m and a few others, note the codec used for MIPI will be RAW, so you need to adjust the main loop, to get packet (raw) only and not frames (there is no decoding in this case).
Regarding the vid-3.mp4, i don’t think i can share it without paying the author. But you find it on pexels, or similar videos like this: https://www.pexels.com/video/people-crossing-the-road-10697014/
and https://www.pexels.com/@georgemorina/gallery/
I have created ff-rknn-v4l2 to read from the camera. I am thinking to share the code on gitlab.
The only issue is the latency, 5s… a bit annoying…
Can you please share your FFmpeg configure steps. I tried to use ./configure --enable-libv4l2 --disable-v4l2-m2m
Try this config:
Look like I have weird problems when i try to build ffmpeg in target.
$ ./configure --prefix=/usr --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --disable-libxvid --disable-libx264 --disable-libx265 --enable-rkmpp --enable-nonfree --enable-gpl --enable-version3 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-libxml2 --enable-librtmp --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --disable-shared --enable-decoder='aac,ac3,flac' --extra-cflags=-I/usr/src/linux-headers-5.10.110-15-rockchip/include --disable-cuvid
ERROR: libdrm not found using pkg-config
even if install the apt libdrm-dev it is reproducing again as it look for libdrm in --extra-cflags=-I/usr/src/linux-headers-5.10.110-15-rockchip/include which is technically not there.
When i skop --extra-cflags option it is continuing and asking many packages like libfreetype-dev libmp3lame-dev libopus-dev libpulse-dev librtmp-dev libv4l-dev libvorbis-dev libxml2-dev
Do we have a right procedure to build the things in target for this FFmpeg. I’m using official bullseye_xfce_b39.img
Let me know I’m fully on to it.
Because you enabled them, it asked for the packages. You should install or disable the packages (disabling them is not a good idea).
Two guys are actively working on FFmpeg, please ask them for the best parameters.
I don’t use the official image.
Obs:
You can suppress --extra-cflags
Okay. I will check these options. Apart from ffmpeg, I did check your previous sdl2-cam where I can use the default rknn.
$ sdl2-cam 1 11 1 model/RK3588/yolov5s-640-640.rknn 640 640
post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Loading mode...
sdk version: 1.2.0 (1867aec5b@2022-01-14T15:16:40) driver version: 0.8.2
model input num: 1, output num: 3
resize_buf: 0x7f8945f010
texture_dst_buf: 0x7f8937d010
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Hot keys:
ESC - quit the program
p - pause video
Camera res: 640x480 [ 342 ]
rga_api version 1.7.3_[6]
loadLabelName ./model/coco_80_labels_list.txt
When I change the rknn that I built with rknn_model_zoo it seem failing after post processing. the same rknn was able to run with rknn_yolov5_demo
$ sdl2-cam 1 11 1 model/RK3588/yolov5.rknn 640 640
post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Loading mode...
terminate called after throwing an instance of 'jsonxx::json_parse_error'
what(): unexpected token
Aborted
Is there any difference for rknn to use during stream vs running with input image? please share how to convert for stream?
Looks like something wrong with your camera setup.
I don’t think so, I hardly not changing anything just use the another rknn which I built and tested from rknn_model_zoo
open@6tops:~/od$ ls
model rknn_model_zoo yolov5.rknn yolov5s-640-640.rknn
open@6tops:~/od$ ls model/
coco_80_labels_list.txt
open@6tops:~/od$ sudo sdl2-cam 1 11 1 yolov5s-640-640.rknn 640 640
sudo: unable to resolve host 6tops: Name or service not known
post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Loading mode...
sdk version: 1.2.0 (1867aec5b@2022-01-14T15:16:40) driver version: 0.8.2
model input num: 1, output num: 3
resize_buf: 0x7f6c044010
texture_dst_buf: 0x7f6bf62010
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Hot keys:
ESC - quit the program
p - pause video
Camera res: 640x480 [ 342 ]
rga_api version 1.7.3_[6]
loadLabelName ./model/coco_80_labels_list.txt
Camera is closing...
Camera closed.
SDL cleanup...
SDL released!
resize_buf: 0x7f6c044010
texture_dst_buf: 0x7f6bf62010
free(texture_dst_buf)
free(resize_buf)
Buffers released and exit now, done!
open@6tops:~/od$ sudo sdl2-cam 1 11 1 yolov5.rknn 640 640
sudo: unable to resolve host 6tops: Name or service not known
post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Loading mode...
terminate called after throwing an instance of 'jsonxx::json_parse_error'
what(): unexpected token
Aborted
That code does not use json, so i don’t know where the error comes from.
Anyway, here is the code example of how to pull frames from the camera using only FFmpeg and SDL2.
There is an issue, high latency. 5s on average.
Possibly you need to re-write the main loop logic to fix it. (like using threads to pull the frames)
Requirements:
ff-rknn-v4l2.c.zip (6.7 KB)
zip missed include files
#include "postprocess.h"
#include "rknn_api.h"
Adjust the command line to build it inside the github project.
These files depend on the RKNN SDK version.
Do you want to use this project again? or only these two files? and build instructions on the README is what we need build correct?
https://github.com/avafinger/ff-rknn/tree/main
Save the file inside the project tree and the build is something like this:
g++ -O2 --permissive -o ff-rknn-v4l2 ff-rknn-v4l2.c postprocess.cc -D_FILE_OFFSET_BITS=64 -D REENTRANT -I./ -I/usr/include/drm -I/usr/include -I/usr/include/SDL2 -lSDL2 -lavformat -lavcodec -lavdevice -lavutil -lswscale -lswresample -lz -lm -lpthread -ldrm -lrockchip_mpp -llzma -lrtmp -lssl -lcrypto -lbz2 -lxml2 -lX11 -lxcb -lXv -lXext -lv4l2 -lrga -lva -lxcb-shm -lxcb-xfixes -lxcb-render -lxcb-shape -lxcb -lXau -lXdmcp -lxcb -lxcb-shape -lXmu -lXext -lXi -lv4l2 -lasound -lpulse -lsndio -lGL -lpulse -lGL -lGLESv2 -lsndio -lfreetype -lvorbis -lvorbisenc -lopus -lrknnrt -lmp3lame -L ./lib1.5
That depends on your environment…
Nothing is moving as expected, so I did try from scratch. My root has only X11, XFCE, libMali.
I have pushed all your reference files to repo
https://github.com/edgeble/ff-rknn
open@6tops:~/ff-rknn$ ffmpeg -filters | grep rkrga
ffmpeg version 6b13265 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100
... overlay_rkrga VV->V Rockchip RGA (2D Raster Graphic Acceleration) video compositor
... scale_rkrga V->V Rockchip RGA (2D Raster Graphic Acceleration) video resizer and format converter
... vpp_rkrga V->V Rockchip RGA (2D Raster Graphic Acceleration) video post-process (scale/crop/transpose)
open@6tops:~/ff-rknn$ ffmpeg -encoders | grep rkmppv
ffmpeg version 6b13265 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100
open@6tops:~/ff-rknn$ ffmpeg -decoders | grep rkmppv
ffmpeg version 6b13265 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100
Installed ff-rknn-v4l2 with few changes from your files to satisfy build
https://github.com/edgeble/ff-rknn/commit/b4ffeb4b881dc842f504cd84ee4cd54625bd839a
Try to run with pixel file you pointed
https://www.pexels.com/video/people-crossing-the-road-10697014/
With mp4 file:
./ff-rknn-v4l2 -i /mnt/object.mp4 -x 960 -y 540 -l 0 -t 0 -m model/RK3588/yolov5s-640-640.rknn
Model: model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
Raw video support only!
With MIPI:
./ff-rknn-v4l2 -f v4l2 -p h264 -s 1920x1080 -i /dev/video11 -m model/RK3588/yolov5s-640-640.rknn -x 960 -y 540
Model: model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
[video4linux2,v4l2 @ 0xaaaad74b4070] Not a video capture device.
Cannot open input file '/dev/video11'
v4l2 on the same MIPI is working as expected.
$ v4l2-ctl -d /dev/video11 --set-fmt-video=width=1920,| Cannot open /dev/ttyUSB0! |stream-mmap=4 --stream-skip=100 --stream-to=11.yuv --stream-count=1 --stream-poll
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<
Please let me know what is wrong, I’m on it.
Link FFmpeg statically and it will work!
You mean, build the ffmpeg statically?
Yes, build it statically. Did you check if the MIPI patch is applied to your FFmpeg?
And remember you will need re-write the way to pull the frames to get rid of the annoying latency.