Yes, build it statically. Did you check if the MIPI patch is applied to your FFmpeg?
And remember you will need re-write the way to pull the frames to get rid of the annoying latency.
IMX415 + NPU demo on ROCK 5B
I will try to decode the mp4 first, look like ffmpeg don’t have --enable-static option to build it static
The lack of --enable-shared means it will build a static library.
Note:
./ff-rknn-v4l2 is only for the CSI MIPI Device, not for decoding mp4!
Make sure you can record a video like this:
./ffmpeg -f v4l2 -framerate 30 -video_size 640x480 -i /dev/video11 output.mkv
What will this command do, it decode vid-3.mp4 and detect objects isn’t it?
./ff-rknn -i …/…/videos_rknn/vid-3.mp4 -x 960 -y 540 -l 0 -t 0 -m ./model/RK3588/yolov5s-640-640.rknn
This command will do that.
This one you showed will NOT:
./ff-rknn-v4l2 -i /mnt/object.mp4 -x 960 -y 540 -l 0 -t 0 -m model/RK3588/yolov5s-640-640.rknn
Still show dynamic, does it require rkmpp, rkrga need to build static as well?
./configure --prefix=/usr --enable-gpl --enable-version3 --enable-rkmpp --enable-rkrga --enable-shared --enable-libdrm
file ffmpeg
ffmpeg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=84c1df373224617621d6fee9a588f5b4d6a
b48a4, for GNU/Linux 3.7.0, stripped
I am on 3W right now, take this as an example:
You must link with:
./libavformat/libavformat.a
./libavutil/libavutil.a
./libavdevice/libavdevice.a
./libpostproc/libpostproc.a
./libavfilter/libavfilter.a
./libswscale/libswscale.a
./libswresample/libswresample.a
./libavcodec/libavcodec.a
I don’t know how you will do it on openedev…
are you taking about, ff-rknn repo in my openedev?
I did used your config you mentioned before in the same thread as well.
--prefix=/usr --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --disable-libxvid --disable-libx264 --disable-libx265 --enable-rkmpp --enable-nonfree --enable-gpl --enable-version3 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-libxml2 --enable-librtmp --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --disable-shared --enable-decoder='aac,ac3,flac' --disable-cuvid
Would you please share your image I will try to test and recheck the libraries. Please share the image that has ff-rknn.
Please give me some time to create the sdcard img. I will see what i can do.
ff-rknn revisited. A single thread, cpu/gpu/npu ondemand governor.
The File:
- Build command line:
g++ -O3 --permissive -o ff-rknn ff-rknn.c postprocess.cc -D_FILE_OFFSET_BITS=64 -D REENTRANT -I./ -I/usr/include/drm -I/usr/include -I/usr/include/SDL2 -lSDL2 -lSDL2_ttf -lz -lm -lpthread -ldrm -lrockchip_mpp -llzma -lrtmp -lssl -lcrypto -lbz2 -lxml2 -lX11 -lxcb -lXv -lXext -lv4l2 -lrga -lva -lxcb-shm -lxcb-xfixes -lxcb-render -lxcb-shape -lxcb -lXau -lXdmcp -lxcb -lxcb-shape -lXmu -lXext -lXi -lv4l2 -lasound -lpulse -lsndio -lGL -lpulse -lGL -lGLESv2 -lsndio -lfreetype -lvorbis -lvorbisenc -lopus -lrknnrt -lmp3lame -L ./lib1.5 -lavutil -lavcodec -lavformat -lavdevice -lavfilter -lswscale -lswresample -lpostproc
-
The project is in the ~/rockchip/npu/ff-rknn path.
-
Decoding a mp4 file, FullHD (1920x1080) resized to 960x580 and positioned at 0,0 (weston):
rock@rock5b:~/rockchip/npu/ff-rknn$ ./ff-rknn -i 2180394928.mp4 -x 960 -y 580 -l 0 -t 0 -m ./model/RK3588/yolov5s-640-640.rknn
Model: ./model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
INFO: SDL: compiled with=2.0.20 linked against=2.0.20
arm_release_ver of this libmali is ‘g6p0-01eac0’, rk_so_ver is ‘6’.
[h264_rkmpp @ 0x557dd4efc0] Decoder noticed an info change (1920x1080), format=0
rga_api version 1.9.3_[0]
loadLabelName ./model/coco_80_labels_list.txt
INFO: Program quit after 5630 ticks
Avg FPS: 23.0
- Decoding an RTSP camera example:
rock@rock5b:~/rockchip/npu/ff-rknn$ ./ff-rknn -f rtsp -i rtsp://192.168.254.205:554/user=admin_password=kiFDlXTz_channel=1_stream=1.sdp?real_stream -m ./model/RK3588/yolov5s-640-640.rknn
Model: ./model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
INFO: SDL: compiled with=2.0.20 linked against=2.0.20
arm_release_ver of this libmali is ‘g6p0-01eac0’, rk_so_ver is ‘6’.
[h264_rkmpp @ 0x55b403a0f0] Decoder noticed an info change (640x360), format=0
rga_api version 1.9.3_[0]
loadLabelName ./model/coco_80_labels_list.txt
INFO: Program quit after 34221 ticks
Avg FPS: 12.3
SD card Image with weston/Wayland with Ubuntu Minimum:
sb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
OS Image is a bare minimum, specific to my tablet-like needs. HDMI works, the sound needs some workout, Wifi/bt disabled. For educational purposes only, use as is.
sdl2 port attached, SD card to be finished and hosted somewhere…
sdl2_port.patch.zip (1.6 KB)
For HDMI output, change the line in the file /boot/extlinux/extlinux.conf:
DEFAULT kernel-5.10.110-rk3588-v4l2_TABLET
to
DEFAULT kernel-5.10.110-rk3588-v4l2
Thanks. Can I get the sdcard image seems like it is missing on your thread.
Attached is the fix for the v4l2 NV12 raw, so it’s possible to process the image from the camera.
governor: performance
./ff-rknn -f v4l2 -p nv12 -s 1920x1080 -i /dev/video11 -x 1920 -y 1080 -b 28 -a 40 -m ./model/RK3588/yolov5s-640-640.rknn
Model: ./model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
[video4linux2,v4l2 @ 0x55609b8b90] ioctl(VIDIOC_G_INPUT): Inappropriate ioctl for device
[video4linux2,v4l2 @ 0x55609b8b90] ioctl(VIDIOC_G_PARM): Inappropriate ioctl for device
[video4linux2,v4l2 @ 0x55609b8b90] Time per frame unknown
[video4linux2,v4l2 @ 0x55609b8b90] Stream #0: not enough frames to estimate rate; consider increasing probesize
NV12 Raw video support only!
INFO: SDL: compiled with=2.0.20 linked against=2.0.20
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '6'.
rga_api version 1.9.3_[0]
loadLabelName ./model/coco_80_labels_list.txt
INFO: Program quit after 189726 ticks
Avg FPS: 30.1
Apply the patch over the original code from github.
check your mail for the link and the missing instructions.
v4l2_sdl2_fix.patch.zip (3.7 KB)
I checked the e-mail it seem the thread is not updating any information. This is my e-mail jagan@edgeble.ai
I meant forum mail. Check your e-mail again, there are two links to mega.nz , see if it is not blocked or in spam. It’s all i can do.
Tips for testing:
The image has turbo_on and turbo_off grabbed from here:
([Guide] Best Options for a Single Board Computer 🎮)
Worth mentioning:
The camera has HFLIP and VFLIP properties on the dtb, it was prepared for TABLET mode (the front camera), with no mirror, change it to suit your need:
Thanks for your time and e-mail. Please replied to e-mail, I did tried couple of times to dd the weston_minimum seems like p2 is not able to mount.
The annoying 5 second latency has been fixed (raw video). FFmpeg buffered some frames to gather stream info, discarding that is the fix. No more latency. I’m working to encode the frames with FFmpeg, but I ran into a problem.
The frames are NV12 1920x1080, hstride=1920, and vstride=1080 but the encoder expects vstride=1088, i think.
The encoded result (hevc) looks like this:
@boogiepop, @nyanmisaka
Any suggestion to fix that?
