IMX415 + NPU demo on ROCK 5B

Jagan · February 25, 2024, 4:20pm

are you taking about, ff-rknn repo in my openedev?

I did used your config you mentioned before in the same thread as well.

--prefix=/usr --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --disable-libxvid --disable-libx264 --disable-libx265 --enable-rkmpp --enable-nonfree --enable-gpl --enable-version3 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-libxml2 --enable-librtmp --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --disable-shared --enable-decoder='aac,ac3,flac' --disable-cuvid

Jagan · February 28, 2024, 6:46pm

Would you please share your image I will try to test and recheck the libraries. Please share the image that has ff-rknn.

avaf · February 28, 2024, 7:37pm

Please give me some time to create the sdcard img. I will see what i can do.

avaf · February 29, 2024, 10:37pm

ff-rknn revisited. A single thread, cpu/gpu/npu ondemand governor.

The File:

curl -O -J -L https://vod-progressive.akamaized.net/exp=1709328217~acl=%2Fvimeo-prod-skyfire-std-us%2F01%2F2323%2F19%2F486618242%2F2180394928.mp4~hmac=d6d1453f95f315050a666830c2010cfe9c4a3dd5db84913ec862147f585ba29d/vimeo-prod-skyfire-std-us/01/2323/19/486618242/2180394928.mp4?download=1&filename=pexels-george-morina-6059506+(1080p).mp4

Build command line:

g++ -O3 --permissive -o ff-rknn ff-rknn.c postprocess.cc -D_FILE_OFFSET_BITS=64 -D REENTRANT -I./ -I/usr/include/drm -I/usr/include -I/usr/include/SDL2 -lSDL2 -lSDL2_ttf -lz -lm -lpthread -ldrm -lrockchip_mpp -llzma -lrtmp -lssl -lcrypto -lbz2 -lxml2 -lX11 -lxcb -lXv -lXext -lv4l2 -lrga -lva -lxcb-shm -lxcb-xfixes -lxcb-render -lxcb-shape -lxcb -lXau -lXdmcp -lxcb -lxcb-shape -lXmu -lXext -lXi -lv4l2 -lasound -lpulse -lsndio -lGL -lpulse -lGL -lGLESv2 -lsndio -lfreetype -lvorbis -lvorbisenc -lopus -lrknnrt -lmp3lame -L ./lib1.5 -lavutil -lavcodec -lavformat -lavdevice -lavfilter -lswscale -lswresample -lpostproc

The project is in the ~/rockchip/npu/ff-rknn path.
Decoding a mp4 file, FullHD (1920x1080) resized to 960x580 and positioned at 0,0 (weston):

rock@rock5b:~/rockchip/npu/ff-rknn$ ./ff-rknn -i 2180394928.mp4 -x 960 -y 580 -l 0 -t 0 -m ./model/RK3588/yolov5s-640-640.rknn
Model: ./model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
INFO: SDL: compiled with=2.0.20 linked against=2.0.20
arm_release_ver of this libmali is ‘g6p0-01eac0’, rk_so_ver is ‘6’.
[h264_rkmpp @ 0x557dd4efc0] Decoder noticed an info change (1920x1080), format=0
rga_api version 1.9.3_[0]
loadLabelName ./model/coco_80_labels_list.txt
INFO: Program quit after 5630 ticks
Avg FPS: 23.0

Decoding an RTSP camera example:

rock@rock5b:~/rockchip/npu/ff-rknn$ ./ff-rknn -f rtsp -i rtsp://192.168.254.205:554/user=admin_password=kiFDlXTz_channel=1_stream=1.sdp?real_stream -m ./model/RK3588/yolov5s-640-640.rknn
Model: ./model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
INFO: SDL: compiled with=2.0.20 linked against=2.0.20
arm_release_ver of this libmali is ‘g6p0-01eac0’, rk_so_ver is ‘6’.
[h264_rkmpp @ 0x55b403a0f0] Decoder noticed an info change (640x360), format=0
rga_api version 1.9.3_[0]
loadLabelName ./model/coco_80_labels_list.txt
INFO: Program quit after 34221 ticks
Avg FPS: 12.3

SD card Image with weston/Wayland with Ubuntu Minimum:

sb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

OS Image is a bare minimum, specific to my tablet-like needs. HDMI works, the sound needs some workout, Wifi/bt disabled. For educational purposes only, use as is.

sdl2 port attached, SD card to be finished and hosted somewhere…

sdl2_port.patch.zip (1.6 KB)

For HDMI output, change the line in the file /boot/extlinux/extlinux.conf:
DEFAULT kernel-5.10.110-rk3588-v4l2_TABLET
to
DEFAULT kernel-5.10.110-rk3588-v4l2

Jagan · March 3, 2024, 11:42am

Thanks. Can I get the sdcard image seems like it is missing on your thread.

avaf · March 3, 2024, 1:17pm

Attached is the fix for the v4l2 NV12 raw, so it’s possible to process the image from the camera.

governor: performance

./ff-rknn -f v4l2 -p nv12 -s 1920x1080 -i /dev/video11 -x 1920 -y 1080 -b 28 -a 40 -m ./model/RK3588/yolov5s-640-640.rknn
Model: ./model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
[video4linux2,v4l2 @ 0x55609b8b90] ioctl(VIDIOC_G_INPUT): Inappropriate ioctl for device
[video4linux2,v4l2 @ 0x55609b8b90] ioctl(VIDIOC_G_PARM): Inappropriate ioctl for device
[video4linux2,v4l2 @ 0x55609b8b90] Time per frame unknown
[video4linux2,v4l2 @ 0x55609b8b90] Stream #0: not enough frames to estimate rate; consider increasing probesize
NV12 Raw video support only!
INFO: SDL: compiled with=2.0.20 linked against=2.0.20
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '6'.
rga_api version 1.9.3_[0]
loadLabelName ./model/coco_80_labels_list.txt
INFO: Program quit after 189726 ticks
Avg FPS: 30.1

Apply the patch over the original code from github.

check your mail for the link and the missing instructions.
v4l2_sdl2_fix.patch.zip (3.7 KB)

avaf · March 3, 2024, 1:25pm

Hey Jagan, there are two Jagan Teki, @Jagan and @Jagan_Teki, who is the original?

Jagan · March 3, 2024, 1:30pm

Both seems mine only. Will try to remove @Jagan_Teki

Jagan · March 3, 2024, 1:33pm

I checked the e-mail it seem the thread is not updating any information. This is my e-mail jagan@edgeble.ai

avaf · March 3, 2024, 2:29pm

I meant forum mail. Check your e-mail again, there are two links to mega.nz , see if it is not blocked or in spam. It’s all i can do.

Tips for testing:

The image has turbo_on and turbo_off grabbed from here:
([Guide] Best Options for a Single Board Computer 🎮)

Worth mentioning:

The camera has HFLIP and VFLIP properties on the dtb, it was prepared for TABLET mode (the front camera), with no mirror, change it to suit your need:

Jagan · March 3, 2024, 3:10pm

Thanks for your time and e-mail. Please replied to e-mail, I did tried couple of times to dd the weston_minimum seems like p2 is not able to mount.

avaf · March 18, 2024, 1:50pm

The annoying 5 second latency has been fixed (raw video). FFmpeg buffered some frames to gather stream info, discarding that is the fix. No more latency. I’m working to encode the frames with FFmpeg, but I ran into a problem.

The frames are NV12 1920x1080, hstride=1920, and vstride=1080 but the encoder expects vstride=1088, i think.

The encoded result (hevc) looks like this:

@boogiepop, @nyanmisaka
Any suggestion to fix that?

nyanmisaka · March 18, 2024, 7:24pm

The encoder handled the drm frame (hs=1080) from rkrga filter without a problem.

avaf · March 18, 2024, 9:32pm

I see.

In the camera case, the src is hs=1080 and dst hs=1080.
The grabbed frame size = 1920 x 1080 * 3 / 2 (AV_PIX_FMT_NV12) and the destination frame size to encode is the same.

I will test a 1920 x 1080 fake buffer and a 1920 x 1088 (AV_PIX_FMT_YUV420P) and see what i get.
Thanks anyway.

avaf · March 18, 2024, 11:16pm

The tests confirmed the dst frame buffer that will be encoded must have hs=1088.
the frame size is 1920 x 1088 * 3 / 2 for NV12.

nyanmisaka · March 19, 2024, 6:02am

github.com

rockchip-linux/mpp/blob/develop/doc/Rockchip_Developer_Guide_MPP_EN.md#361-widthand-height-of-input-image-and-stride

# MPP Development Reference

| Project： | MPP         |
| --------- | ----------- |
| Version： | 0.7         |
| Author：  | Herman Chen |
| Date：    | 10/17/2023  |

| **Revision** | **Date**   | **Description**                                              | **Author**                              |
| ------------ | ---------- | ------------------------------------------------------------ | --------------------------------------- |
| 0.1          | 04/18/2018 | Initial version                                              | Herman Chen                             |
| 0.2          | 05/07/2018 | Add decoder control command description, encoder part description and demo part description | Herman Chen                             |
| 0.3          | 05/22/2018 | Fix some clerical errors and explanation errors, rearrange page numbers | Herman Chen <br/>Xiongbin Xie(精英智通) |
|              | 07/08/2019 | Translation                                                  | Lily Chen                               |
| 0.4          | 11/28/2018 | 1、Updated the memory layout instructions of the encoder input image.<br/>2、Correct the encoder flowchart error | Herman Chen<br>Vyagoo                   |
| 0.5          | 06/08/2020 | Update encoder new configuration interface, no longer supports RK3188 | Herman Chen                             |
| 0.6          | 06/11/2020 | Translation                                                  | Lily Chen                               |
| 0.7          | 10/17/2023 | add markdown document                                        | Xueman Ruan<br>Yandong Lin              |
|              |            |                                                              |                                         |
|              |            |                                                              |                                         |

This file has been truncated. show original

According to the MPP docs you seem to need to copy the planes to satisfy the padding requirements.

Vinicius · March 19, 2024, 3:58pm

hey anyone using the npu on dietpi? I’m not sure it supports hardware acceleration
i have run LLM inference on cpu and the performance was great but am wondering if i need to change OS to use the npu

avaf · March 19, 2024, 9:09pm

I thought RGA would handle this padding, but no.

I converted NV12 (no padding, the frame grabbed from the camera) to RGB24, and that worked fine during encoding.

But i think there is a penalty during encoding, RGB24 will be converted again, right?

avaf · March 20, 2024, 12:45am

What’s the equivalent for this command in ffmpeg?

mpi_enc_test -i /dev/video11 -f 0 -w 1920 -h 1080 -o out_1920x1080.h265 -t 16777220 -n 500 -g 6:6 -rc 1 -bps 2000000 -fps 30

I tried this one but got different results:

ffmpeg -f v4l2 -framerate 30 -video_size 1920x1080 -i /dev/video11 -c:v hevc_rkmpp -b:v 2M -rc_mode 1 -g 6 -vframes 500 -y out_1920x1080.mkv

nyanmisaka · March 20, 2024, 4:44am

ffmpeg -init_hw_device rkmpp -f v4l2 -framerate 30 -video_size 1920x1080 -i /dev/video11 \
-vf hwupload,scale_rkrga=w=1920:h=1080:format=rgb24 -c:v hevc_rkmpp -b:v 2M -rc_mode 1 -g 6 -vframes 500 -y out_1920x1080.mkv