IMX415 + NPU demo on ROCK 5B

[avaf]

There is a little gem here:

packages/arm64 · linux-5.10-gen-rkr1 · RK3588_Linux / linux / debian · GitLab

There you find all pre-built deb packages if you don’t want to spend time building/compiling and fixing all stuff.
Start with a minimal image that does not have any broken dependencies and then install the required packages. I think rkr1 was meant for kernel 5.10.66 ( ~ 72 - patches) and rkr4 for 5.10.110. But rkr1 will work on 5.10.110 with some minor features missing.

Care must be taken by the order you install the packages. And don’t make the mistake to install all at once and then reboot, install one package, reboot and check, repeat…

Tip:

1. librga
2. libmpp
3. xserver (X11)
4. etc…

[phiber]

yeah, like i said nothing that’s installable and usable out of the box.
thanks for the link though

[incognito]

Can you explain what this is for?

[avaf]

Not sure i understand the question, i will try to answer and be brief.

Those are deb built by Rockchip members to get most of the board working features, like HW video acceleration, 3D, 2D, and encoder, and decoder working as expected on their BSP/SDK.

If you don’t want to spend hours, weeks, or months figuring out how to build it yourselves you just install the required deb to make gstreamer work. In order not to have dependencies errors like the above, you start with a clean image and then install what you need.

You can use a ready-to-use image if you don’t want those hassles. At some point, you will need to get your hands dirty to fix things that broke when you update to the latest kernel version.

[Ozayr_Moorad]

GitHub - Qengineering/Rock-5-Ubuntu-22-image: Rock Pi 5 image with Ubuntu 22,...

Just want to leave this here might be of help

[avaf]

i have extended the demo to be able to play the streaming inference (kind of) in real time from a remote server (rtmp).

As the video demo shows, the objects detected are pushed to the rtmp server and can be played with ffplay, rtmp or http protocols.

Screencast made with FFmpeg, 3840x1080, Xcfe and video demo is available here:
https://mega.nz/file/pHJWXAaK#-jRlbgicsAoNbTOlMDkA73vm272Jc_vSHSSsc_wPYgg

Eventually, if times permits i will show how it was made here:

Object detection with NPU - h264 streams - h264 camera - rtsp

I played a bit more with ff-rknn on X11 and added some parameters to detect a single object. DISPLAY=:0.0 ./ff-rknn -i ~/weston/apps/videos_rknn/vid-13.mp4 -x 960 -y 540 -l 960 -t 0 -m ./model/RK3588/yolov5s-640-640.rknn -b 40 -o bird -a 60 Where -a = confidence, and -b = alpha-blend mask [birds_3] Some benchmarks showed still reasonable to display 12 streams and record a screencast at the same time. I tried to use the NPU SRAM to reduce DDR bandwidth described here: (https://github.com/rock…

[phiber]

For Frigate and rk3588 there’s a repo with a branch specific to OrangePi5 and rk3588.

After compiling the Docker image

make arm64

you should config your frigate detectors.
This branch comes with two rk3588-specific frigate detectors: “armnn” which is the ARM-optimized code and rknn which is the rk3588 onboard NPU. armnn works for basic detection and one 4k camera, while the rknn detector is a hit-and-miss right now.

[arifoni]

Thank you for sharing this interesting data. What type of neural network are your using for inference? YOLOV8/variants or something different?

[avaf]

yolov5s (rockchip)

[Jagan]

@avaf can you share the build steps for creating sdl2-cam, I’m trying to enhance SDL2 features to support more resolution streams.

[avaf]

Hi @Jagan , not sure i understood the question.

Will you work on the imx415 kernel driver to support more resolutions?
The SDL2 i used for the demo is the distro version, the old version 2.0.20 with some hacks to work with libmali (i think panfrost does not need it) and to render it on screen (opengles2 or kms/drm), it renders as texture.

I called it sdl2-cam cause it does not use opencv to render it, only to grab the frames.
No 2FA for me on github… (yet)… But i can point to the code it was based on if needed.

[avaf]

Build:
g++ -g -O2 cam.cpp postprocess.cc -o sdl2-cam -I/usr/local/include/SDL2 -I../include -I../../.././runtime/Linux/librknn_api/include/ -I/usr/include -I/usr/include/opencv4 -D REENTRANT -lSDL2 -lGLESv2 -lpthread -ldl -lm -lopencv_highgui -lopencv_videoio -lopencv_imgproc -lopencv_core -lrga -L/home/rock/camera/rknpu2/examples/rknn_cam2/ -lrknnrt

SDL2 texture: https://github.com/hbiyik/FFmpeg/blob/master/fftools/ffplay.c#L834

[Jagan]

Thanks for the details, I’m trying to stream 1980p, 4k - H.254 resolutions with i.MX415 sensor.

[Jagan]

Would you please share the exact source code, where can i find cam.cpp and do we need SDL2 and rknpu2 to build sdl2-cam?

[avaf]

You don’t need rknpu2 and sdl2 to stream video, for that use gstreamer.
Instructions here:

ROCK 5B Camera streaming with RTSP / RTMP

Here are the basic instructions to stream a video from the imx415 camera using rtsp protocol. For the instructions, i am using Debian 11.5 with X11, but it really does not matter which distro. ROCK 5B IP: 192.168.254.228 (our server) Client IP: 92.168.254.200 (our client where we display the camera content) pre-requisites rkmpp encoding capabilities (hw encode enabled) imx415 camera gstreamer with the rkmpp plugin For learning purposes, i have built the gstreamer rtsp server. Checking gs…

If you want to use NPU and stream the result, you can use the code from here:
(https://github.com/avafinger/ff-rknn)

and use this FFmpeg that has imx415 support, I am not sure if it was updated or not:
The fix:

Encoding raw NV12 from MIPI camera

There is a small bug in the mipi support when releasing the buffers, but it is usable.

I don’t know which kernel version you use, mainline perhaps?

[Jagan]

Apart from SDL, FFmpeg (./configure && make) do we need any other dependencies related to rga. I do have libv4l-rkmpp=1.4.0-1 already.

I’m encounter this rga error
ff-rknn.c:66:10: fatal error: rga/RgaApi.h: No such file or directory
66 | #include <rga/RgaApi.h>

I did build the SDL, FFmpeg as mentioned in INSTALL.md no explicit config change.

Yes, I’m trying on 5.10 and idea is to use Mainline next if it is working.

[avaf]

ff-rknn.c:66:10: fatal error: rga/RgaApi.h: No such file or directory
66 | #include <rga/RgaApi.h>

You need librga-dev.

ii librga-dev 2.2.0-1 arm64 Userspace interface to Rockchip RGA 2D accelerator
ii librga2 2.2.0-1 arm64 Userspace interface to Rockchip RGA 2D accelerator

or point to where your include file is stored, in my case:

/usr/include/rga/RgaApi.h

Note: you need to adjust FFmpeg to grab NV12 (not DRMPRIME) and convert NV12 to RGB888 using virtual buffer addr, not dma-buf.

Good luck!

[Jagan]

ff-rknn.c: In function ‘int main(int, char**)’:
ff-rknn.c:712:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
  712 |         ifmt = av_find_input_format("video4linux2");
      |                ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
      |                                    |
      |                                    const AVInputFormat*
ff-rknn.c:727:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
  727 |         ifmt = av_find_input_format("flv");
      |                ~~~~~~~~~~~~~~~~~~~~^~~~~~~
      |                                    |
      |                                    const AVInputFormat*
ff-rknn.c:752:70: warning: invalid conversion from ‘AVCodec**’ to ‘const AVCodec**’ [-fpermissive]
  752 |  av_find_best_stream(input_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &codec, 0);
      |                                                             ^~~~~~
      |                                                             |
      |                                                             AVCodec**

In file included from /usr/local/include/libavdevice/avdevice.h:57,
                 from ff-rknn.c:61:
/usr/local/include/libavformat/avformat.h:2079:41: note:   initializing argument 5 of ‘int av_find_best_stream(AVFormatContext*, AVMediaType, int, int, const AVCodec**, int)’
 2079 |                         const AVCodec **decoder_ret,
      |                         ~~~~~~~~~~~~~~~~^~~~~~~~~~~
ff-rknn.c:776:33: warning: invalid conversion from ‘const AVCodec*’ to ‘AVCodec*’ [-fpermissive]
  776 |     codec = avcodec_find_decoder(codecpar->codec_id);
      |             ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
      |                                 |
      |                                 const AVCodec*
ff-rknn.c:835:18: error: ‘SDL_INIT_EVERYTHING’ was not declared in this scope; did you mean ‘SDL_INIT_EVENTS’?
  835 |     if (SDL_Init(SDL_INIT_EVERYTHING) < 0) {
      |                  ^~~~~~~~~~~~~~~~~~~
      |                  SDL_INIT_EVENTS

Look like SDL config issue, I didn’t find SDL_INIT_EVERYTHING in my fs. any clue?

[Jagan]

One change I made was to install libsdl2-dev and update header path as below, not sure it impacts.

diff --git a/ff-rknn.c b/ff-rknn.c
index 7267ce8..37baf81 100644
--- a/ff-rknn.c
+++ b/ff-rknn.c
@@ -37,9 +37,9 @@
  */
 
 #include "SDL3/SDL.h"
-#include "SDL_syswm.h"
+#include "SDL2/SDL_syswm.h"
 
-#include <drm_fourcc.h>
+#include <libdrm/drm_fourcc.h>
 #include <errno.h>
 #include <fcntl.h>
 #include <getopt.h>

[avaf]

Hmm, looks like in my setup i mixed up sdl2 and sdl3 headers. I had both sdl2 and sdl3 installed.