Can you explain what this is for?
IMX415 + NPU demo on ROCK 5B
Not sure i understand the question, i will try to answer and be brief.
Those are deb built by Rockchip members to get most of the board working features, like HW video acceleration, 3D, 2D, and encoder, and decoder working as expected on their BSP/SDK.
If you don’t want to spend hours, weeks, or months figuring out how to build it yourselves you just install the required deb to make gstreamer work. In order not to have dependencies errors like the above, you start with a clean image and then install what you need.
You can use a ready-to-use image if you don’t want those hassles. At some point, you will need to get your hands dirty to fix things that broke when you update to the latest kernel version.
2 Likes
Just want to leave this here might be of help
i have extended the demo to be able to play the streaming inference (kind of) in real time from a remote server (rtmp).
As the video demo shows, the objects detected are pushed to the rtmp server and can be played with ffplay, rtmp or http protocols.
Screencast made with FFmpeg, 3840x1080, Xcfe and video demo is available here:
https://mega.nz/file/pHJWXAaK#-jRlbgicsAoNbTOlMDkA73vm272Jc_vSHSSsc_wPYgg
Eventually, if times permits i will show how it was made here:
2 Likes
For Frigate and rk3588 there’s a repo with a branch specific to OrangePi5 and rk3588.
After compiling the Docker image
make arm64
you should config your frigate detectors.
This branch comes with two rk3588-specific frigate detectors: “armnn” which is the ARM-optimized code and rknn which is the rk3588 onboard NPU. armnn works for basic detection and one 4k camera, while the rknn detector is a hit-and-miss right now.
1 Like
Thank you for sharing this interesting data. What type of neural network are your using for inference? YOLOV8/variants or something different?
yolov5s (rockchip)
@avaf can you share the build steps for creating sdl2-cam, I’m trying to enhance SDL2 features to support more resolution streams.
Hi @Jagan , not sure i understood the question.
Will you work on the imx415 kernel driver to support more resolutions?
The SDL2 i used for the demo is the distro version, the old version 2.0.20 with some hacks to work with libmali (i think panfrost does not need it) and to render it on screen (opengles2 or kms/drm), it renders as texture.
I called it sdl2-cam cause it does not use opencv to render it, only to grab the frames.
No 2FA for me on github… (yet)… But i can point to the code it was based on if needed.
Build:
g++ -g -O2 cam.cpp postprocess.cc -o sdl2-cam -I/usr/local/include/SDL2 -I../include -I../../.././runtime/Linux/librknn_api/include/ -I/usr/include -I/usr/include/opencv4 -D REENTRANT -lSDL2 -lGLESv2 -lpthread -ldl -lm -lopencv_highgui -lopencv_videoio -lopencv_imgproc -lopencv_core -lrga -L/home/rock/camera/rknpu2/examples/rknn_cam2/ -lrknnrt
SDL2 texture: https://github.com/hbiyik/FFmpeg/blob/master/fftools/ffplay.c#L834
Thanks for the details, I’m trying to stream 1980p, 4k - H.254 resolutions with i.MX415 sensor.
Would you please share the exact source code, where can i find cam.cpp and do we need SDL2 and rknpu2 to build sdl2-cam?
You don’t need rknpu2 and sdl2 to stream video, for that use gstreamer.
Instructions here:
If you want to use NPU and stream the result, you can use the code from here:
(https://github.com/avafinger/ff-rknn)
and use this FFmpeg that has imx415 support, I am not sure if it was updated or not:
The fix:
There is a small bug in the mipi support when releasing the buffers, but it is usable.
I don’t know which kernel version you use, mainline perhaps?
Apart from SDL, FFmpeg (./configure && make) do we need any other dependencies related to rga. I do have libv4l-rkmpp=1.4.0-1 already.
I’m encounter this rga error
ff-rknn.c:66:10: fatal error: rga/RgaApi.h: No such file or directory
66 | #include <rga/RgaApi.h>
I did build the SDL, FFmpeg as mentioned in INSTALL.md no explicit config change.
Yes, I’m trying on 5.10 and idea is to use Mainline next if it is working.
You need librga-dev.
ii librga-dev 2.2.0-1 arm64 Userspace interface to Rockchip RGA 2D accelerator
ii librga2 2.2.0-1 arm64 Userspace interface to Rockchip RGA 2D accelerator
or point to where your include file is stored, in my case:
/usr/include/rga/RgaApi.h
Note: you need to adjust FFmpeg to grab NV12 (not DRMPRIME) and convert NV12 to RGB888 using virtual buffer addr, not dma-buf.
Good luck!
ff-rknn.c: In function ‘int main(int, char**)’:
ff-rknn.c:712:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
712 | ifmt = av_find_input_format("video4linux2");
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
| |
| const AVInputFormat*
ff-rknn.c:727:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
727 | ifmt = av_find_input_format("flv");
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~
| |
| const AVInputFormat*
ff-rknn.c:752:70: warning: invalid conversion from ‘AVCodec**’ to ‘const AVCodec**’ [-fpermissive]
752 | av_find_best_stream(input_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &codec, 0);
| ^~~~~~
| |
| AVCodec**
In file included from /usr/local/include/libavdevice/avdevice.h:57,
from ff-rknn.c:61:
/usr/local/include/libavformat/avformat.h:2079:41: note: initializing argument 5 of ‘int av_find_best_stream(AVFormatContext*, AVMediaType, int, int, const AVCodec**, int)’
2079 | const AVCodec **decoder_ret,
| ~~~~~~~~~~~~~~~~^~~~~~~~~~~
ff-rknn.c:776:33: warning: invalid conversion from ‘const AVCodec*’ to ‘AVCodec*’ [-fpermissive]
776 | codec = avcodec_find_decoder(codecpar->codec_id);
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
| |
| const AVCodec*
ff-rknn.c:835:18: error: ‘SDL_INIT_EVERYTHING’ was not declared in this scope; did you mean ‘SDL_INIT_EVENTS’?
835 | if (SDL_Init(SDL_INIT_EVERYTHING) < 0) {
| ^~~~~~~~~~~~~~~~~~~
| SDL_INIT_EVENTS
Look like SDL config issue, I didn’t find SDL_INIT_EVERYTHING in my fs. any clue?
One change I made was to install libsdl2-dev and update header path as below, not sure it impacts.
diff --git a/ff-rknn.c b/ff-rknn.c
index 7267ce8..37baf81 100644
--- a/ff-rknn.c
+++ b/ff-rknn.c
@@ -37,9 +37,9 @@
*/
#include "SDL3/SDL.h"
-#include "SDL_syswm.h"
+#include "SDL2/SDL_syswm.h"
-#include <drm_fourcc.h>
+#include <libdrm/drm_fourcc.h>
#include <errno.h>
#include <fcntl.h>
#include <getopt.h>
Hmm, looks like in my setup i mixed up sdl2 and sdl3 headers. I had both sdl2 and sdl3 installed.
+++ b/ff-rknn.c
@@ -37,9 +37,9 @@
*/
#include "SDL3/SDL.h"
-#include "SDL_syswm.h"
+#include "SDL2/SDL_syswm.h"
-#include <drm_fourcc.h>
+#include <libdrm/drm_fourcc.h>
#include <errno.h>
#include <fcntl.h>
#include <getopt.h>
@@ -73,6 +73,8 @@ extern "C" {
#include "postprocess.h"
#include "rknn_api.h"
+#define SDL_INIT_EVERYTHING ~0U
+
#define ALIGN(x, a) ((x) + (a - 1)) & (~(a - 1))
#define DRM_ALIGN(val, align) ((val + (align - 1)) & ~(align - 1))
Look like we need some explicit lib’s to export
# g++ -O2 --permissive -o ff-rknn ff-rknn.c postprocess.cc -I/usr/include/libdrm -I/usr/include -D_FILE_OFFSET_BITS=64 -D REENTRANT `pkg-config --cflags --libs sdl3` -lz -lm -lpthread -ldrm -lrockchip_mpp -lrga -lvorbis -lvorbisenc -ltiff -lopus -logg -lmp3lame -llzma -lrtmp -lssl -lcrypto -lbz2 -lxml2 -lX11 -lxcb -lXv -lXext -lv4l2 -lasound -lpulse -lGL -lGLESv2 -lsndio -lfreetype -lxcb -lxcb-shm -lxcb -lxcb-xfixes -lxcb-render -lxcb-shape -lxcb -lxcb-shape -lxcb -lavutil -lavcodec -lavformat -lavdevice -lavfilter -lswscale -lswresample -lpostproc -lrknnrt
ff-rknn.c: In function ‘int main(int, char**)’:
ff-rknn.c:714:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
714 | ifmt = av_find_input_format("video4linux2");
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
| |
| const AVInputFormat*
ff-rknn.c:729:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
729 | ifmt = av_find_input_format("flv");
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~
| |
| const AVInputFormat*
ff-rknn.c:754:70: warning: invalid conversion from ‘AVCodec**’ to ‘const AVCodec**’ [-fpermissive]
754 | av_find_best_stream(input_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &codec, 0);
| ^~~~~~
| |
| AVCodec**
In file included from /usr/local/include/libavdevice/avdevice.h:57,
from ff-rknn.c:61:
/usr/local/include/libavformat/avformat.h:2079:41: note: initializing argument 5 of ‘int av_find_best_stream(AVFormatContext*, AVMediaType, int, int, const AVCodec**, int)’
2079 | const AVCodec **decoder_ret,
| ~~~~~~~~~~~~~~~~^~~~~~~~~~~
ff-rknn.c:778:33: warning: invalid conversion from ‘const AVCodec*’ to ‘AVCodec*’ [-fpermissive]
778 | codec = avcodec_find_decoder(codecpar->codec_id);
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
| |
| const AVCodec*
/usr/bin/ld: cannot find -lvorbis
/usr/bin/ld: cannot find -lvorbisenc
/usr/bin/ld: cannot find -ltiff
/usr/bin/ld: cannot find -lopus
/usr/bin/ld: cannot find -logg
/usr/bin/ld: cannot find -lmp3lame
/usr/bin/ld: cannot find -llzma
/usr/bin/ld: cannot find -lrtmp
/usr/bin/ld: cannot find -lssl
/usr/bin/ld: cannot find -lcrypto
/usr/bin/ld: cannot find -lbz2
/usr/bin/ld: cannot find -lxml2
/usr/bin/ld: cannot find -lfreetype
/usr/bin/ld: cannot find -lxcb-shm
/usr/bin/ld: cannot find -lxcb-xfixes
/usr/bin/ld: cannot find -lxcb-render
/usr/bin/ld: cannot find -lxcb-shape
/usr/bin/ld: cannot find -lxcb-shape
/usr/bin/ld: cannot find -lpostproc
/usr/bin/ld: cannot find -lrknnrt
collect2: error: ld returned 1 exit status
Ah, now I remember why I linked SDL2 and SDL3, FFmpeg uses SDL2!
You’re better off just using SDL2, changing a few things and sticking with SDL2, and you’ll be good to go.