@avaf can you share the build steps for creating sdl2-cam, I’m trying to enhance SDL2 features to support more resolution streams.
IMX415 + NPU demo on ROCK 5B
Hi @Jagan , not sure i understood the question.
Will you work on the imx415 kernel driver to support more resolutions?
The SDL2 i used for the demo is the distro version, the old version 2.0.20 with some hacks to work with libmali (i think panfrost does not need it) and to render it on screen (opengles2 or kms/drm), it renders as texture.
I called it sdl2-cam cause it does not use opencv to render it, only to grab the frames.
No 2FA for me on github… (yet)… But i can point to the code it was based on if needed.
Build:
g++ -g -O2 cam.cpp postprocess.cc -o sdl2-cam -I/usr/local/include/SDL2 -I../include -I../../.././runtime/Linux/librknn_api/include/ -I/usr/include -I/usr/include/opencv4 -D REENTRANT -lSDL2 -lGLESv2 -lpthread -ldl -lm -lopencv_highgui -lopencv_videoio -lopencv_imgproc -lopencv_core -lrga -L/home/rock/camera/rknpu2/examples/rknn_cam2/ -lrknnrt
SDL2 texture: https://github.com/hbiyik/FFmpeg/blob/master/fftools/ffplay.c#L834
Thanks for the details, I’m trying to stream 1980p, 4k - H.254 resolutions with i.MX415 sensor.
Would you please share the exact source code, where can i find cam.cpp and do we need SDL2 and rknpu2 to build sdl2-cam?
You don’t need rknpu2 and sdl2 to stream video, for that use gstreamer.
Instructions here:
If you want to use NPU and stream the result, you can use the code from here:
(https://github.com/avafinger/ff-rknn)
and use this FFmpeg that has imx415 support, I am not sure if it was updated or not:
The fix:
There is a small bug in the mipi support when releasing the buffers, but it is usable.
I don’t know which kernel version you use, mainline perhaps?
Apart from SDL, FFmpeg (./configure && make) do we need any other dependencies related to rga. I do have libv4l-rkmpp=1.4.0-1 already.
I’m encounter this rga error
ff-rknn.c:66:10: fatal error: rga/RgaApi.h: No such file or directory
66 | #include <rga/RgaApi.h>
I did build the SDL, FFmpeg as mentioned in INSTALL.md no explicit config change.
Yes, I’m trying on 5.10 and idea is to use Mainline next if it is working.
You need librga-dev.
ii librga-dev 2.2.0-1 arm64 Userspace interface to Rockchip RGA 2D accelerator
ii librga2 2.2.0-1 arm64 Userspace interface to Rockchip RGA 2D accelerator
or point to where your include file is stored, in my case:
/usr/include/rga/RgaApi.h
Note: you need to adjust FFmpeg to grab NV12 (not DRMPRIME) and convert NV12 to RGB888 using virtual buffer addr, not dma-buf.
Good luck!
ff-rknn.c: In function ‘int main(int, char**)’:
ff-rknn.c:712:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
712 | ifmt = av_find_input_format("video4linux2");
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
| |
| const AVInputFormat*
ff-rknn.c:727:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
727 | ifmt = av_find_input_format("flv");
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~
| |
| const AVInputFormat*
ff-rknn.c:752:70: warning: invalid conversion from ‘AVCodec**’ to ‘const AVCodec**’ [-fpermissive]
752 | av_find_best_stream(input_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &codec, 0);
| ^~~~~~
| |
| AVCodec**
In file included from /usr/local/include/libavdevice/avdevice.h:57,
from ff-rknn.c:61:
/usr/local/include/libavformat/avformat.h:2079:41: note: initializing argument 5 of ‘int av_find_best_stream(AVFormatContext*, AVMediaType, int, int, const AVCodec**, int)’
2079 | const AVCodec **decoder_ret,
| ~~~~~~~~~~~~~~~~^~~~~~~~~~~
ff-rknn.c:776:33: warning: invalid conversion from ‘const AVCodec*’ to ‘AVCodec*’ [-fpermissive]
776 | codec = avcodec_find_decoder(codecpar->codec_id);
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
| |
| const AVCodec*
ff-rknn.c:835:18: error: ‘SDL_INIT_EVERYTHING’ was not declared in this scope; did you mean ‘SDL_INIT_EVENTS’?
835 | if (SDL_Init(SDL_INIT_EVERYTHING) < 0) {
| ^~~~~~~~~~~~~~~~~~~
| SDL_INIT_EVENTS
Look like SDL config issue, I didn’t find SDL_INIT_EVERYTHING in my fs. any clue?
One change I made was to install libsdl2-dev and update header path as below, not sure it impacts.
diff --git a/ff-rknn.c b/ff-rknn.c
index 7267ce8..37baf81 100644
--- a/ff-rknn.c
+++ b/ff-rknn.c
@@ -37,9 +37,9 @@
*/
#include "SDL3/SDL.h"
-#include "SDL_syswm.h"
+#include "SDL2/SDL_syswm.h"
-#include <drm_fourcc.h>
+#include <libdrm/drm_fourcc.h>
#include <errno.h>
#include <fcntl.h>
#include <getopt.h>
Hmm, looks like in my setup i mixed up sdl2 and sdl3 headers. I had both sdl2 and sdl3 installed.
+++ b/ff-rknn.c
@@ -37,9 +37,9 @@
*/
#include "SDL3/SDL.h"
-#include "SDL_syswm.h"
+#include "SDL2/SDL_syswm.h"
-#include <drm_fourcc.h>
+#include <libdrm/drm_fourcc.h>
#include <errno.h>
#include <fcntl.h>
#include <getopt.h>
@@ -73,6 +73,8 @@ extern "C" {
#include "postprocess.h"
#include "rknn_api.h"
+#define SDL_INIT_EVERYTHING ~0U
+
#define ALIGN(x, a) ((x) + (a - 1)) & (~(a - 1))
#define DRM_ALIGN(val, align) ((val + (align - 1)) & ~(align - 1))
Look like we need some explicit lib’s to export
# g++ -O2 --permissive -o ff-rknn ff-rknn.c postprocess.cc -I/usr/include/libdrm -I/usr/include -D_FILE_OFFSET_BITS=64 -D REENTRANT `pkg-config --cflags --libs sdl3` -lz -lm -lpthread -ldrm -lrockchip_mpp -lrga -lvorbis -lvorbisenc -ltiff -lopus -logg -lmp3lame -llzma -lrtmp -lssl -lcrypto -lbz2 -lxml2 -lX11 -lxcb -lXv -lXext -lv4l2 -lasound -lpulse -lGL -lGLESv2 -lsndio -lfreetype -lxcb -lxcb-shm -lxcb -lxcb-xfixes -lxcb-render -lxcb-shape -lxcb -lxcb-shape -lxcb -lavutil -lavcodec -lavformat -lavdevice -lavfilter -lswscale -lswresample -lpostproc -lrknnrt
ff-rknn.c: In function ‘int main(int, char**)’:
ff-rknn.c:714:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
714 | ifmt = av_find_input_format("video4linux2");
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
| |
| const AVInputFormat*
ff-rknn.c:729:36: warning: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
729 | ifmt = av_find_input_format("flv");
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~
| |
| const AVInputFormat*
ff-rknn.c:754:70: warning: invalid conversion from ‘AVCodec**’ to ‘const AVCodec**’ [-fpermissive]
754 | av_find_best_stream(input_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &codec, 0);
| ^~~~~~
| |
| AVCodec**
In file included from /usr/local/include/libavdevice/avdevice.h:57,
from ff-rknn.c:61:
/usr/local/include/libavformat/avformat.h:2079:41: note: initializing argument 5 of ‘int av_find_best_stream(AVFormatContext*, AVMediaType, int, int, const AVCodec**, int)’
2079 | const AVCodec **decoder_ret,
| ~~~~~~~~~~~~~~~~^~~~~~~~~~~
ff-rknn.c:778:33: warning: invalid conversion from ‘const AVCodec*’ to ‘AVCodec*’ [-fpermissive]
778 | codec = avcodec_find_decoder(codecpar->codec_id);
| ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
| |
| const AVCodec*
/usr/bin/ld: cannot find -lvorbis
/usr/bin/ld: cannot find -lvorbisenc
/usr/bin/ld: cannot find -ltiff
/usr/bin/ld: cannot find -lopus
/usr/bin/ld: cannot find -logg
/usr/bin/ld: cannot find -lmp3lame
/usr/bin/ld: cannot find -llzma
/usr/bin/ld: cannot find -lrtmp
/usr/bin/ld: cannot find -lssl
/usr/bin/ld: cannot find -lcrypto
/usr/bin/ld: cannot find -lbz2
/usr/bin/ld: cannot find -lxml2
/usr/bin/ld: cannot find -lfreetype
/usr/bin/ld: cannot find -lxcb-shm
/usr/bin/ld: cannot find -lxcb-xfixes
/usr/bin/ld: cannot find -lxcb-render
/usr/bin/ld: cannot find -lxcb-shape
/usr/bin/ld: cannot find -lxcb-shape
/usr/bin/ld: cannot find -lpostproc
/usr/bin/ld: cannot find -lrknnrt
collect2: error: ld returned 1 exit status
Ah, now I remember why I linked SDL2 and SDL3, FFmpeg uses SDL2!
You’re better off just using SDL2, changing a few things and sticking with SDL2, and you’ll be good to go.
Made necessary changes, and able to built ff-rknn. Look like imx415 (MIPI-based) not possible to test isn’t it? Tried rkisp_mainpath video11 but it is not working. Can you share vid-3.mp4 file for testing? let me know the change for mipi camera.
tops@ncm6b:~/ff-rknn$ sudo ./ff-rknn -f v4l2 -p h264 -s 1920x1080 -i /dev/video11 -m model/RK3588/yolov5s-640-640.rknn -x 960 -y 540
Model: model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
[video4linux2,v4l2 @ 0xaaaafb2ce070] Not a video capture device.
Cannot open input file '/dev/video11'
Log for mainpath video file, imx415
You need the FFmpeg with MIPI support, see my previous comment.
Yes I did build the FFmpeg with v4l2. - https://github.com/hbiyik/FFmpeg.git
$./configure --enable-libv4l2
$ ffmpeg -f v4l2 -pixel_format nv12 -framerate 30 -video_size 1920x1080 -i /dev/video11 out.h264
ffmpeg version 65f90322bf Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --enable-libv4l2
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
[video4linux2,v4l2 @ 0xaaaaeea58350] ioctl(VIDIOC_G_INPUT): Inappropriate ioctl for device
[video4linux2,v4l2 @ 0xaaaaeea58350] ioctl(VIDIOC_G_PARM): Inappropriate ioctl for device, using framerate 30/1
Input #0, video4linux2,v4l2, from '/dev/video11':
Duration: N/A, start: 3740.028140, bitrate: 829 kb/s
Stream #0:0: Video: rawvideo (NV12 / 0x3231564E), nv12, 1920x1080, 829 kb/s, 0.03 fps, 0.03 tbr, 1000k tbn
File 'out.h264' already exists. Overwrite? [y/N] y
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0xaaaaeea69770] Could not find a valid device
[h264_v4l2m2m @ 0xaaaaeea69770] can't configure encoder
[vost#0:0/h264_v4l2m2m @ 0xaaaaeea694a0] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!
This is an FFmpeg issue. Most likely you need to disable h264_v4l2m2m and a few others, note the codec used for MIPI will be RAW, so you need to adjust the main loop, to get packet (raw) only and not frames (there is no decoding in this case).
Regarding the vid-3.mp4, i don’t think i can share it without paying the author. But you find it on pexels, or similar videos like this: https://www.pexels.com/video/people-crossing-the-road-10697014/
and https://www.pexels.com/@georgemorina/gallery/
I have created ff-rknn-v4l2 to read from the camera. I am thinking to share the code on gitlab.
The only issue is the latency, 5s… a bit annoying…
Can you please share your FFmpeg configure steps. I tried to use ./configure --enable-libv4l2 --disable-v4l2-m2m
Try this config: