Update 2:
Added support for:
- rtsp (h264)
- rtmp (h264 / flv)
- http (h264 / flv)
- v4l2 (h264)
Unfortunately, i haven’t found a way to fix latency on rtmp and http, usually 5 secs.
Update 2:
Added support for:
Unfortunately, i haven’t found a way to fix latency on rtmp and http, usually 5 secs.
I tested it with X11. I had to remove some optimizations due to license issues, but still fine with X11. I am just facing some issues with raw h264 coming from cameras, rkdev2 keeps resetting from time to time with kernel 5.10.110 if using dma32 on 4GB, it does not happen with kernel 5.10.66. Looks like one will need to remove that on kernel 5.10.110 if the board has 4GB. But i think the code is ready to be published.
mpp_rkvdec2 fdc38100.rkvdec-core: resetting...
mpp_rkvdec2 fdc38200.rkvdec-core: resetting...
I fixed it here, and now I am able to render NV12 10-bit, NV15/NV16, so basically, HEVC and H264 are working here in my version.
BTW, @boogiepop, you should review (when possible) h264_rkmpp_decoder, it get stuck on a second session.
what do you mean by the session?
If you run it the first time, decoding is OK. Run the second time and it gets stuck. I think it puts the board in a somewhat unstable state, i couldn’t access the directory i was working on and after a reboot had to give permission to my user to access the directory tree (the one i was working…). I haven’t looked at the code but i ‘think’ some resetting or releasing is missing.
Update:
Perhaps is some video stream format causing this. Tried a few ones now and it seems OK.
rock@rock5b:/apps/rockchip/ff-rknn-new$ DISPLAY=:0.0 ./ff-rknn -i /apps/Videos/Poltergeist_DOM_TrailerD-1080p-HDTN.mp4 -x 960 -y 540 -l 0 -t 0 -m ./model/RK3588/yolov5s-640-640.rknn -b 80 -o person -a 50~
Model: ./model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
[h264_rkmpp_decoder @ 0x555c49eba0] Picture format is drm_prime.
INFO: SDL: compiled with=3.0.0 linked against=3.0.0
xcb_connection_has_error() returned true
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '6'.
[h264_rkmpp_decoder @ 0x555c49eba0] Decoder is set to DRM Prime with format NV16.
[h264_rkmpp_decoder @ 0x555c49eba0] Decoder noticed an info change
rga_api version 1.8.1_[5]
loadLabelName ./model/coco_80_labels_list.txt
Avg FPS: 25.8
rock@rock5b:/apps/rockchip/ff-rknn-new$ DISPLAY=:0.0 ./ff-rknn -i /apps/Videos/Poltergeist_DOM_TrailerD-1080p-HDTN.mp4 -x 960 -y 540 -l 0 -t 0 -m ./model/RK3588/yolov5s-640-640.rknn -b 80 -o person -a 50
Model: ./model/RK3588/yolov5s-640-640.rknn - size: 8502208.
sdk version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20) driver version: 0.8.2
model input num: 1, output num: 3
model: 640x640x3
[h264_rkmpp_decoder @ 0x558974fba0] Picture format is drm_prime.
INFO: SDL: compiled with=3.0.0 linked against=3.0.0
xcb_connection_has_error() returned true
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '6'.
[h264_rkmpp_decoder @ 0x558974fba0] Decoder is set to DRM Prime with format NV16.
[h264_rkmpp_decoder @ 0x558974fba0] Decoder noticed an info change
rga_api version 1.8.1_[5]
loadLabelName ./model/coco_80_labels_list.txt
Avg FPS: 25.1
When can I use this with frigate for object detection? And what are the requirements for this to run?
frigate is python/docker, ff-rknn is C/C++ for performance. rknn with python is available long ago, i don’t know why there was no interest in integrating it into frigate. Most of the AI people here in the forum are python experts, i am not.
(https://github.com/avafinger/ff-rknn#system-requirements)
Note: Some pixel format breaks RGA3 if the board has more than 4GB of RAM, but it has been fixed, waiting to be pushed.
FFmpeg is getting ready for HW Encoding, it is not hard to add the encoder to it and make it a simple NVR, but for a real NVR it needs multithreading and some optimizations to be re-written.
I played a bit more with ff-rknn on X11 and added some parameters to detect a single object.
DISPLAY=:0.0 ./ff-rknn -i ~/weston/apps/videos_rknn/vid-13.mp4 -x 960 -y 540 -l 960 -t 0 -m ./model/RK3588/yolov5s-640-640.rknn -b 40 -o bird -a 60
Where -a = confidence, and -b = alpha-blend mask
Some benchmarks showed still reasonable to display 12 streams and record a screencast at the same time. I tried to use the NPU SRAM to reduce DDR bandwidth described here: (https://github.com/rockchip-linux/rknpu2/blob/master/doc/RK3588_NPU_SRAM_usage.md) and ROCK 5B Debug Party Invitation but the results were not good.
screencast (3840x1080):
https://mega.nz/file/MfBE3QAB#DmSDu9bc83havKszngPDLzS3RiEX8CnIh2nIPY1EyQ8
Here i push the results to an RTMP server and play back in real-time with ffplay.
Video here: 3840x1080
https://mega.nz/file/5HpXRb7R#JSabqmlTK-XFkfuIskGocyWnqGO_ObEjzqMt0dbVg2M
Demo explained:
The code is really useful, but recent SDL commits ( https://discourse.libsdl.org/t/sdl-sdl-syswm-h-has-been-removed-and-replaced-with-window-properties/47246 ) break the code.
Some older SDL commits ( https://discourse.libsdl.org/t/sdl-sdl3-with-hardware-acceleration-for-rockchip-platform-mali-blob/41325 ) also lack some other required functions
Maybe SDL3 is still too unstable. In any case, can you suggest a specific SDL3 commit where this code compiles and works? I will try some other commits in-between these 2
I think SDL3 api is changing a lot.
Last commit i have:
a94d724f17d7236e9307a02ec946997aa192778e
The hardware acceleration for Rockchip is still applied on the latest SDL3 commit, but i think if you have Panfrost enabled this is really not necessary, it was meant for libmali and should be harmless.
Hello, thanks again for your code and your help
I have a couple of questions about the code:
I see it only supports H264. Does it not support Raw NV12? Is there some technical limitation? My camera stream is Raw NV12.
For hardware acceleration. Is it good enough to just have Open GL ES support during compile, or do I also need SDL_Rockchip to show up as ON ?
For some reason I am not able to get SDL_Rockchip appear as enabled during compile, even though I have full hardware acceleration on X11 and EGLFS on my board. I have to also say I am using RK3566 and I am compiling everything from Buildroot Cross Compiler, maybe I am missing a flag or something?
Thanks again for giving the community such a nice code!
Not a limitation, a feature. It use DRMPRIME and dma-buf. It should be changed drom DRMPRIME to NV12 here:
and modify drm_rga_buf to use virtual buffer instead of dma-buf, FFmpeg return vbuffer in NV12 format:
You probably using Panfrost.
dmesg|grep -i panfrost
Note: There some minor changes you should pay attention, like:
Hello, thanks a lot
I am not using Panfrost, I am using libmali, but I might have had some old files pending clean-up along the way, I really experimented a lot when trying to get GPU acceleration. I will do a make clean and try again.
Hello, I tried make clean, and did make again, still no SDL_Rockchip.
I did “dmesg | grep -i panfrost” and nothing showed up
This is my configure log on SDL3:
– Performing Test HAVE_SA_SIGACTION
– Performing Test HAVE_SA_SIGACTION - Success
– Looking for dlopen
– Looking for dlopen - found
– Performing Test HAVE_O_CLOEXEC
– Performing Test HAVE_O_CLOEXEC - Success
– Performing Test HAVE_OSS_SYS_SOUNDCARD_H
– Performing Test HAVE_OSS_SYS_SOUNDCARD_H - Success
– Found ALSA: /home/avi/Buildroot-2/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/libasound.so (found version “1.2.7.2”)
– dynamic libasound -> libasound.so.2
– Checking for module ‘jack’
– Package ‘jack’, required by ‘virtual:world’, not found
– Checking for module ‘libpipewire-0.3>=0.3.20’
– Package ‘libpipewire-0.3’, required by ‘virtual:world’, not found
– Checking for module ‘libpulse>=5.0’
– Found libpulse, version 16.1
– dynamic libpulse -> libpulse.so.0
– Checking for module ‘sndio’
– Package ‘sndio’, required by ‘virtual:world’, not found
– Found X11: /home/avi/Buildroot-2/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/include
– Looking for XOpenDisplay in /home/avi/Buildroot-2/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/libX11.so;/home/avi/Buildroot-2/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/libXext.so
– Looking for XOpenDisplay in /home/avi/Buildroot-2/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/libX11.so;/home/avi/Buildroot-2/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/libXext.so - found
– Looking for gethostbyname
– Looking for gethostbyname - found
– Looking for connect
– Looking for connect - found
– Looking for remove
– Looking for remove - found
– Looking for shmat
– Looking for shmat - found
– Looking for IceConnectionNumber in ICE
– Looking for IceConnectionNumber in ICE - found
– dynamic libX11 -> libX11.so.6
– dynamic libXext -> libXext.so.6
– dynamic libXcursor -> libXcursor.so.1
– dynamic libXi -> libXi.so.6
– dynamic libXfixes -> libXfixes.so.3
– dynamic libXrandr -> libXrandr.so.2
– dynamic libXrender -> libXrender.so.1
– dynamic libXss -> libXss.so.1
– Looking for shmat
– Looking for shmat - found
– Performing Test HAVE_XGENERICEVENT
– Performing Test HAVE_XGENERICEVENT - Success
– Looking for XkbKeycodeToKeysym
– Looking for XkbKeycodeToKeysym - found
– Performing Test HAVE_XINPUT2_MULTITOUCH
– Performing Test HAVE_XINPUT2_MULTITOUCH - Success
– Performing Test HAVE_XFIXES_H
– Performing Test HAVE_XFIXES_H - Success
– Could NOT find OpenGL (missing: OPENGL_opengl_LIBRARY OPENGL_glx_LIBRARY OPENGL_INCLUDE_DIR)
– Performing Test HAVE_OPENGL_EGL
– Performing Test HAVE_OPENGL_EGL - Success
– Checking for module ‘libdrm’
– Found libdrm, version 2.4.112
– Checking for module ‘gbm’
– Found gbm, version 10.4.0
– dynamic libdrm -> libdrm.so.2
– dynamic libgbm -> libmali-bifrost-g31-rxp0-wayland-gbm.so
– Performing Test HAVE_OPENGL_GLX
– Performing Test HAVE_OPENGL_GLX - Failed
– Performing Test HAVE_OPENGL
– Performing Test HAVE_OPENGL - Failed
– Performing Test HAVE_OPENGLES_V1
– Performing Test HAVE_OPENGLES_V1 - Success
– Performing Test HAVE_OPENGLES_V2
– Performing Test HAVE_OPENGLES_V2 - Success
– Checking for modules ‘wayland-client>=1.18;wayland-egl;wayland-cursor;egl;xkbcommon>=0.5.0’
– Found wayland-client, version 1.20.0
– Found wayland-egl, version 7.10
– Found wayland-cursor, version 1.20.0
– Found egl, version 7.10
– Found xkbcommon, version 1.4.0
– dynamic libwayland-client -> libwayland-client.so.0
– dynamic libwayland-egl -> libmali-bifrost-g31-rxp0-wayland-gbm.so
– dynamic libwayland-cursor -> libwayland-cursor.so.0
– dynamic libxkbcommon -> libxkbcommon.so.0
– Checking for module ‘libdecor-0’
– Package ‘libdecor-0’, required by ‘virtual:world’, not found
– Performing Test HAVE_INPUT_EVENTS
– Performing Test HAVE_INPUT_EVENTS - Success
– Performing Test HAVE_INPUT_KD
– Performing Test HAVE_INPUT_KD - Success
– Looking for libudev.h
– Looking for libudev.h - found
– dynamic libudev -> libudev.so.1
– Looking for sys/inotify.h
– Looking for sys/inotify.h - found
– Looking for inotify_init
– Looking for inotify_init - found
– Looking for inotify_init1
– Looking for inotify_init1 - found
– Checking for one of the modules ‘dbus-1;dbus’
– Checking for one of the modules ‘ibus-1.0;ibus’
– Performing Test LIBC_HAS_WORKING_LIBUNWIND
– Performing Test LIBC_HAS_WORKING_LIBUNWIND - Failed
– Performing Test LIBUNWIND_HAS_WORKINGLIBUNWIND
– Performing Test LIBUNWIND_HAS_WORKINGLIBUNWIND - Failed
– Checking for modules ‘libunwind;libunwind-generic’
– Package ‘libunwind’, required by ‘virtual:world’, not found
– Package ‘libunwind-generic’, required by ‘virtual:world’, not found
– Checking for module ‘libusb-1.0’
– Package ‘libusb-1.0’, required by ‘virtual:world’, not found
– Performing Test HAVE_PTHREADS
– Performing Test HAVE_PTHREADS - Success
– Performing Test HAVE_RECURSIVE_MUTEXES
– Performing Test HAVE_RECURSIVE_MUTEXES - Success
– Performing Test HAVE_PTHREADS_SEM
– Performing Test HAVE_PTHREADS_SEM - Success
– Performing Test COMPILER_HAS_SEM_TIMEDWAIT
– Performing Test COMPILER_HAS_SEM_TIMEDWAIT - Success
– Looking for pthread.h
– Looking for pthread.h - found
– Looking for pthread_np.h
– Looking for pthread_np.h - not found
– Performing Test HAVE_PTHREAD_SETNAME_NP
– Performing Test HAVE_PTHREAD_SETNAME_NP - Success
– Looking for clock_gettime in c
– Looking for clock_gettime in c - found
– Performing Test HAVE_ENABLE_NEW_DTAGS
– Performing Test HAVE_ENABLE_NEW_DTAGS - Success
– Found Perl: /usr/bin/perl (found version “5.26.1”)
–
– SDL3 was configured with the following options:
–
– Platform: Linux
– 64-bit: TRUE
– Compiler: /home/avi/Buildroot-2/buildroot/output/host/bin/aarch64-buildroot-linux-gnu-gcc
– Revision: SDL-3.0.0-no-vcs
– Vendor:
–
– Subsystems:
– Atomic: ON
– Audio: ON
– Video: ON
– Render: ON
– Events: ON
– Joystick: ON
– Haptic: ON
– Hidapi: ON
– Power: ON
– Threads: ON
– Timers: ON
– File: ON
– Loadso: ON
– CPUinfo: ON
– Filesystem: ON
– Sensor: ON
– Locale: ON
– Misc: ON
–
– Options:
– SDL_ALSA (Wanted: ON): ON
– SDL_ALSA_SHARED (Wanted: ON): ON
– SDL_ALTIVEC (Wanted: OFF): OFF
– SDL_ARMNEON (Wanted: ON): ON
– SDL_ARMNEON_BLITTERS (Wanted: OFF): OFF
– SDL_ARMSIMD (Wanted: OFF): OFF
– SDL_ASAN (Wanted: OFF): OFF
– SDL_ASSEMBLY (Wanted: ON): ON
– SDL_ASSERTIONS (Wanted: auto): auto
– SDL_AVX (Wanted: OFF): OFF
– SDL_AVX2 (Wanted: OFF): OFF
– SDL_AVX512F (Wanted: OFF): OFF
– SDL_BACKGROUNDING_SIGNAL (Wanted: OFF): OFF
– SDL_CCACHE (Wanted: OFF): OFF
– SDL_CLANG_TIDY (Wanted: OFF): OFF
– SDL_CLOCK_GETTIME (Wanted: ON): ON
– SDL_COCOA (Wanted: OFF): OFF
– SDL_DBUS (Wanted: ON): ON
– SDL_DIRECTX (Wanted: OFF): OFF
– SDL_DISABLE_INSTALL (Wanted: OFF): OFF
– SDL_DISABLE_UNINSTALL (Wanted: OFF): OFF
– SDL_DISKAUDIO (Wanted: ON): ON
– SDL_DUMMYAUDIO (Wanted: ON): ON
– SDL_DUMMYVIDEO (Wanted: ON): ON
– SDL_FOREGROUNDING_SIGNAL (Wanted: OFF): OFF
– SDL_GCC_ATOMICS (Wanted: ON): ON
– SDL_HIDAPI (Wanted: ON): ON
– SDL_HIDAPI_JOYSTICK (Wanted: ON): ON
– SDL_HIDAPI_LIBUSB (Wanted: ON): OFF
– SDL_HIDAPI_LIBUSB_SHARED (Wanted: ON): OFF
– SDL_IBUS (Wanted: ON): OFF
– SDL_INSTALL_TESTS (Wanted: OFF): OFF
– SDL_JACK (Wanted: ON): OFF
– SDL_JACK_SHARED (Wanted: ON): OFF
– SDL_KMSDRM (Wanted: ON): ON
– SDL_KMSDRM_SHARED (Wanted: ON): ON
– SDL_LASX (Wanted: OFF): OFF
– SDL_LIBC (Wanted: ON): ON
– SDL_LIBUDEV (Wanted: ON): ON
– SDL_LSX (Wanted: OFF): OFF
– SDL_METAL (Wanted: OFF): OFF
– SDL_MMX (Wanted: OFF): OFF
– SDL_OFFSCREEN (Wanted: ON): ON
– SDL_OPENGL (Wanted: ON): OFF
– SDL_OPENGLES (Wanted: ON): ON
– SDL_OSS (Wanted: ON): ON
– SDL_PIPEWIRE (Wanted: ON): OFF
– SDL_PIPEWIRE_SHARED (Wanted: ON): OFF
– SDL_PTHREADS (Wanted: ON): ON
– SDL_PTHREADS_SEM (Wanted: ON): ON
– SDL_PULSEAUDIO (Wanted: ON): ON
– SDL_PULSEAUDIO_SHARED (Wanted: ON): ON
– SDL_RENDER_D3D (Wanted: OFF): OFF
– SDL_RENDER_METAL (Wanted: OFF): OFF
– SDL_ROCKCHIP (Wanted: ON): OFF
– SDL_RPATH (Wanted: ON): ON
– SDL_RPI (Wanted: ON): OFF
– SDL_SNDIO (Wanted: ON): OFF
– SDL_SNDIO_SHARED (Wanted: ON): OFF
– SDL_SSE (Wanted: OFF): OFF
– SDL_SSE2 (Wanted: OFF): OFF
– SDL_SSE3 (Wanted: OFF): OFF
– SDL_SSE4_1 (Wanted: OFF): OFF
– SDL_SSE4_2 (Wanted: OFF): OFF
– SDL_STATIC_PIC (Wanted: OFF): OFF
– SDL_SYSTEM_ICONV (Wanted: ON): ON
– SDL_TESTS (Wanted: OFF): OFF
– SDL_TESTS_LINK_SHARED (Wanted: ON): OFF
– SDL_VIRTUAL_JOYSTICK (Wanted: ON): ON
– SDL_VIVANTE (Wanted: OFF): OFF
– SDL_VULKAN (Wanted: ON): ON
– SDL_WASAPI (Wanted: OFF): OFF
– SDL_WAYLAND (Wanted: ON): ON
– SDL_WAYLAND_LIBDECOR (Wanted: ON): OFF
– SDL_WAYLAND_LIBDECOR_SHARED (Wanted: ON): OFF
– SDL_WAYLAND_QT_TOUCH (Wanted: ON): ON
– SDL_WAYLAND_SHARED (Wanted: ON): ON
– SDL_X11 (Wanted: ON): ON
– SDL_X11_SHARED (Wanted: ON): ON
– SDL_X11_XCURSOR (Wanted: ON): ON
– SDL_X11_XDBE (Wanted: ON): ON
– SDL_X11_XFIXES (Wanted: ON): ON
– SDL_X11_XINPUT (Wanted: ON): ON
– SDL_X11_XRANDR (Wanted: ON): ON
– SDL_X11_XSCRNSAVER (Wanted: ON): ON
– SDL_X11_XSHAPE (Wanted: ON): ON
– SDL_XINPUT (Wanted: OFF): OFF
–
– Build Shared Library: ON
– Build Static Library: OFF
–
– If something was not detected, although the libraries
– were installed, then make sure you have set the
– CMAKE_C_FLAGS and CMAKE_PREFIX_PATH CMake variables correctly.
–
– Configuring done
– Generating done
“– SDL_ROCKCHIP (Wanted: ON): OFF” This should be ON, correct? I am using Buildroot crosscompiler so maybe I am missing something
Yeap.
Sorry. I can’t help with buildroot, i don’t crosscompile, i do everything natively.
I think you should change this line to be able to crosscompile:
dep_option(SDL_ROCKCHIP “Use ROCKCHIP Hardware Acceleration video driver” ON “UNIX_SYS;SDL_CPU_ARM32 OR SDL_CPU_ARM64”
Try:
dep_option(SDL_ROCKCHIP “Use ROCKCHIP Hardware Acceleration video driver” ON “UNIX_SYS” OFF)
It still doesn’t work, I’ll do my research. You’ve helped plenty already! Thanks a lot