I optimized and enhanced the code to feed NPU with 4 streams (h264 1080p) to detect objects and scaled it down to fit in 1920x1080 display with a reasonable result.
The code runs in wayland/weston and keeping the CPU at low frequency and low Temp.
Object color:
Video teaser
https://drive.google.com/file/d/1SUiZPgnJiQu7F_dBepeH4YMoeMpXggXS/view?usp=sharing
What are your frames like?
Not sure what you mean, the frames are 1920x1080, 29 fps, maybe rendered at 25 ~ 29 fps i guess.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/apps/videos_rknn/vid-3.mp4':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42mp41isomavc1
creation_time : 2020-12-03T00:03:56.000000Z
Duration: 00:00:27.39, start: 0.000000, bitrate: 5963 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080, 5704 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 60k tbc (default)
Metadata:
creation_time : 2020-12-03T00:03:56.000000Z
handler_name : L-SMASH Video Handler
vendor_id : [0][0][0][0]
encoder : AVC Coding
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 253 kb/s (default)
Metadata:
creation_time : 2020-12-03T00:03:56.000000Z
handler_name : L-SMASH Audio Handler
vendor_id : [0][0][0][0]Sorry, had short handed that. I had meant frames per second. That’s pretty damn good.
11 streamers (1080p) + realtime camera (1080p) but needs more CPU usage %
https://drive.google.com/file/d/1qZp87y6Hk6mtk20FpfUsKX1VoNcK0myv/view?usp=sharing
screencast (bps 2M) with ffmpeg while running the demo at same time: (GitHub - hbiyik/FFmpeg: PLEASE USE https://github.com/nyanmisaka/ffmpeg-rockchip REPO INSTEAD.)
Some benchmarks:
CPU0-3 CPU4-5 CPU6-7 DDR DSU GPU NPU
1416 816 816 2112 1416 300 1000
1800 600 1416 2112 1800 300 1000
1200 1608 408 2112 1200 300 1000
816 1416 408 2112 816 300 1000
1416 1416 1416 2112 1416 300 1000
1200 600 816 2112 1200 300 1000
816 408 408 2112 816 300 1000
1800 1416 1608 2112 1800 300 1000
408 1416 408 2112 396 300 1000
Performance:
CPU0-3 CPU4-5 CPU6-7 DDR DSU GPU NPU
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
1800 2352 2256 2112 1800 1000 1000
NPU is the bottleneck.
CPU0-3 CPU4-5 CPU6-7 DDR DSU GPU NPU
408 408 408 2112 396 1000 1000
408 408 408 2112 396 1000 1000
408 408 408 2112 396 1000 1000
408 408 408 2112 396 1000 1000
408 408 408 2112 396 1000 1000
408 408 408 2112 396 1000 1000
408 408 408 2112 396 1000 1000
408 408 408 2112 396 1000 1000
408 408 408 2112 396 1000 1000
Realistically with the npu only 3 1080p30 video steams to get decent interfering at ~30/35 fps and that is with a dedicated core to each stream.
Biggest issue I’m finding with video decoding/npu/gpu isn’t the cpu usage but the thermals. With active cooling the temperatures are reasonable. However with passive cooling and processing 16+ streams the temperatures can easily progress towards 80 degrees (celsius). For embedded solutions we’d prefer passive cooling to avoid undue maintenance.
Have you tried 16+ streams?
No. I haven’t.
I don’t have here with me a 2k or 4k display monitor to be able to display that.
Is this launched via a python script? And is any of this work open source, on GitHub, that I can quickly clone and replicate?
(If not, I understand, I don’t expect everything to be free, and realize there could be intellectual property in this work.)
It is C/C++.
I need to clean up the code and a way to handle rtsp, and release the code. Not sure about the license or any possible patent issue. I will push it to github when done.
Updating the progress on this. h264 rtsp is working, h264 USB camera and h264 streams. Cleaning up the code. The code will be licensed under Apache 2.0.
For this to work you will need:
Current limitations are:
I will try to provide all instructions later.
SDL3 for ROCKCHIP
If you don’t have SDL3 working on your board, you need to build it from sources and make sure 3D accel works.
git clone https://github.com/libsdl-org/SDL --depth=1
cd SDL
mkdir sdl3
cd sdl3
cmake .. -DSDL_TESTS=1 -DCMAKE_BUILD_TYPE=Release
make -j8
sudo cmake --install . --prefix /usr/local
sudo ldconfig
Test it:
cd test/
./testgles2
You will see a spinning cube, when you exit you see the FPS, which should be 1000 ~ 3000, depending on the Desktop (X11 or Wayland). If you get ~ 300 it is software rendered, you missed some prerequisites for HW accel during the build.
screenshot for imx415 fisheye rtsp module and h264 USB camera without yolov5 inference on video.
@jack , please, can you ask Rockchip or @RadxaYuntian to update libmali deb package to rk4.1 so we can test this on latest kernel?
still? i have updated with hevc and vp8 encoder support in ffmpeg. nevermind, i thought this was encoder related.
Yeah, i think RGA outputs buf with padding, without padding it scrambles the result as i pointed out on the git encoder issue, i guess for the same reason.
That’s the reason i would like to run with rk4.1.
ok, i dont understand those npu things, but if you could give a duplication method, i can gdb it and fix.
npu is a separate layer, but it needs RGB888 buf.
Here is the relevant code you can test.
Pass a dma buf to be converted to buf.
src: DRM_FORMAT_NV12_10 / DRM_FORMAT_NV15 and DRM_FORMAT_NV16 (aka YCbCr_420_SP_10B and YCbCr_422_SP)
dst: RK_FORMAT_RGB_888
make this work and we are good to go with H265 and 10bit
static int drm_rga_buf(int src_Width, int src_Height, int src_fd, int src_format, int dst_Width, int dst_Height, int dst_format, int frameSize, char *buf)
{
rga_info_t src;
rga_info_t dst;
int ret;
int hStride = (src_Height + 15) & (~15);
int wStride = (src_Width + 15) & (~15);
//int dhStride = (dst_Height + 15) & (~15);
//int dwStride = (dst_Width + 15) & (~15);
memset(&src, 0, sizeof(rga_info_t));
src.fd = src_fd;
src.mmuFlag = 1;
memset(&dst, 0, sizeof(rga_info_t));
dst.fd = -1;
dst.virAddr = buf;
dst.mmuFlag = 1;
rga_set_rect(&src.rect, 0, 0, src_Width, src_Height, wStride, hStride, src_format);
rga_set_rect(&dst.rect, 0, 0, dst_Width, dst_Height, dst_Width, dst_Height, dst_format);
ret = c_RkRgaBlit(&src, &dst, NULL);
return ret;
}yeah ok, i know rga is being a b**tch when converting RGB, currently i also dont have so much idea about that. So basically rga converts when it inputs from h264_rkmpp_decoder but not from hevc or vp8?
It converts (drm nv12 10bit → rga → rgb888 ) but here is the result of Big_Buck_Bunny_1080_10s_30MB.mp4 (@nyanmisaka) to RGB_888
I have an FFmpeg that can convert it to YCbCr_420_P (i think it is I420, the one the ffplay uses) but it uses dst hstride and vstride, that’s what i mean by padding, but RGB_888 can’t be with padding.
I guess @mtx512rk had exactly the same problem, that is why only h264.
The problem may be in the kernel side (rga).
Here is the interesting part:
[268867.672676] rga_debugger: yuv2rgb mode is 1
[268867.672679] rga_debugger: set core = 0, priority = 0, in_fence_fd = 0
[268867.672685] rga_policy: start policy on core = 1
[268867.672689] rga_policy: start policy on core = 2
[268867.672691] rga_policy: start policy on core = 4
[268867.672694] rga_policy: optional_cores = 7
[268867.672697] rga_policy: assign core: 1
[268867.673041] rga_dma_buf: iova_align size = 6221824
[268867.675029] rga3_reg: render_mode:bitblt, bitblit_mode=0, rotate_mode:0
[268867.675034] rga3_reg: win0: y = fe1d0000 uv = fe3ce000 v = fe44d800 src_w = 1920 src_h = 1080
[268867.675039] rga3_reg: win0: vw = 1920 vh = 1088 xoff = 0 yoff = 0 format = YCbCr420SP
[268867.675043] rga3_reg: win0: dst_w = 1920, dst_h = 1080, rd_mode = 0
[268867.675048] rga3_reg: win0: rot_mode = 1, en = 1, compact = 1, endian = 0
[268867.675051] rga3_reg: wr: y = fcef0010 uv = fd0ea410 v = fd168d10 vw = 1920 vh = 1080
[268867.675056] rga3_reg: wr: ovlp_xoff = 0 ovlp_yoff = 0 format = RGB888 rdmode = 0
[268867.675059] rga3_reg: mmu: win0 = 00 win1 = 00 wr = 00
[268867.675062] rga3_reg: alpha: flag 0 mode0=0 mode1=0
[268867.675066] rga3_reg: blend mode is no blend
[268867.675069] rga3_reg: yuv2rgb mode is 0
[268867.675164] rga_job: job: reqeust_id = 154184, priority = 0, core = 1
[268867.678651] rga_job: request[154184] finished 1 failed 0
[268867.678742] rga: Blit mode: request id = 154185
[268867.678747] rga_debugger: render_mode = 0, bitblit_mode=0, rotate_mode = 1
[268867.678752] rga_debugger: src: y = 2f uv = 0 v = 1fe000 aw = 1920 ah = 1080 vw = 1920 vh = 1088
[268867.678756] rga_debugger: src: xoff = 0, yoff = 0, format = 0xa, rd_mode = 1
[268867.678760] rga_debugger: dst: y=0 uv=7f8a476010 v=7f8a4da010 aw=640 ah=640 vw=640 vh=640
[268867.678763] rga_debugger: dst: xoff = 0, yoff = 0, format = 0x2, rd_mode = 1
[268867.678765] rga_debugger: mmu: mmu_flag=80000521 en=1
[268867.678768] rga_debugger: alpha: rop_mode = 0
[268867.678770] rga_debugger: yuv2rgb mode is 1
[268867.678773] rga_debugger: set core = 0, priority = 0, in_fence_fd = 0
[268867.678780] rga_policy: start policy on core = 1
[268867.678783] rga_policy: start policy on core = 2
[268867.678786] rga_policy: start policy on core = 4
[268867.678789] rga_policy: optional_cores = 7
[268867.678792] rga_policy: assign core: 1
[268867.678903] rga_dma_buf: iova_align size = 1232896
[268867.679035] rga3_reg: render_mode:bitblt, bitblit_mode=0, rotate_mode:0
[268867.679039] rga3_reg: win0: y = fe1d0000 uv = fe3ce000 v = fe44d800 src_w = 1920 src_h = 1080
[268867.679043] rga3_reg: win0: vw = 1920 vh = 1088 xoff = 0 yoff = 0 format = YCbCr420SP
[268867.679046] rga3_reg: win0: dst_w = 640, dst_h = 640, rd_mode = 0
[268867.679049] rga3_reg: win0: rot_mode = 1, en = 1, compact = 1, endian = 0
[268867.679053] rga3_reg: wr: y = fd3b0010 uv = fd414010 v = fd42d010 vw = 640 vh = 640
[268867.679056] rga3_reg: wr: ovlp_xoff = 0 ovlp_yoff = 0 format = RGB888 rdmode = 0
[268867.679059] rga3_reg: mmu: win0 = 00 win1 = 00 wr = 00
[268867.679062] rga3_reg: alpha: flag 0 mode0=0 mode1=0
[268867.679064] rga3_reg: blend mode is no blend
[268867.679066] rga3_reg: yuv2rgb mode is 0
[268867.679084] rga_job: job: reqeust_id = 154185, priority = 0, core = 1
[268867.680672] rga_job: request[154185] finished 1 failed 0