Low frame rate ( Zero 3w and 8m 219)

#1

I’ve been trying to get the advertised framerate out of a Radxa 8m 219 camera with hardware acceleration.

With ffmpeg installed as described here

https://docs.radxa.com/en/zero/zero3/app-development/rtsp?target=ffmpeg

ffmpeg -re -i /dev/video0 output.mkv

This gives me 20fps, without re i get 30 fps which drops to 20 fps.

Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_rkmpp_encoder))
Press [q] to stop, [?] for help
[h264_rkmpp_encoder @ 0xaaab18294ef0] Picture format is yuv420p.
[h264_rkmpp_encoder @ 0xaaab18294ef0] Rate Control mode is set to CBR
[h264_rkmpp_encoder @ 0xaaab18294ef0] Bitrate Target/Min/Max is set to 6000000/5625000/6375000
[h264_rkmpp_encoder @ 0xaaab18294ef0] Profile is set to HIGH
[h264_rkmpp_encoder @ 0xaaab18294ef0] 8x8 Transform is enabled
[h264_rkmpp_encoder @ 0xaaab18294ef0] Level is set to 0
[h264_rkmpp_encoder @ 0xaaab18294ef0] Coder is set to CABAC
[h264_rkmpp_encoder @ 0xaaab18294ef0] Quality Min/Max is set to 50%(Quant=30) / 100%(Quant=10)
Output #0, matroska, to ‘output.mkv’:
Metadata:
encoder : Lavf60.3.100
Stream #0:0: Video: h264 (High) (H264 / 0x34363248), yuv420p(progressive), 1920x1080, q=2-31, 6000 kb/s, 1000k fps, 1k tbn
Metadata:
encoder : Lavc60.3.100 h264_rkmpp_encoder
[h264_rkmpp_encoder @ 0xaaab18294ef0] Reconfigured with w=1920, h=1080, format=yuv420p.
frame= 401 fps= 19 q=-0.0 Lsize= 1885kB time=00:00:20.62 bitrate= 748.4kbits/s speed= 1x

It appears to be using the rkmpp encoder so should be hardware accelerated.

…

Gstreamer gives me blit errors.

Has anyone got higher framerates on this setup ?

Thanks

#2

The tutorial has been updated (https://docs.radxa.com/en/zero/zero3/app-development/rtsp?target=ffmpeg), please try re-flash the image to try

#3

After my testing this tutorial is not compatible with our rk356x series systems, you can follow this tutorial to compile ffmpeg https://github.com/nyanmisaka/ffmpeg-rockchip/wiki/Compilation

1 Like
#4

Thanks @ChenJaly.
It turns out to be the other ways around , if I compile with instructions from GitHub page I get 20 FPS max , using the trap tutorial I can get 60 FPS by setting -s 1920x1080 before input or 15fps at full resolution, which goes against all advise I have received but seems to work.

3 Likes
#5

Can you write a full tutorial on how you got 60 FPS?
I’m still experimenting with Android.

#6

Hi @Viktor_Nikitin ,

I’m using the debian variant of radxaos on a zero 3w. I guess if ffmpeg-rockchip will compile on android then it may work.

All did was follow the

https://docs.radxa.com/en/zero/zero3/app-development/rtsp?target=ffmpeg

tutorial and add -s 1920x1080 befer the input ( as in before -i /dev/video0 ) and up the framerate before the output ( as in -vf "fps=60 after the input ). If i use 4k res i get 15 fps ( which i guess will be the limits of the h264/hevc_rkmpp encoder .

I’ve not looked at running android on any of these sbc’s so have no experience there.

Good luck.

#7

I look at Android as an alternative to Debian and conduct tests in parallel, but it has its own applications. This manual:
https://docs.radxa.com/en/zero/zero3/app-development/rtsp?target=ffmpeg
didn’t help me
(my system is updated via rsetup)

The following manual helped me:

as well as the command:

nohup ./mediamtx &
ffmpeg -f v4l2 -i /dev/video-camera0 -vf “fps=60” -s 1920x1080 -c:v h264_rkmpp -rc_mode AVBR -b:v 20M -minrate 10M -maxrate 40M -profile:v main -level 5.1 -f rtsp rtsp://0.0.0.0:8554/stream

Although it seems like our sensor fps=60 at 1920x1080 does not support, but everything seems to work, which is strange.

#8

I had similar results using that tutorial.
I’m using the 8m 219 camera (imx 219 sensor) .

The only difference between this and the radxa rtsp tutorial seems to be the rga lib, so maybe this was an issue for me.

Clean os install and the rstp tutorial worked for me, or at least appears to work.

The way you describe gave me between 7 and 15 fps with ffmpeg , i get up to 20 fps with gstreamer.
rstp tutorial gave me 60 fps at 1080 , 15 at 4k.

Good luck.

…
have you tried this

ffmpeg -f v4l2 -s 1920x1080 -i /dev/video-camera0 -vf “fps=60” -pix_fmt yuv420p -c:v h264_rkmpp -rc_mode AVBR -b:v 20M -minrate 10M -maxrate 40M -profile:v main -level 5.1 -f rtsp rtsp://0.0.0.0:8554/stream

which is closer to what i have been using.

#9

@Viktor_Nikitin and @ChenJaly

I had to install a similar setup on another box and have realized that you have to mix and match both tutorials to get this working … ?

If you install mpp by following the first part of this tutorial

i.e.

Build MPP

mkdir -p ~/dev && cd ~/dev
git clone -b jellyfin-mpp --depth=1 https://github.com/nyanmisaka/mpp.git rkmpp
pushd rkmpp
mkdir rkmpp_build
pushd rkmpp_build
cmake
-DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_BUILD_TYPE=Release
-DBUILD_SHARED_LIBS=ON
-DBUILD_TEST=OFF
…
make -j $(nproc)
make install

DO NOT INSTALL DRM from this tutorial ( well , it didn’t work when i did it this way )

Then follow this

It works for me on zero 3w, if I dont do the mpp compilation I get seg faults from ffmpeg, if I use the rockchip-ffmpeg drm compilation I dont get more than 20fps.

Russ

#10

You may need to install the compilation dependencies at https://docs.radxa.com/en/rock5/rock5b/app-development/rtsp#installation-of-compilation-dependencies first, then compile rkmpp and rga according to https://github.com/nyanmisaka/ffmpeg-rockchip/wiki/Compilation Compile rkmpp and rga to override the dependencies we just installed, then proceed to compile ffmpeg using our tutorials

#11

I decided to write a step-by-step instruction for the RTSP server.
Install Debian B6
Update the system via rsetup, activate the 8M camera.

Installation of compilation dependencies

sudo apt-get install build-essential cmake git libdrm-dev librga-dev librockchip-mpp-dev libsdl2*-dev libx264-dev libx265-dev pkg-config meson

Build MPP

mkdir -p ~/dev && cd ~/dev
git clone -b jellyfin-mpp --depth=1 https://github.com/nyanmisaka/mpp.git rkmpp
pushd rkmpp
mkdir rkmpp_build
pushd rkmpp_build
cmake
-DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_BUILD_TYPE=Release
-DBUILD_SHARED_LIBS=ON
-DBUILD_TEST=OFF
…
make -j $(nproc)
sudo make install
cd ~

Build RGA

mkdir -p ~/dev && cd ~/dev
git clone -b jellyfin-rga --depth=1 https://github.com/nyanmisaka/rk-mirrors.git rkrga
meson setup rkrga rkrga_build
–prefix=/usr
–libdir=lib
–buildtype=release
–default-library=shared
-Dcpp_args=-fpermissive
-Dlibdrm=false
-Dlibrga_demo=false
meson configure rkrga_build
sudo ninja -C rkrga_build install
cd ~

#Compile and install ffmpeg-rockchip

git clone https://github.com/nyanmisaka/ffmpeg-rockchip
pushd ffmpeg-rockchip/

./configure --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga --enable-libx264 --enable-libx265 --enable-ffplay

make -j$(nproc)
sudo make install
popd

#Running mediamtx and ffmpeg-rockchip
wget https://github.com/bluenviron/mediamtx/releases/download/v1.9.0/mediamtx_v1.9.0_linux_arm64v8.tar.gz
tar -xzvf mediamtx_v1.9.0_linux_arm64v8.tar.gz
nohup ./mediamtx & ffmpeg-rockchip/ffmpeg -re -s 1920x1080 -i /dev/video-camera0 -vf “fps=60” -pix_fmt yuv420p -preset ultrafast -tune zero_latency -c:v hevc_rkmpp -rc_mode AVBR -b:v 20M -minrate 10M -maxrate 40M -profile:v main -level 5.1 -f rtsp -omit_video_pes_length 0 rtsp://0.0.0.0:8554/stream

#12

Has anyone achieved more then ~20 fps with gstreamer?
Any inputs on why it is not supported would be appreciated

#13

Build rtsp server:

Install dependencies

sudo apt update
sudo apt install build-essential gobject-introspection libcgroup-dev libgirepository1.0-dev libgstreamer-plugins-bad1.0-dev libgstreamer-plugins-base1.0-dev meson librga-dev

sudo apt-get install build-essential cmake git libdrm-dev librga-dev librockchip-mpp-dev libsdl2*-dev libx264-dev libx265-dev pkg-config

This needs to be compiled (only Build MPP and Build RGA)

Next, compile according to the instructions:

Then run (/rockchip/gstreamer/rtsp-server/gst-rtsp-server-1.18.4/build/examples )
./test-launch “( v4l2src device=/dev/video-camera0 io-mode=dmabuf ! video/x-raw,format=NV12,width=1280,height=720,framerate=60/1 ! mpph265enc ! rtph265pay name=pay0 pt=96 )”

or you can try this instruction:

#14

Is your stream delivering more than ~20 FPS?

Try displaying the stream with the following GStreamer pipeline:

gst-launch-1.0 rtspsrc location=rtsp://192.168.50.101:8554/test latency=10 ! rtph265depay ! h265parse ! avdec_h265 ! queue ! fpsdisplaysink

You’ll notice on the display that the actual frame rate you get is not more than 20 FPS.

image
image.jpg1260×270 76 KB

However, if I use the following FFmpeg example:

nohup ./mediamtx & ffmpeg -re -s 1920x1080 -i /dev/video-camera0 -vf "fps=60" -pix_fmt yuv420p -preset ultrafast -tune zero_latency -c:v hevc_rkmpp -rc_mode AVBR -b:v 20M -minrate 10M -maxrate 40M -profile:v main -level 5.1 -f rtsp -omit_video_pes_length 0 rtsp://0.0.0.0:8554/test

I can capture the stream in the same way and actually achieve ~60 FPS.

image
image.jpg1680×270 99.5 KB

I think is something with the v4l2 capture because even if i capture with v4l2-ctl i can’t do better then 20 FPS:

$ v4l2-ctl --device=/dev/video-camera0 --set-fmt-video=width=640,height=360,pixelformat=NV12 --stream-mmap --stream-count=0
<<<<<<<<<<<<<<<<<<<<< 19.39 fps
<<<<<<<<<<<<<<<<<<< 19.39 fps
<<<<<<<<<<<<<<<<<<<< 19.39 fps
<<<<<<<<<<<<<<<<<<< 19.39 fps
<<<<<<<<<<<<<<<<<<< 19.39 fps
<<<<<<<<<<<<<<<<<<<< 19.39 fps

#15

I have been messing around with the simple task of setting up an RTSP server from a mipi camera and receiving this stream with minimal delays for about half a year now.
In my tests, I first receive the Happytime Rtsp Client X64 stream on Windows, and only then try to receive it on RAdxa 3W or Orangepi CM4.
RTSP server based on RADXA 3W
And I encountered the fact that there is a fake number of frames and resolution in the case of the Gstreamer. It gave me 4K and 20 FPS as real, but the receiver displayed the resolution that was set, or if you do not set the resolution, it gave 4K 120 FPS, although the camera physically cannot give this.

In a neighboring topic, I encountered the output of the camera to HDMI with the fact that the camera does not switch modes for some reason and as I assume, I may be wrong, we have an interpolated resolution from the base 4K 20 fps. Although the camera supports other modes, for testing I took Raspb Camera V2 with the same sensor, it behaves the same, but for some reason the modes do not switch. Therefore, it is not yet clear what we actually get.

The server based on ffmpeg gives worse delays. I refused it.
In my opinion, the compiled gst-rtsp-server-1.18.4 gives better quality and minimal delays. I am currently receiving the stream on a Windows PC. Stream parameters (number of frames and FPS there seem to be correctly shown)

You can compile version 1.20.3 using this instruction


I have not had time to test it yet.
I started testing the FPS quantity and quality with a wave of my hand and finger :slight_smile:

And the second problem is to receive this stream with minimal delays as on a PC. It is obvious that it can be received using a gstreamer
At the moment, I am trying to figure out how to receive a stream on an Orange pi CM4 board, which has a similar processor. I chose Armbian as the OS, but in the minimal IOT version I could not get Gstreamer and MPP to work together
Now I have installed Armbian with a desktop and deleted it, there is a built-in gstreamer and MPP, but so far I have encountered an access rights error, while I am trying to figure this out, although I have deleted gnome and X. The command gst-launch-1.0 videotestsrc ! kmssink works correctly.

I will switch to Radxa 3W as a receiver after CM4.
There is a problem with Armbian for Radxa 3W with a desktop, which I described on the Armbian forum.
There used to be an Armbian Server image, but for some reason they removed it. Theoretically, you can build your own image with the necessary components using the Armbian constructor.

At the moment, I don’t have a good solution yet.

#16

Hi everyone,
According to the information I learned from the BSP engineer, due to the performance restrictions of the camera, it can only run in the 4K 20FPS mode.It’s not the problem of coding performance