ROCK 5B Camera streaming with RTSP / RTMP

Here are the basic instructions to stream a video from the imx415 camera using rtsp protocol.
For the instructions, i am using Debian 11.5 with X11, but it really does not matter which distro.

ROCK 5B IP: 192.168.254.228 (our server)
Client IP: 92.168.254.200 (our client where we display the camera content)

pre-requisites

  • rkmpp encoding capabilities (hw encode enabled)
  • imx415 camera
  • gstreamer with the rkmpp plugin

For learning purposes, i have built the gstreamer rtsp server.

Checking gstreamer encoder:

sudo gst-inspect-1.0|grep -i rock
rockchipmpp:  mppjpegdec: Rockchip's MPP JPEG image decoder
rockchipmpp:  mppvideodec: Rockchip's MPP video decoder
rockchipmpp:  mppjpegenc: Rockchip Mpp JPEG Encoder
rockchipmpp:  mppvp8enc: Rockchip Mpp VP8 Encoder
rockchipmpp:  mpph265enc: Rockchip Mpp H265 Encoder
rockchipmpp:  mpph264enc: Rockchip Mpp H264 Encoder

Build gstreamer rtsp server:

mkdir -p rockchip/gstreamer/rtsp-server
cd rockchip/gstreamer/rtsp-server/
wget https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.18.4.tar.xz
tar -xf gst-rtsp-server-1.18.4.tar.xz
cd gst-rtsp-server-1.18.4
mkdir build && cd build
meson --prefix=/usr --wrap-mode=nofallback -D buildtype=release -D package-origin=https://gstreamer.freedesktop.org/src/gstreamer/ -D package-name="GStreamer 1.18.4" ..
ninja -j4
sudo ninja install
sudo ldconfig

Testing our rtsp server with videotestsrc:

cd example
~/rockchip/gstreamer/rtsp-server/gst-rtsp-server-1.18.4/build/examples$ 
sudo ./test-launch "( videotestsrc ! video/x-raw,format=NV12,width=1920,height=1080 ! mpph264enc ! rtph264pay name=pay0 pt=96 )"
    stream ready at rtsp://127.0.0.1:8554/test
    rga_api version 1.8.1_[0]

From our client, we check if it is OK, here we use ffplay:

ffplay -rtsp_transport tcp rtsp://192.168.254.228:8554/test

Now we have the video displayed with some patterns.
We can now stream our video:

~/rockchip/gstreamer/rtsp-server/gst-rtsp-server-1.18.4/build/examples$ 
sudo ./test-launch "( v4l2src device=/dev/video11 io-mode=dmabuf ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! mpph264enc ! rtph264pay name=pay0 pt=96 )"
stream ready at rtsp://127.0.0.1:8554/test
rga_api version 1.8.1_[0]

Again, we display the content using ffplay with the same URI:

ffplay -rtsp_transport tcp rtsp://192.168.254.228:8554/test
ffplay version 5.0 Copyright (c) 2003-2022 the FFmpeg developers
  built with gcc 8 (Ubuntu 8.4.0-1ubuntu1~18.04)
  configuration: --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libmp3lame --enable-libopus --enable-libpulse --enable-librsvg --enable-librtmp --enable-libv4l2 --enable-libvorbis --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libxvid --enable-libxml2 --enable-opengl --enable-openssl --enable-gpl --enable-nonfree --disable-librga
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
  libpostproc    56.  3.100 / 56.  3.100
Input #0, rtsp, from 'rtsp://192.168.254.228:8554/test': 0B f=0/0   
  Metadata:
    title           : Session streamed with GStreamer
    comment         : rtsp-server
  Duration: N/A, start: 0.034367, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 1920x1080, 30 fps, 30 tbr, 90k tbn
  72.28 M-V: -0.005 fd=   2 aq=    0KB vq=  360KB sq=    0B f=0/0 

Have fun!

CPU usage:
rtsp

3 Likes

Setting up RTMP (Real-Time Messaging Protocol) to go Live on Youtube:

1. RTMP protocol

We can use RTMP (Real-Time Messaging Protocol) protocol used by Video-on-demand and Youtube and share the link to go Live

2. Configuring Youtube LiveStream

Go to Youtube Studio and click on [Create] and select [Go live],
Fill out the Stream Settings and get the Stream Key, like this

Once you get the URL and the key, fire it on ROCK 5B like this:

rtmp://url/key

sudo gst-launch-1.0 v4l2src device=/dev/video11 io-mode=dmabuf ! 'video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1' ! mpph264enc ! mux. audiotestsrc wave=silence ! voaacenc bitrate=128000 ! aacparse ! mux. flvmux streamable=true name=mux ! queue ! rtmpsink location="rtmp://a.rtmp.youtube.com/live2/zx9z-rw5p-suqy-6vxg-zcpu"

Wait a few seconds for the stream to show up on youtube, and then click [SHARE] to get the URL to share the video.

RTMP CPU usage is higher than RTSP, possibly due to the dummy sound required to send the stream to youtube but still fine:
rtmp

This can be used to stream HDMI IN as well and I’ve tested the 4K60 is working. Just change to v4l2src device=/dev/video0.

One problem is that I cannot find anyway to set bitrate limit on the stockmpph64enc/mpph265enc

1 Like

A few notes:

  • with a camera attached, hdmi-in changes to /dev/video20

I haven’t looked at the code to see if there are any properties to be used.

In my case, if i attach an HDMI from a computer i cannot change the format from BGR to NV12.
If i remember the format is read from HDMI port address and is always BGR and when i change to NV12 kernel complains and tells me i can only have BGR.

Makes me wonder if this is device specific?

It is device specific indeed. From my own experience, 4K60 inputs will be NV12, and 1080p inputs will be BGR, and you have to adjust the command for the corresponding format.

I cannot say that every HDMI device will behave the same way as mine, so you have to look at the v4l2 information to confirm the format before running the server.

I will find a 4k tv monitor to try out.
Looks like 1080@60 is BGR as you noted.
Converting from BGR to NV12 does not look like a good solution, i think it will be by SW conversion.

rock@rock5b:~$ v4l2-ctl -d /dev/video20 --all 
Driver Info:
	Driver name      : rk_hdmirx
	Card type        : rk_hdmirx
	Bus info         : fdee0000.hdmirx-controller
	Driver version   : 5.10.66
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Priority: 2
DV timings:
	Active width: 1920
	Active height: 1080
	Total width: 2200
	Total height: 1125
	Frame format: progressive
	Polarities: -vsync -hsync
	Pixelclock: 148496000 Hz (60.00 frames per second)
	Horizontal frontporch: 88
	Horizontal sync: 44
	Horizontal backporch: 148
	Vertical frontporch: 4
	Vertical sync: 5
	Vertical backporch: 36
	Standards: 
	Flags: 
DV timings capabilities:
	Minimum Width: 640
	Maximum Width: 4096
	Minimum Height: 480
	Maximum Height: 2160
	Minimum PClock: 20000000
	Maximum PClock: 600000000
	Standards: CTA-861
	Capabilities: Interlaced, Progressive
Format Video Capture Multiplanar:
	Width/Height      : 1920/1080
	Pixel Format      : 'BGR3' (24-bit BGR 8-8-8)
	Field             : None
	Number of planes  : 1
	Flags             : premultiplied-alpha, 0x000000fe
	Colorspace        : Unknown (0x106a5df0)
	Transfer Function : Unknown (0x000000b8)
	YCbCr/HSV Encoding: Unknown (0x000000ff)
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 5760
	   Size Image     : 6220800

Digital Video Controls

                  power_present 0x00a00964 (bitmask): max=0x00000001 default=0x00000000 value=0x00000001 flags=read-only

rock@rock5b:~$ v4l2-ctl -d /dev/video20 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture Multiplanar

[0]: 'BGR3' (24-bit BGR 8-8-8)
[1]: 'NV24' (Y/CbCr 4:4:4)
[2]: 'NV16' (Y/CbCr 4:2:2)
[3]: 'NV12' (Y/CbCr 4:2:0)


rock@rock5b:~$ v4l2-ctl -d /dev/video20 --set-fmt-video=pixelformat=NV12 --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: failed: Invalid argument

-- LOG
[46289.678510] fdee0000.hdmirx-controller: hdmirx_get_pix_fmt: pix_fmt: RGB888
[46289.678514] fdee0000.hdmirx-controller: hdmirx_get_colordepth: color_depth: 24, reg_val:4
[46289.678521] fdee0000.hdmirx-controller: get timings from ctrl
[46289.678525] fdee0000.hdmirx-controller: act:1920x1080, total:2200x1125, fps:60, pixclk:148496000
[46289.678529] fdee0000.hdmirx-controller: hfp:84, hs:48, hbp:148, vfp:4, vs:5, vbp:36
[46289.678531] fdee0000.hdmirx-controller: tmds_clk:148496000
[46289.678534] fdee0000.hdmirx-controller: interlace:0, fmt:0, vic:127, color:24, mode:dvi

Some graphics cards allow changing the pixel format no NV16 for the HDMI output, unfortunately not my case.
Checking with my 4k tv set.

This works with CPU: ~95%

sudo gst-launch-1.0 v4l2src device=/dev/video20 io-mode=dmabuf ! ‘video/x-raw,format=BGR,width=1920,height=1080,framerate=30/1’ ! videoconvert ! ‘video/x-raw,width=1920,height=1080,format=NV12,framerate=30/1’ ! mpph264enc ! h264parse ! mux. audiotestsrc wave=silence ! voaacenc bitrate=128000 ! aacparse ! mux. flvmux streamable=true name=mux ! queue ! rtmpsink location=“rtmp://a.rtmp.youtube.com/live2/x9z-rw5p-suqy-6vxg-fcpu”

Hi, have you found a way to set bitrate limit on mpph264enc/mpph265enc?

have you tried using mpph265enc bps=5000000 bps-max=5172000 ?

1 Like

rkmpphas a bps option and rc-mode option.

You can set cbr or vbr rc mode with bps to try to limit bitrate.

I found this in the source code: https://github.com/JeffyCN/rockchip_mirrors/blob/546cb913e3feabc4a41a6220fa80d81e0dea71c2/gst/rockchipmpp/gstmppenc.h#L75

I really hate the lack of proper documentation from Rockchip and we have to dig into the header files for such simple option.

1 Like

@gnattu @bytecode64

Thank you guys! I tried here and its working using these parameters.

Here is the instructions for Ubuntu 22.04 , X11, xfce4, latest kernel.
Using the @bytecode64 suggestion, i passed the parms to the encoder that reduces the bandwidth , but if you happen (or anyone else) to know better parms to reduce even more bandwidth and keep quality, please suggest.

Build the server:

mkdir -p rtsp-server
cd rtsp-server/
wget https://gstreamer.freedesktop.org/src/gst-rtsp-server/gst-rtsp-server-1.20.3.tar.xz
tar -xf gst-rtsp-server-1.20.3.tar.xz
cd gst-rtsp-server-1.20.3/
mkdir build && cd build
meson --prefix=/usr --wrap-mode=nofallback -D buildtype=release -D package-origin=https://gstreamer.freedesktop.org/src/gstreamer/ -D package-name="GStreamer 1.20.3" ..
ninja -j8
sudo ninja install
sudo ldconfig
cd examples/

Run the server with 4k streaming:

./test-launch "( v4l2src device=/dev/video11 io-mode=dmabuf ! video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1 ! mpph265enc bps=2000000 bps-max=3172000 rc-mode=vbr ! rtph265pay name=pay0 pt=96 )"

CPU Usage average: 408 ~1200 Mhz ( 8 hours testing the best streaming 4k 30 fps)

To be tested (audio):

./test-launch "( v4l2src device=/dev/video11 io-mode=dmabuf ! video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1 ! mpph265enc bps=1000000 bps-max=3172000 rc-mode=vbr ! rtph265pay name=pay0 pt=96 ! audiotestsrc ! alawenc ! rtppcmapay name=pay1 pt=97 )"

AC3 test

./test-launch "( v4l2src device=/dev/video11 io-mode=dmabuf ! video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1 do-timestamp=true ! mpph265enc bps=1000000 bps-max=3172000 rc-mode=vbr ! rtph265pay name=pay0 pt=96 ! audiotestsrc ! avenc_ac3 ! rtpac3pay name=pay1 pt=97 )"

and:

./test-launch "( v4l2src device=/dev/video11 io-mode=dmabuf ! video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1 ! mpph265enc bps=1000000 bps-max=3172000 rc-mode=vbr ! rtph265pay name=pay0 pt=96 ! alsasrc device=sysdefault:CARD=rockchipes8316 ! alawenc ! rtppcmapay name=pay1 pt=97 )"

Taken from:

record -L
null
    Discard all samples (playback) or generate zero samples (capture)
samplerate
    Rate Converter Plugin Using Samplerate Library
speexrate
    Rate Converter Plugin Using Speex Resampler
jack
    JACK Audio Connection Kit
oss
    Open Sound System
pulse
    PulseAudio Sound Server
upmix
    Plugin for channel upmix (4,6,8)
vdownmix
    Plugin for channel downmix (stereo) with a simple spacialization
default
    Playback/recording through the PulseAudio sound server
usbstream:CARD=rockchiphdmi0
    rockchip-hdmi0
    USB Stream Output
usbstream:CARD=rockchiphdmi1
    rockchip-hdmi1
    USB Stream Output
usbstream:CARD=rockchipdp0
    rockchip,dp0
    USB Stream Output
hw:CARD=rockchipes8316,DEV=0
    rockchip-es8316, fe470000.i2s-ES8316 HiFi ES8316 HiFi-0
    Direct hardware device without any conversions
plughw:CARD=rockchipes8316,DEV=0
    rockchip-es8316, fe470000.i2s-ES8316 HiFi ES8316 HiFi-0
    Hardware device with all software conversions
sysdefault:CARD=rockchipes8316
    rockchip-es8316, fe470000.i2s-ES8316 HiFi ES8316 HiFi-0
    Default Audio Device
dsnoop:CARD=rockchipes8316,DEV=0
    rockchip-es8316, fe470000.i2s-ES8316 HiFi ES8316 HiFi-0
    Direct sample snooping device
usbstream:CARD=rockchipes8316
    rockchip-es8316
    USB Stream Output
hw:CARD=rockchiphdmiin,DEV=0
    rockchip,hdmiin, fddf8000.i2s-dummy_codec hdmiin-dc-0
    Direct hardware device without any conversions
plughw:CARD=rockchiphdmiin,DEV=0
    rockchip,hdmiin, fddf8000.i2s-dummy_codec hdmiin-dc-0
    Hardware device with all software conversions
sysdefault:CARD=rockchiphdmiin
    rockchip,hdmiin, fddf8000.i2s-dummy_codec hdmiin-dc-0
    Default Audio Device
dsnoop:CARD=rockchiphdmiin,DEV=0
    rockchip,hdmiin, fddf8000.i2s-dummy_codec hdmiin-dc-0
    Direct sample snooping device
usbstream:CARD=rockchiphdmiin
    rockchip,hdmiin
    USB Stream Output

And finally, if you want to play the rtsp stream in your ROCK 5B:

GST_GL_API=gles2 GST_GL_PLATFORM=egl gst-launch-1.0 uridecodebin uri=rtsp://192.168.254.10:8554/test ! glimagesink
Setting pipeline to PAUSED ...
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Pipeline is live and does not need PREROLL ...
Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayGBM\)\ gldisplaygbm0";
Progress: (open) Opening Stream
Pipeline is PREROLLED ...
Prerolled, waiting for progress to finish...
Progress: (connect) Connecting to rtsp://192.168.254.10:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (request) SETUP stream 1
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Redistribute latency...
Redistribute latency...
Redistribute latency...
Progress: (request) Sending PLAY request
Redistribute latency...
Progress: (request) Sent PLAY request
Redistribute latency...
Redistribute latency...
Redistribute latency...
Redistribute latency...
Redistribute latency...
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:25.353149438
Setting pipeline to NULL ...
Freeing pipeline ...

That’s all folks!

2 Likes

u can add n-treads=3 parameter to videoconvert, it will spread load for CPU