Decoding and rendering 16 1080p streams with Gstreamer

[avaf]

It’s now possible to decode and render 16 1080p streams on X11 with gstreamer. This is done by two xvimagesink properties created by JeffyCN on the latest patch.
The patch adds the new properties to xvimagesink, render-rectangle, and decoration, where you can now position the decoded stream on screen coordinates and without decoration, for example:

gst-launch-1.0 filesrc location=~/Videos/ElephantsDream.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,0,480,270>' decorations=0

Full Video here:
https://mega.nz/file/RfpGSQqa#vw0iZ_zVFWPfCiY1HO-2-T9udRJq6gO-F8DPnDp2f-Q

[diggonomics]

This looks very impressive, I assume each stream is a file and played via a separate decoder. Is there a step-by-step for a noob to reproduce this? I want to see if possible to get multiple cameras on screen at the same time but getting the linux distro to work on 3588 is a pain for a beginner - trying to find an image that “works” did not yield good results yet

[avaf]

assume each stream is a file and played via a separate decode

Yeah.

#!/bin/bash
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/rickiandtheflash-tlr2_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,0,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Mad_Max_Fury_Road_2015_Trailer_F4_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<480,0,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Star_Wars_Episode_VII_The_Force_Awakens_2015_Trailer_B_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<960,0,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Poltergeist_DOM_TrailerD-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<1440,0,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/BatmanvSuperman_TLR-1_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,270,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Tomorrowland_2015_Trailer_H_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<480,270,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Fantastic_Four_2015_Trailer_2_Intl_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<960,270,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/TheLastWitchHunter_TRLR1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<1440,270,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Jurassic_World_TrailerA_International_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,540,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/San_Andreas_2015_International_Trailer_3_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<480,540,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Furious_7_2015_International_Trailer_2_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<960,540,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/hell-fest-trailer-1_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<1440,540,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/smallfoot-trailer-1_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,810,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/evil-dead-rise-trailer-2_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<480,810,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/john-wick-chapter-four-trailer-3_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<960,810,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/heavenknowswhat-tlr1_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<1440,810,480,270>' decorations=0&

to get multiple cameras on screen at the same time but getting the linux

Someone here on the forum posted instructions to get hw video accel and hw decoding for X11.
@boogiepop or @amazingfate may push the latest gstreamer with that patch on their distro package.
Note the aspect ratio is not maintained, for that i use ffplay which has higher CPU usage.

[avaf]

I’ve done some basic experiments about ffplay vs gstreamer to see how it performs displaying the same 16 video streams.

The actual test consists of displaying the streams and monitoring the CPU usage with htop and seeing the best performance.

The latest gstreamer in use here has a DMABUF 0-copy and uses arm-afbc while ffplay uses memcpy and rga2.
i tried to implement 0-copy but failed miserably, maybe lack of knowledge.

Here you can compare the two and choose the one that suits you best:

1 – ffplay

2 - gstreamer

[avaf]

little improvement, redirect output and error to /dev/null.

Update: wrong image, fixed.

1 - xvimagesink

2 - glimagesink

Maybe wayland can reduce CPU usage.

[boogiepop]

Cool stuff. Did you give any chance to mpv with drm prime since we last discussed? I am currently lazily working on ffmpeg encoder so i did not give so much of a thought afterwards.

[avaf]

I tried mpv but for some reason, i could not get hw decoding to work. I patched the mpv stock version but did not find the cause.

I’m a ffmpeg fanboy, so i managed to build ffplay DRM version but it still gets some flickering, at least i get audio and the correct fps with minimum CPU usage.

By now i am trying to dig into this thing called gstreamer.

[mtx512rk]

I’m a ffmpeg fanboy, so i managed to build ffplay DRM version but it still gets some flickering, at least i get audio and the correct fps with minimum CPU usage.

Your prone to flickering partly because there is no h/w vsync so X11 renders the frames per video window as and when it completes processing. Using gstreamer under X11 won’t help.

[avaf]

In theory, xvimage should have dmabuf 0-copy but xorg still has high CPU usage.
Can you tell what you use in your demo? ffmpeg,gstreamer or else?
I am curious to know if running this sample in Wayland would get better results. I still not running Wayland (yet).

[mtx512rk]

In the end I went down the custom coding route using mpp sample as a starting point and custom rendering (Wayland) as my use case requires resizing and overlaying graphics over the video. To be honest wouldn’t recommend that approach unless you have a good understanding of the underlying tech stacks as there is lot of debugging required as not everything works as its should. Took me 2 months just to get the resizing to work and overlaying graphics still work in progress.

[avaf]

I agree, I think it is a lot of work (or rework?), but if you want to get the best performance with less CPU usage, this is the way to go.

[avaf]

Just for fun, i was able to play mpv with hw decode (drm_prime) with the expected results for 1080p output, low CPU usage. The problem is i can’t output to a X11 (opengl es), no matter if --vo=gpu or --vo=opengl

file: vp9_4k_60_sdr.webm --vo=gpu

Title: 8K VIDEOS | World 8K Videos HDR UltraHD  (120 FPS) | Sony Demo
[vo/gpu] VT_GETMODE failed: Inappropriate ioctl for device
[vo/gpu/opengl] Failed to set up VT switcher. Terminal switching will be unavailable.
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
AO: [pulse] 48000Hz stereo 2ch float
Using hardware decoding (rkmpp).
VO: [gpu] 3840x2160 drm_prime[nv12]
[vo/gpu] Using HW-overlay mode. No GL filtering is performed on the video!
AV: 00:01:14 / 00:05:12 (24%) A-V:  0.000 Dropped: 1

Exiting... (End of file)

[avaf]

Running with Weston, better results.

[fdalto]

Hi,
Could you help me adding AUDIO to this code?
gst-launch-1.0 v4l2src device=/dev/video0 io-mode=dmabuf ! ‘video/x-raw,width=1280,height=720,framerate=20/1’ ! mpph264enc ! filesink location=test_h264.mp4

I want to capture USB device (usb video capture) with video and audio and recort to a file.

[avaf]

There is an interesting reading here about gstreamer and AUDIO:

ROCK 5B Camera streaming with RTSP / RTMP

Here are the basic instructions to stream a video from the imx415 camera using rtsp protocol. For the instructions, i am using Debian 11.5 with X11, but it really does not matter which distro. ROCK 5B IP: 192.168.254.228 (our server) Client IP: 92.168.254.200 (our client where we display the camera content) pre-requisites rkmpp encoding capabilities (hw encode enabled) imx415 camera gstreamer with the rkmpp plugin For learning purposes, i have built the gstreamer rtsp server. Checking gs…

First, ensure you can record from the USB using arecord, then find the corresponding USB card device and use it instead of rockchipes8316.

arecord examples:
(https://www.geeksforgeeks.org/arecord-command-in-linux-with-examples/)