Decoding and rendering 16 1080p streams with Gstreamer

avaf · April 30, 2023, 3:23am

It’s now possible to decode and render 16 1080p streams on X11 with gstreamer. This is done by two xvimagesink properties created by JeffyCN on the latest patch.
The patch adds the new properties to xvimagesink, render-rectangle, and decoration, where you can now position the decoded stream on screen coordinates and without decoration, for example:

gst-launch-1.0 filesrc location=~/Videos/ElephantsDream.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,0,480,270>' decorations=0

Full Video here:
https://mega.nz/file/RfpGSQqa#vw0iZ_zVFWPfCiY1HO-2-T9udRJq6gO-F8DPnDp2f-Q

diggonomics · May 1, 2023, 10:58am

This looks very impressive, I assume each stream is a file and played via a separate decoder. Is there a step-by-step for a noob to reproduce this? I want to see if possible to get multiple cameras on screen at the same time but getting the linux distro to work on 3588 is a pain for a beginner - trying to find an image that “works” did not yield good results yet

avaf · May 1, 2023, 12:42pm

Yeah.

#!/bin/bash
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/rickiandtheflash-tlr2_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,0,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Mad_Max_Fury_Road_2015_Trailer_F4_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<480,0,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Star_Wars_Episode_VII_The_Force_Awakens_2015_Trailer_B_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<960,0,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Poltergeist_DOM_TrailerD-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<1440,0,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/BatmanvSuperman_TLR-1_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,270,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Tomorrowland_2015_Trailer_H_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<480,270,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Fantastic_Four_2015_Trailer_2_Intl_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<960,270,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/TheLastWitchHunter_TRLR1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<1440,270,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Jurassic_World_TrailerA_International_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,540,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/San_Andreas_2015_International_Trailer_3_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<480,540,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/Furious_7_2015_International_Trailer_2_5.1-1080p-HDTN.mp4 ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<960,540,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/hell-fest-trailer-1_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<1440,540,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/smallfoot-trailer-1_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<0,810,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/evil-dead-rise-trailer-2_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<480,810,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/john-wick-chapter-four-trailer-3_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<960,810,480,270>' decorations=0&
DISPLAY=:0.0 gst-launch-1.0 filesrc location=~/Videos/heavenknowswhat-tlr1_h1080p.mov ! qtdemux ! h264parse ! mppvideodec width=480 height=270 ! xvimagesink render-rectangle='<1440,810,480,270>' decorations=0&

Someone here on the forum posted instructions to get hw video accel and hw decoding for X11.
@boogiepop or @amazingfate may push the latest gstreamer with that patch on their distro package.
Note the aspect ratio is not maintained, for that i use ffplay which has higher CPU usage.

avaf · May 7, 2023, 12:05am

I’ve done some basic experiments about ffplay vs gstreamer to see how it performs displaying the same 16 video streams.

The actual test consists of displaying the streams and monitoring the CPU usage with htop and seeing the best performance.

The latest gstreamer in use here has a DMABUF 0-copy and uses arm-afbc while ffplay uses memcpy and rga2.
i tried to implement 0-copy but failed miserably, maybe lack of knowledge.

Here you can compare the two and choose the one that suits you best:

1 – ffplay

2 - gstreamer

avaf · May 12, 2023, 11:47am

little improvement, redirect output and error to /dev/null.

Update: wrong image, fixed.

1 - xvimagesink

2 - glimagesink

Maybe wayland can reduce CPU usage.

boogiepop · May 10, 2023, 11:37pm

Cool stuff. Did you give any chance to mpv with drm prime since we last discussed? I am currently lazily working on ffmpeg encoder so i did not give so much of a thought afterwards.

avaf · May 11, 2023, 1:28am

I tried mpv but for some reason, i could not get hw decoding to work. I patched the mpv stock version but did not find the cause.

I’m a ffmpeg fanboy, so i managed to build ffplay DRM version but it still gets some flickering, at least i get audio and the correct fps with minimum CPU usage.

By now i am trying to dig into this thing called gstreamer.

mtx512rk · May 11, 2023, 12:04pm

I’m a ffmpeg fanboy, so i managed to build ffplay DRM version but it still gets some flickering, at least i get audio and the correct fps with minimum CPU usage.

Your prone to flickering partly because there is no h/w vsync so X11 renders the frames per video window as and when it completes processing. Using gstreamer under X11 won’t help.

avaf · May 11, 2023, 12:45pm

In theory, xvimage should have dmabuf 0-copy but xorg still has high CPU usage.
Can you tell what you use in your demo? ffmpeg,gstreamer or else?
I am curious to know if running this sample in Wayland would get better results. I still not running Wayland (yet).

mtx512rk · May 12, 2023, 7:34am

In the end I went down the custom coding route using mpp sample as a starting point and custom rendering (Wayland) as my use case requires resizing and overlaying graphics over the video. To be honest wouldn’t recommend that approach unless you have a good understanding of the underlying tech stacks as there is lot of debugging required as not everything works as its should. Took me 2 months just to get the resizing to work and overlaying graphics still work in progress.

avaf · May 12, 2023, 11:52am

I agree, I think it is a lot of work (or rework?), but if you want to get the best performance with less CPU usage, this is the way to go.

avaf · May 13, 2023, 3:52pm

Just for fun, i was able to play mpv with hw decode (drm_prime) with the expected results for 1080p output, low CPU usage. The problem is i can’t output to a X11 (opengl es), no matter if --vo=gpu or --vo=opengl

file: vp9_4k_60_sdr.webm --vo=gpu

Title: 8K VIDEOS | World 8K Videos HDR UltraHD  (120 FPS) | Sony Demo
[vo/gpu] VT_GETMODE failed: Inappropriate ioctl for device
[vo/gpu/opengl] Failed to set up VT switcher. Terminal switching will be unavailable.
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
AO: [pulse] 48000Hz stereo 2ch float
Using hardware decoding (rkmpp).
VO: [gpu] 3840x2160 drm_prime[nv12]
[vo/gpu] Using HW-overlay mode. No GL filtering is performed on the video!
AV: 00:01:14 / 00:05:12 (24%) A-V:  0.000 Dropped: 1

Exiting... (End of file)

avaf · May 28, 2023, 2:10pm

Running with Weston, better results.

htop