Glitchy video capture

#1

I’m capturing video from a Sony A7s sending 4k30 video to the HDMI input of the Rock 5b, but regardless of encoding I’m strange frames that are a mix of a current frame and an old frame.

Has anyone else encountered a similar issue? Or does anyone have a guess as to what might be going on?

Here’s the pipeline I’m using (though the problem also presents itself if I just encode frames and dump them to a file; I’ve tried mpph265, mpph264, and mppjpeg):

gst-launch-1.0 -e v4l2src device=/dev/video0 ! video/x-raw,format=BGR,width=3840,height=2160 ! queue ! mpph265enc ! h265parse ! queue ! mpegtsmux ! filesink location="/storage/hdmi4k_$(date +"%Y%m%d_%H%M%S").ts" sync=false

And here’s a sample video, recording a stopwatch (on an iPhone) moving back and forth to accentuate the jumps: https://www.youtube.com/watch?v=A2TYCEBN-tA

Any help would be much appreciated, I don’t have any leads at this point. I’ve tried:

  • Swapping HDMI cables
  • Trying a different camera
  • Different ways of mapping buffers into memory in v4l2src
  • All three mpp encoders
HDMI Input: 4K negotiation loop?
#2

I would suggest a few things:

  • Try to use NV12 instead of BGR, and see if you can set it to capture NV12, i think if the device > 1080p it would be possible

  • Output the image to video without the encoding process and see if it is smooth

  • Add io-mode=dmabuf

#3

Hi avaf, thank you so much for your response!

Try to use NV12 instead of BGR, and see if you can set it to capture NV12, i think if the device > 1080p it would be possible

When I try using NV12 (or NV16 or NV24 with ! videoconvert) I get this error:

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Device '/dev/video0' has no supported format Additional debug info: ../sys/v4l2/gstv4l2object.c(3994): gst_v4l2_object_set_format_full (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Call to TRY_FMT failed for NV12 @ 3840x2160: Invalid argument

Full output below [0]. I believe this is because the input is BGR. Full timing info below [1]. This is what the camera sends if my EDID is 4k-300mhz or 4k-600mhz; if I set the EDID to 4k-170mhz the camera will simply refuse to send 4k at all (the 4k HDMI option in menu is greyed out). So AFAICT v4l2 won’t let me use NV12 unless the input is NV12, and the camera only seems to want to send BGR.

I also tried putting videoconvert ! video/x-raw,format=NV12 ! between the capture and encoding, but this only produced a few frames in the 30 seconds I let it run.

Output the image to video without the encoding process and see if it is smooth

I assume you mean to the screen (since the raw data is ~8gbps and far exceeds the write capabilities of any available media) - I’ve tried that with ximagesink, xvimagesinx, and glimagesink.

I’m using the Debian image, and the former two create windows that are visible on the task bar and that draw large but empty frames; no video is visible. The latter - glimagesink - was giving an error about a missing library, but now (after updating last night) gives:

arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'. ERROR: from element /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink: Failed to bind OpenGL API: EGL_BAD_PARAMETER Additional debug info: ../ext/gl/gstglimagesink.c(1095): _ensure_gl_setup (): /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink

I can try to run this new error down.

Add io-mode=dmabuf

The issue still occurs with io-mode=dmabuf. io-mode 2 and 4 work (1, 3, and 5 result in ‘not negotiated’) but both show the issue.

Thanks again for your help! Let me know if you have any other ideas, otherwise I’ll see if I can get glimagesink working when I find some free time.

 
 

[0] Full output of Gstreamer with NV12

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Device '/dev/video0' has no supported format
Additional debug info:
../sys/v4l2/gstv4l2object.c(3994): gst_v4l2_object_set_format_full (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Call to TRY_FMT failed for NV12 @ 3840x2160: Invalid argument
Execution ended after 0:00:00.001284167
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Freeing pipeline ...

[1] Full timing info

#! /bin/bash

v4l2-ctl -d /dev/video0 --get-fmt-video
v4l2-ctl -d /dev/video0 --get-dv-timings
rock@rock-5b:~$ ./check_hdmi.sh
Format Video Capture Multiplanar:
        Width/Height      : 3840/2160
        Pixel Format      : 'BGR3' (24-bit BGR 8-8-8)
        Field             : None
        Number of planes  : 1
        Flags             : premultiplied-alpha, 0x000000fe
        Colorspace        : Unknown (0x433dab80)
        Transfer Function : Default
        YCbCr/HSV Encoding: Unknown (0x000000ff)
        Quantization      : Default
        Plane 0           :
           Bytes per Line : 11520
           Size Image     : 24883200
DV timings:
        Active width: 3840
        Active height: 2160
        Total width: 4400
        Total height: 2250
        Frame format: progressive
        Polarities: -vsync -hsync
        Pixelclock: 296696000 Hz (29.97 frames per second)
        Horizontal frontporch: 176
        Horizontal sync: 88
        Horizontal backporch: 296
        Vertical frontporch: 8
        Vertical sync: 10
        Vertical backporch: 72
        Standards:
        Flags:
#4

Try something like this:

sudo v4l2-ctl --set-fmt-video=width=3840,height=2160,pixelformat=NV12 -d /dev/video0
sudo gst-launch-1.0 v4l2src device=/dev/video0 ! queue ! video/x-raw,format=NV12,width=3840,height=2160 ! xvimagesink

#5

This is going to be very dependent on what the output device decides to send, which is determined based on EDID.

@Nathan_Bigelow Check:
v4l2-ctl --get-edid | edid-decode
And look for:

Block 1, CTA-861 Extension Block:
  Revision: 3
  Underscans IT Video Formats by default
  Basic audio support
  Supports YCbCr 4:4:4
  Supports YCbCr 4:2:2

Or something similar. If you don’t have the Supports YCbCr 4:4:4 or Supports YCbCr 4:4:2 line:

v4l2-ctl --get-edid > edid.txt
v4l2-ctl --set-edid file=edid.txt,format=hex,ycbcr444 # And/or ycbcr422

will make those available for the remote device to select. (In this case, since you’re an HDMI sink, you basically only get to say “I support this” and have to take whatever the source device provides.)

Those edit the existing EDID - if you’re using the EDIDs that v4l2-ctl can generate, then add ,ycbcr444 or such to the end of the options to v4l2-ctl --set-edid.

#6

@avaf I’m getting “invalid argument” for the first command you suggested:

rock@rock-5b:~$ sudo v4l2-ctl --set-fmt-video=width=3840,height=2160,pixelformat=NV12 -d /dev/video0
VIDIOC_S_FMT: failed: Invalid argument

I’ll see if I can get the display to work with something similar to your second command.

#7

Thanks @Sarah! I’ll see if I can coax the camera into sending an NV12 signal with the EDID magic you suggested.

Interestingly, though, I tried sending the signal to a 4k-capable field monitor and then to the capture card. The video looked good on the field monitor, though the glitches could still be there if it’s handling them better (e.g. by skipping bad frames entirely). The signal that reached the rock 5 was NV16 and showed the same glitchy issue.

#8

Looks like you can’t ask the device to send NV12.
This will display the captured video on your screen:

sudo gst-launch-1.0 v4l2src device=/dev/video0 ! queue ! video/x-raw,format=BGR,width=3840,height=2160 ! videoconvert ! ‘video/x-raw,format=NV12’ ! xvimagesink

if xvimagesink does not work, try glimagesink, but you need a desktop with gles2 enabled.
And the correct way to use hardware acceleration is like this:

sudo GST_GL_API=gles2 GST_GL_PLATFORM=egl gst-launch-1.0 v4l2src device=/dev/video0 ! queue ! ‘video/x-raw,format=BGR,width=3840,height=2160’ ! glimagesink

If you don’t see the glitches on your screen the problem is the encoder, which is not a surprise. There are many versions and fixes available from Rockchip, finding one that works with 4k is the key.

1 Like
#9

@avaf thank you, that command works to show the video on the screen! It’s showing about 1 FPS, but so far no visible issues. I’ll keep watching.

I also played around with the pipeline earlier today and found that dropping about half the frames by using ! videorate ! video/x-raw,framerate=15/1 dramatically cut down on the issue, which also led me to suspect the issue was with the encoder. Similarly increasing the frame rate beyond 30 FPS rapidly created more issues. So I played around with options and found that setting zero-copy-pkt=false seems to reduce the frequency of the issue by 5-10x.

If you were looking for other versions of the encoder to try, where would you look? Thanks again for your help!

#10

You can find the latest version here:

There is also a fix for flickering that could be the problem:

If you wish you can try this small tool (Debian 11) that displays the contents of the camera or any v4l2 device on the screen with few resources (~5% CPU usage). It uses DRM, so you need to stop X11 and be in a console (CLI) or ssh session. It displays the contents on HDMI-1 only.

  • Unzip the capture tool and make it exec

    sudo chmod +x capture

  • Stop X11, log in to an ssh session, or type in a cmd line

    sudo systemctl stop lightdm

  • ALT+F1 and log in to CLI.

  • Run, this will display 2000 frames coming from your camera to HDMI-1, adjust to a higher number of frames so you can observe any glitches

[sudo] ./capture -d /dev/video0 -f NV16 -c 2000 -s 3840x2160
HDMI-A-1 -> connected
HDMI-A-2 -> not connected
DP-1 -> not connected
rga_api version 1.8.1_[4]

You can have a PrintScreen (png file) at the 100th frame (that gives you true FPS) with this command:

[sudo] ./capture -d /dev/video0 -f NV16 -K 100 -c 2000 -s 3840x2160

If your device is BGR3

[sudo] ./capture -d /dev/video0 -f BGR3 -K 100 -c 2000 -s 3840x2160

If your device is NV12

[sudo] ./capture -d /dev/video0 -f NV12 -K 100 -c 2000 -s 3840x2160

The screenshot file will be:

screenshot_3840x2160.png

Hope it helps

capture.zip (24.7 KB)

4K camera not working on ROCK5C
#12

Cool, I haven’t tested it yet but it sounds promising!
What about input lag?
Are the sources available?