So I played around with video capture a bunch more today. I got sound working fairly easily, and it sounds pretty reasonable to my ears. Here’s the gstreamer command that’s working for me:
gst-launch-1.0 -e v4l2src device=/dev/video0 ! video/x-raw,format=BGR,width=3840,height=2160 ! queue ! mpph265enc ! h265parse ! queue ! mux. alsasrc device=hw:3,0 ! audio/x-raw,channels=2 ! audioconvert ! voaacenc ! queue ! mux. matroskamux name=mux ! filesink location="/storage/hdmi4s_$(date +"%Y%m%d_%H%M%S").mkv"
I occasionally get the message that sound buffers were dropped, which is a problem I need to look into. I started with lamemp3enc and experimented with vorbis as well, but the dropped buffers message is least frequent with voaacenc.
The problem that consumed most of my time is that I’m seeing corrupt frames from time to time. sometimes it’ll be roughly every 10th frame, sometimes it’ll go a minute or two without one. And the corruption is very specific: the frame will look correct at the top, and then somewhere part of the way down it’ll switch to what appears to be an old frame from many frames ago (further back than the previous frame).
This happens even with a simple pipeline that encodes frames to h265 or h264 and dumps them directly to a file, so I suspect it’s happening in v4l2, perhaps as a result of connection / cable / interference issues. So my next step will be to try swapping out the HDMI cable a few times and see if it helps.
This issue is interesting because I can’t find anyone else reporting the same, yet I feel like I’ve seen various instances of it constantly - when working with v4l2 and webcams on a Raspberry Pi (where I would expect USB to retry if there were link issues - does HDMI do the same?), and I’ve also seen it very occasionally when working with Luxonis cameras that encode to h265 on device - it’ll show a single previous or torn frame without the weird artifacts you’d expect if the frame was encoded prior to being corrupted or reordered.
My wild guess is that these systems use a series of buffers for frames writing to them in series and then looping back to the beginning. If the writing to a particular buffer gets interrupted for some reason you’ll end up with a frame that starts with the current frame and then transitions to the last frame that was written to the buffer. So e.g. if there are 10 frame buffers and the writing gets interrupted 1/3 of the way through, you’ll end up with the current frame covering the top 1/3 of the screen and the old frame (from 10 frames back) covering the bottom 2/3 of the screen.
@Sarah have you run into anything similar? Maybe I’m just searching badly, but I’m surprised that I’m encountering this in so many different systems and yet I can’t find anyone talking about it.
