Video encoding and streaming issues with Radxa CM5 and gstreamer

Hello all,
I am trying to live stream a USB webcam using Radxa CM5 with RPi4 CM4 io-board.
OS: Linux radxa-cm5-rpi-cm4-io 6.1.84-6-rk2410 #6 SMP
Camera: Logitech C270 connected over USB2.0

When I try to stream the webcam using the following gstreamer string using software H264, video feed has a huge delay as expected but is smooth without any interruptions / hitches:

./test-launch “( v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 ! x264enc ! rtph264pay name=pay0 pt=97 )”

But when I try to use the hardware encoder, the video feed is dropping frames, and the frames have some sort of visual artifacts. The string I used here is:

./test-launch “( v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 ! mpph264enc ! rtph264pay name=pay0 pt=97 )”

The log is as follows:

stream ready at rtsp://127.0.0.1:8554/test
0:00:00.579223243  3288 0xaaaafff3bcc0 WARN                     mpp gstmpp.c:121:gst_mpp_use_rga: RGA enabled
0:00:00.612141367  3288 0xffffa0010c60 WARN                    v4l2 gstv4l2object.c:4410:gst_v4l2_object_set_crop:<v4l2src0:src> VIDIOC_S_CROP failed
0:00:00.648434894  3288 0xffffa0010c60 WARN                    v4l2 gstv4l2object.c:3228:gst_v4l2_object_reset_compose_region:<v4l2src0:src> Failed to get default compose rectangle with VIDIOC_G_SELECTION: Invalid argument
0:00:03.623937968  3288 0xffffa0010f00 WARN              rtpsession gstrtpsession.c:2435:gst_rtp_session_chain_send_rtp_common:<rtpsession0> Can't determine running time for this packet without knowing configured latency
0:00:03.624451293  3288 0xaaaafff3bcc0 FIXME              rtspmedia rtsp-media.c:4622:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing
0:00:03.707274219  3288 0xffffa0010f00 WARN                    v4l2 gstv4l2object.c:4410:gst_v4l2_object_set_crop:<v4l2src1:src> VIDIOC_S_CROP failed
0:00:03.743811284  3288 0xffffa0010f00 WARN                    v4l2 gstv4l2object.c:3228:gst_v4l2_object_reset_compose_region:<v4l2src1:src> Failed to get default compose rectangle with VIDIOC_G_SELECTION: Invalid argument
0:00:04.005183475  3288 0xffffa0010c60 WARN              rtpsession gstrtpsession.c:2435:gst_rtp_session_chain_send_rtp_common:<rtpsession1> Can't determine running time for this packet without knowing configured latency
0:00:04.005480387  3288 0xaaaafff3bcc0 FIXME              rtspmedia rtsp-media.c:4622:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing
0:00:04.007455521  3288 0xaaaafff3bcc0 FIXME              rtspmedia rtsp-media.c:4622:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing
0:00:04.007484687  3288 0xaaaafff3bcc0 WARN               rtspmedia rtsp-media.c:4661:gst_rtsp_media_suspend: media 0xffffa80553b0 was not prepared
0:00:04.014181827  3288 0xaaaafff3bcc0 FIXME              rtspmedia rtsp-media.c:2900:gst_rtsp_media_seek_trickmode:<GstRTSPMedia@0xffffa80553b0> Handle going back to 0 for none live not seekable streams.
0:00:04.014852649  3288 0xffffa0010c60 WARN              rtpsession gstrtpsession.c:2435:gst_rtp_session_chain_send_rtp_common:<rtpsession1> Can't determine running time for this packet without knowing configured latency
0:00:06.869702504  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 4 - ts: 0:00:02.815922449
0:00:06.873295778  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 9 - ts: 0:00:03.151793449
0:00:13.518124947  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 1 - ts: 0:00:07.403717448
0:00:13.520568490  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 4 - ts: 0:00:07.567672448
0:00:13.523042366  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 21 - ts: 0:00:08.307669448
0:00:13.525574865  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 1 - ts: 0:00:08.375714449
0:00:13.528188739  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 26 - ts: 0:00:09.275919448
0:00:13.533403945  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 14 - ts: 0:00:09.807632740
0:00:15.128077115  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 26 - ts: 0:00:11.407607741
0:00:16.269517090  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 6 - ts: 0:00:12.543635740
0:00:18.199555607  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 32 - ts: 0:00:14.475567740
0:00:19.036063109  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 1 - ts: 0:00:15.307542740
0:00:24.685638476  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 4 - ts: 0:00:17.175573032
0:00:24.689312540  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 6 - ts: 0:00:17.407516032
0:00:24.691960538  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 35 - ts: 0:00:18.607486740
0:00:24.696727168  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 27 - ts: 0:00:19.747638740
0:00:24.708223895  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 7 - ts: 0:00:20.299537032
0:00:24.795726245  3288 0xffffa0010f00 WARN                 v4l2src gstv4l2src.c:1351:gst_v4l2src_create:<v4l2src1> lost frames detected: count = 9 - ts: 0:00:20.903417740

It is the same when using mpp265enc as well.

To debug further I tried the ffmpeg + mediamtx for RTSP from the Radxa wiki, this seems to be working, the commands are:

nohup ./mediamtx &
ffmpeg-rockchip/ffmpeg -f v4l2 -i /dev/video0 -vf “fps=30” -s 640x480 -c:v h264_rkmpp -rc_mode AVBR -b:v 20M -minrate 10M -maxrate 40M -profile:v main -level 5.1 -f rtsp rtsp://0.0.0.0:8554/stream

So the the hardware seems to be fine, looks like it is an issue with gstreamer implementations of the encoders perhaps.
Any guidance on how to resolve the issue with gstreamer would be greatly appreciated as my whole application is built upon gstreamer.

Thank You

Turn to the author of rockchip gstreamer plugin on github for help.

1 Like