[FFmpeg] Introduce FFmpeg-Rockchip for hyper fast video transcoding via CLI

Yes. It is used automatically.

1 Like

Oh derp i thought i already installed the libraries oops…

What was the way to view the ffmpeg in real time instead of it encoding onto my desktop for computer vision?

P.S your a life saver.

If viewing the video in real time is enough, then use ffplay (this method is not the most efficient because ffplay does not support zero-copy).

ffplay -vcodec h264_rkmpp -i /path/to/video

For a better playback experience, you can use MPV or Kodi instead.

1 Like

If you want to see on real time see Avafinger’s thread, look for the last replies on it

https://forum.radxa.com/t/object-detection-with-npu-h264-streams-h264-camera-rtsp

I don’t know any way of doing it via CLI. But I’ve found it easier to do with Gstreamer

gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! rkximagesink sync=false

I’ll try 4K encoding later, really excited to try it out. If I can get at least 15fps I’ll be very happy

Got 20 FPS with 3840x2160 on RK3566 with HEVC encoding. I’m very sastified

Good result. If you need more, the RK3588 can give you 4K 120FPS.

1 Like

My custom board has RK3566 on it. I am fully committed at this point. Besides I think 20fps is good enough

1 Like

Hi @nyanmisaka,

I was wondering if with your ffmpeg version is possible to do the following using a RK3588 board.

I want to receive either a SRT or RTMP stream, add an html overlay (scoreboard) and restream back to SRT or RTMP(S)

Of course ffmpeg doesn’t support html overlay, but I see through rga filter, it is possible to blend an image.

Now, the key here would be the possibility to “refresh” say every second the image to be blended in the ffmpeg pipe, and in parallel, run a headless browser instance that could just “capture” the image also every second.

Not very elegant for sure, but…

The other alternative I’m aware of would be RidgeRun’s HTML overlay plugin for gstreamer, but is expensive and doesn’t support RK3588.

So, would this be possible?

Thank you

As far as I know FFmpeg cannot interpret HTML/JS text into RGBA images. Both the software filter overlay and the hardware filter overlay_rkrga require two images as input and another image as output. The output can be fed to the hardware encoder and streamed through RTMP.

What you should do is render the HTML/JS text to RGBA images using an arbitrary method and pipe it to FFmpeg CLI as an overlay input. In addition, you also need to ensure that the two inputs have matching timestamps or framerates, otherwise they will become out of sync.

ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -i "rtmp://..." \
-f rawvideo -s 1920x1080 -pix_fmt rgba -i - \
-filter_complex "[1:v]hwupload[html];[0:v][html]overlay_rkrga=format=nv12:afbc=1:eof_action=pass:repeatlast=0" \
-c:v h264_rkmpp -b:v 6M -maxrate 6M -g:v 250 \
-c:a copy -sn -dn \
-f rtmp "rtmp://..."

If you can use FFmpeg through libav* libraries, then you only need to wrap each rendered RGBA image into an AVFrame and blend it with another AVFrame decoded by the decoder from RTMP/H264 input. Don’t forget to use hwupload filter (sw frame -> hw frame, required by rkrga hw filters).

Alternatively, if you are familiar with libavfilter, you can integrate any c/cpp-friendly HTML renderer to FFmpeg as a filter/vsrc. IMO this is probably the most elegant solution.

Yes, I’m aware ffmpeg is not able to directly process html content. That is why I said, a headless browser is running as a different process and “capturing” an image every whatever time (suggested 1 second for very simple case)

But the key is, as you say, I need to blend two images, one will come from the video source, and the other is just generated by that capture process.

But can I make this “dynamic”, I mean, make ffmpeg use “a refresh” every second?

As for potential out of sync situations, is not that relevant. Ffmpeg will use the same “to be blended” image for the whole second. In that time should have plenty of time to generate the “next” image.

Perhaps the second way in this answer is feasible. https://stackoverflow.com/questions/64043336/ffmpeg-pipe-input-for-concat

Hi. I was testing the ffmpeg-rockchip project on my RK3568 board. But I found out that aftering compiling files in ffmpeg/doc/examples/ dir. Then type

‘sudo ./hw_decode rkmpp /path/to/video.mp4 rawfile’

And I got ‘Decoder h264 does not support device type rkmpp.’ this error. Which I believe something went wrong in avcodec_get_hw_config(decoder, i), and value of i seems to be 0.
And not just rkmpp doesn’t work, any other hw_type_names[] don’t work too.
While typing

‘./ffmpeg -stream_loop -1 -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -i /path/to/any-h264-video.mp4 -an -sn -vframes 5000 -f null -’

it works fine.

Could you please give me some hint? Is there something Im not doing right?

Hello, this example program only supports ffmpeg’s built-in hw accelerators instead of hw decoders.

For example, Intel QSV (not VAAPI), Nvidia CUVID (not NVDEC) and RKMPP are all hw decoders instead of hw accelerators. ffmpeg uses a trick to make both accels and decoders share the -hwaccel command.

Thank you for your reply! So is it possible if I just use ffmpeg api to decoding and transcoding(using mpp and rga unit) in my own project? Or I would probably have to use mpp and rga lib api instead?

Of course you can. ffmpeg itself uses the libav* API, and you can too, but you can’t use the paths in the hw_decode example.