[FFmpeg] Introduce FFmpeg-Rockchip for hyper fast video transcoding via CLI

[seanp25]

Thanks again nyanmisaka! I’ve been using non-RPi SBCs for about a decade and this is the best SW implementation of a HW video encoder I’ve seen.

I often use markus-perl’s ffmpeg-build-script https://github.com/markus-perl/ffmpeg-build-script to build ffmpeg with the latest versions of all the libs (svt-av1, x265 etc.)

I made a fork of his script so that it builds ffmpeg-rockchip and it’s dependencies (libdrm, rkmpp, rkrga…), as well as all the usual ffmpeg included codecs, instead of the minimal build.

I’ll share the repo here if in case it’s of use to anyone (I’m aware that x264, x265 and probably svt-av1 might not be of much use on RK35xx devices in terms of encoding speed)

GitHub - outhud/ffmpeg-rockchip-build-script: Modified version of...

Use:
./build-ffmpeg --enable-gpl-and-non-free --build

[fdalto]

I installed the repository above
(git clone https://github.com/nyanmisaka/ffmpeg-rockchip.git)
Then when I ran the code to build it:
./build-ffmpeg --enable-gpl-and-non-free --build
But I received this message:
./build-ffmpeg: No such file or directory
It didn’t find the builder.
So I installed the original BUILDER that is in the GITHUB description:
git clone https://github.com/markus-perl/ffmpeg-build-script.git
Then, run the ./build-ffmpeg --enable-gpl-and-non-free --build
But after 4 hours of compiling it did not show the hardware acceleration libraries.
Firstly, it didn’t move the binaries to the root directory. I had to do it manually. After installed and running in PATH, when listing the encoders there is no encoder with hardware acceleration.
$ ffmpeg -hwaccels
Returns no acceleration methods!
What did I do wrong? Could you help me?

[incognito]

It seems that you’ve used a script that builds mainline ffmpeg.

[fdalto]

Yes. I did it wrong.

Now I could do all the process as told here:

Compilation

Finally I could get a FFMPEG that really uses h264_rkmpp:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_rkmpp))

YEY

But It is only possible if you type mannualy the comands in BUILD.

In my device was missing one LIB

sudo apt install libdrm-dev
Fixed the error during install.

Now ffmpeg is working propperly.

Thanks

[meadows]

Hardware decoding with rkmpp doesn’t work with newest mpv 0.39 and the commit where it breaks seems to be this: https://github.com/mpv-player/mpv/commit/4b97c23dac957747d268c71e5c38e0745f3dd255

[nyanmisaka]

Revert these two mpv commits. Not a ffmpeg-rockchip issue.

vo: hwdec: add AVHWDeviceType property to hwdecs

video: hwdec: extend look up of hwdecs to consider device type as well

[avaf]

@nyanmisaka @boogiepop

I would like to report a regression in kernel 6.1.
I have tested the latest mpp + ffmpeg + rga with kernel 6.1 and found some issues:

1 - Decoding, seems to have some problems with alignment (they are back), tested jellyfish-15-mbps-hd-h264.mkv (hevc and 10-bit as well, same problem)

2 - Encoding seems to have a problem with dma-buf, please refer to this ( https://github.com/rockchip-linux/mpp/issues/253 ) but now with NV12

Please, note i tested the same with gstreamer / mpi and it can encode and decode without problem, just to make sure it is not mpp and rga issue.

[boogiepop]

can not test since i burned my device but that 1 px is interesing… how is rga involved in this decoding process?

[avaf]

sorry, do not consider my observation.

I think RGA is involved. My guess only.

gstreamer works forcing RGA:

DISPLAY=:0.0 GST_VIDEO_CONVERT_USE_RGA=1 gst-launch-1.0 filesrc location=./jellyfish-15-mbps-hd-h264.mkv ! matroskademux ! h264parse ! mppvideodec ! waylandsink

[boogiepop]

well its RGA, nobody is responsible for its bugs, but yeah that 1 px is funny, because i would expect the green offset in the same y offset and not finish in the half of the stride, yet i am also speculating.

[avaf]

the old rkmpp works fine in kernel 6.1, and it use RGA, if i recall correctly you fix alignment issues in FFmpeg-encoder back at that time.

[boogiepop]

What do you mean old rkmpp? Old version of mpp?

[avaf]

Jeffy’s code rkmpp.c

[boogiepop]

Ahh that old one, yes but all of those fixes are also in ffmpeg-rockchip. It is no more in a comparible form with ffmpeg-rockchip, even though it works, it utilized lots lots of hacks and was basically doing everthing itself without relying ffmpeg, rga nor mpp, may be thats the reason it works so better not use that even though it works.

[avaf]

Maybe the hstride is wrong. It is a pity you burned your board, but i hope @nyanmisaka can reproduce it.

[boogiepop]

Btw the 2nd picture in the encoding artifacts seems like a dma sync issue, speculatively with rga

[avaf]

To reproduce:

~/rockchip/ffmpeg/ffmpeg-rockchip/ffmpeg -f v4l2 -input_format nv12 -framerate 30 -video_size 1920x1080 -i /dev/video11 -c:v h264_rkmpp -qp_init 22 -movflags frag_keyframe+empty_moov+faststart -b:v 4000K -vprofile main -level:v 4.2 -vf format=yuv420p -r 25 -bufsize 600k cam-1.h264 -y

and then play back the stream.

[nyanmisaka]

In my tests, ffmpeg received 1080p nv12 input from the following sources, and after applying the software format=yuv420p filter, it encoded fine and there is no green line on my end.

The versions of mpp and librga used are

• https://github.com/nyanmisaka/mpp/commits/jellyfin-mpp/
• https://github.com/nyanmisaka/rk-mirrors/commits/jellyfin-rga/

1. 1080p video files (mp4, mkv, raw h264, raw hevc…)

# Prepare
cd ~/
curl -OL https://repo.jellyfin.org/jellyfish/media/jellyfish-15-mbps-hd-h264.mkv

# Software Decode H.264 file (yuv420p) -> Hardware Encode
ffmpeg -i ~/jellyfish-15-mbps-hd-h264.mkv -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/1.mp4

# Hardware Decode H.264 file (nv12) -> RGA2 convert (yuv420p) -> Hardware Encode
ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime -i ~/jellyfish-15-mbps-hd-h264.mkv \
-vf scale_rkrga=format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/2.mp4

2. FFmpeg’s built-in test sources at 1080p -f lavfi testsrc=s=1920x1080,format=nv12

# Raw YUV from lavfi (yuv420p) -> Hardware Encode
ffmpeg -f lavfi -i testsrc=s=1920x1080,format=yuv420p -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/3.mp4

# Raw YUV from lavfi (nv12) -> Software Convert (yuv420p) -> Hardware Encode
ffmpeg -f lavfi -i testsrc=s=1920x1080,format=nv12 -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/4.mp4

3. 1080p Raw YUV files

# Prepare
ffmpeg -f lavfi -i testsrc=s=1920x1080,format=nv12 -t 5 -y ~/nv12.yuv
ffmpeg -f lavfi -i testsrc=s=1920x1080,format=yuv420p -t 5 -y ~/yuv420p.yuv

# Raw YUV file (yuv420p) -> Hardware Encode
ffmpeg -pix_fmt yuv420p -s 1920x1080 -i ~/yuv420p.yuv -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/5.mp4

# Raw YUV file (nv12) -> Software Convert (yuv420p) -> Hardware Encode
ffmpeg -pix_fmt nv12 -s 1920x1080 -i ~/nv12.yuv -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/6.mp4

4. 1080p HDMI RX input
   (Can’t test as of writing this, but I tested it in both 1080p and 4k not long ago)

According to searching, /dev/video11 seems to be RK-ISP used by the camera. I don’t have the hardware to verify it, and can’t say for sure what quirks it contains.

[avaf]

yep, NV12 from a MIPI camera. I don’t know if 12-bit modes (MEDIA_BUS_FMT_SRGGB12_1X12) could trigger this. Note: i think 2016x1080 is 10-bit mode.

I will try to investigate a bit more.

But some feedback:

-- mpp latest commit --
Author: Yanjun Liao <yanjun.liao@rock-chips.com>
Date:   Wed Jul 24 15:47:54 2024 +0800

    fix[265e]:Fix the st refernce frame err in tsvc
    
    This is a bug caused by the mark and use of ltr frames,
    when presence of multiple short temporal refenence frame,
    may lead to errors in reference relationships.
    
    Change-Id: I1962d81e39b704086a51b4e4098ba3feb64c47c6
    Signed-off-by: Yanjun Liao <yanjun.liao@rock-chips.com>

-- rga-multi latest commit --
Author: Yu Qiaowei <cerf.yu@rock-chips.com>
Date:   Thu Aug 29 15:01:34 2024 +0800

    normal: fix wrong full_csc_clip size in memcpy
    
    This causes the gauss mode to be checked when using full_csc with driver
    versions 1.3.5 and above.
    
    update to 1.10.1_[3]
    
    Signed-off-by: Yu Qiaowei <cerf.yu@rock-chips.com>
    Change-Id: I40c3c503c68a77f2c435a8932de0195b447d1d46

-- ffmpeg-rockchip latest commit --
Author: nyanmisaka <nst799610810@gmail.com>
Date:   Wed Oct 23 21:42:03 2024 +0800

    fixup! lavf/rkrga: add RKRGA scale, vpp and overlay filter
    
    fix nv24/nv42 check on rga2p
    
    Signed-off-by: nyanmisaka <nst799610810@gmail.com>

# Software Decode H.264 file (yuv420p) -> Hardware Encode ffmpeg -i ~/jellyfish-15-mbps-hd-h264.mkv -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/1.mp4

works, but dma-buf issue:

# Hardware Decode H.264 file (nv12) -> RGA2 convert (yuv420p) -> Hardware Encode ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime -i ~/jellyfish-15-mbps-hd-h264.mkv \ -vf scale_rkrga=format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/2.mp4

[AVFilterGraph @ 0xaaaae71af360] No such filter: ‘scale_rkrga’

Have i missed some build parameters?

1080p Raw YUV files

works, but dma-buf issue.

Update: the dma-buf issue is only when rendering on screen (ffplay)

[nyanmisaka]

In the RK 6.1 kernel, their custom dma-heap driver was removed. It was replaced with the upstream dma-heap driver, which might be the problem. The patch added in my MPP branch will use the DRM allocator instead of dma-heap by default.

fix[allocator]: Fix validating dma_heap drivers

As for the missing scale_rkrga filter, check if --enable-rkrga is configured in your FFmpeg.

--enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga

Compilation