[FFmpeg] Introduce FFmpeg-Rockchip for hyper fast video transcoding via CLI

fertilizer · August 25, 2024, 1:57am

its works, thanks nayanmisaka!

here is my setup fully command.

# using Ubuntu 24.04 (Noble) 

sudo apt update -y
sudo apt upgrade -y
sudo apt install git meson cmake pkg-config gcc libasound2-dev -y
sudo aptitude install libdrm-dev
# n
# y
# y

build package

Build MPP

mkdir -p ~/dev && cd ~/dev
git clone -b jellyfin-mpp --depth=1 https://github.com/nyanmisaka/mpp.git rkmpp
pushd rkmpp
mkdir rkmpp_build
pushd rkmpp_build
cmake \
    -DCMAKE_INSTALL_PREFIX=/usr \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=ON \
    -DBUILD_TEST=OFF \
    ..
sudo make -j $(nproc)
sudo make install

Build RGA

mkdir -p ~/dev && cd ~/dev
git clone -b jellyfin-rga --depth=1 https://github.com/nyanmisaka/rk-mirrors.git rkrga
meson setup rkrga rkrga_build \
    --prefix=/usr \
    --libdir=lib \
    --buildtype=release \
    --default-library=shared \
    -Dcpp_args=-fpermissive \
    -Dlibdrm=false \
    -Dlibrga_demo=false
meson configure rkrga_build
ninja -C rkrga_build install
# q
# y

Build the FFmpeg with alsa audio

mkdir -p ~/dev && cd ~/dev
git clone --depth=1 https://github.com/nyanmisaka/ffmpeg-rockchip.git ffmpeg
cd ffmpeg
./configure --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga --enable-alsa
sudo make -j $(nproc)

Install FFmpeg

# Try the compiled FFmpeg without installation
./ffmpeg -decoders | grep rkmpp
./ffmpeg -encoders | grep rkmpp
./ffmpeg -filters | grep rkrga
# Install FFmpeg to the prefix path
sudo make install

fertilizer · August 25, 2024, 2:08am

BTW, has anyone successfully recorded video and audio via HDMI-in?

I’m using the following command. It works, but I always get warnings and experience some audio skipping:

arecord -D hw:0,0 -f cd | ffmpeg -re -i /dev/video0 -f wav -i pipe:0 -c:v h264_rkmpp -maxrate 1900k -bufsize 3968k -vf "scale=1920:1080" -c:a aac -b:a 128k -af "volume=7.0" -f flv "rtmp://your_url

The response I get is:

overrun!!! (at least 12.465 ms long)57250kB time=00:51:43.34 bitrate=1471.0kbits/s speed=   1x    
overrun!!! (at least 134.581 ms long)8560kB time=00:51:50.38 bitrate=1471.1kbits/s speed=   1x    
overrun!!! (at least 42.422 ms long)59810kB time=00:51:56.95 bitrate=1471.3kbits/s speed=   1x    
overrun!!! (at least 29.663 ms long)60949kB time=00:52:02.99 bitrate=1471.4kbits/s speed=   1x    
overrun!!! (at least 108.499 ms long)2188kB time=00:52:09.53 bitrate=1471.6kbits/s speed=   1x    
overrun!!! (at least 28.869 ms long)63422kB time=00:52:16.13 bitrate=1471.7kbits/s speed=   1x

seanp25 · August 25, 2024, 9:28am

Thanks again nyanmisaka! I’ve been using non-RPi SBCs for about a decade and this is the best SW implementation of a HW video encoder I’ve seen.

I often use markus-perl’s ffmpeg-build-script https://github.com/markus-perl/ffmpeg-build-script to build ffmpeg with the latest versions of all the libs (svt-av1, x265 etc.)

I made a fork of his script so that it builds ffmpeg-rockchip and it’s dependencies (libdrm, rkmpp, rkrga…), as well as all the usual ffmpeg included codecs, instead of the minimal build.

I’ll share the repo here if in case it’s of use to anyone (I’m aware that x264, x265 and probably svt-av1 might not be of much use on RK35xx devices in terms of encoding speed)

Use:
./build-ffmpeg --enable-gpl-and-non-free --build

fdalto · September 3, 2024, 5:59pm

I installed the repository above
(git clone https://github.com/nyanmisaka/ffmpeg-rockchip.git)
Then when I ran the code to build it:
./build-ffmpeg --enable-gpl-and-non-free --build
But I received this message:
./build-ffmpeg: No such file or directory
It didn’t find the builder.
So I installed the original BUILDER that is in the GITHUB description:
git clone https://github.com/markus-perl/ffmpeg-build-script.git
Then, run the ./build-ffmpeg --enable-gpl-and-non-free --build
But after 4 hours of compiling it did not show the hardware acceleration libraries.
Firstly, it didn’t move the binaries to the root directory. I had to do it manually. After installed and running in PATH, when listing the encoders there is no encoder with hardware acceleration.
$ ffmpeg -hwaccels
Returns no acceleration methods!
What did I do wrong? Could you help me?

incognito · September 4, 2024, 3:45pm

It seems that you’ve used a script that builds mainline ffmpeg.

fdalto · September 4, 2024, 9:33pm

Yes. I did it wrong.

Now I could do all the process as told here:

Finally I could get a FFMPEG that really uses h264_rkmpp:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_rkmpp))

YEY

But It is only possible if you type mannualy the comands in BUILD.

In my device was missing one LIB

sudo apt install libdrm-dev
Fixed the error during install.

Now ffmpeg is working propperly.

Thanks

meadows · October 2, 2024, 12:15pm

Hardware decoding with rkmpp doesn’t work with newest mpv 0.39 and the commit where it breaks seems to be this: https://github.com/mpv-player/mpv/commit/4b97c23dac957747d268c71e5c38e0745f3dd255

nyanmisaka · October 3, 2024, 2:12pm

Revert these two mpv commits. Not a ffmpeg-rockchip issue.

avaf · November 11, 2024, 11:25pm

@nyanmisaka @boogiepop

I would like to report a regression in kernel 6.1.
I have tested the latest mpp + ffmpeg + rga with kernel 6.1 and found some issues:

1 - Decoding, seems to have some problems with alignment (they are back), tested jellyfish-15-mbps-hd-h264.mkv (hevc and 10-bit as well, same problem)

2 - Encoding seems to have a problem with dma-buf, please refer to this ( mpp encoding YUYV to H264 / H265 (RK3568 / RK3588) · Issue #253 · rockchip-linux/mpp · GitHub ) but now with NV12

Please, note i tested the same with gstreamer / mpi and it can encode and decode without problem, just to make sure it is not mpp and rga issue.

boogiepop · November 11, 2024, 11:34pm

can not test since i burned my device but that 1 px is interesing… how is rga involved in this decoding process?

avaf · November 11, 2024, 11:39pm

sorry, do not consider my observation.

I think RGA is involved. My guess only.

gstreamer works forcing RGA:

DISPLAY=:0.0 GST_VIDEO_CONVERT_USE_RGA=1 gst-launch-1.0 filesrc location=./jellyfish-15-mbps-hd-h264.mkv ! matroskademux ! h264parse ! mppvideodec ! waylandsink

boogiepop · November 11, 2024, 11:42pm

well its RGA, nobody is responsible for its bugs, but yeah that 1 px is funny, because i would expect the green offset in the same y offset and not finish in the half of the stride, yet i am also speculating.

avaf · November 11, 2024, 11:49pm

the old rkmpp works fine in kernel 6.1, and it use RGA, if i recall correctly you fix alignment issues in FFmpeg-encoder back at that time.

boogiepop · November 11, 2024, 11:52pm

What do you mean old rkmpp? Old version of mpp?

avaf · November 11, 2024, 11:53pm

Jeffy’s code rkmpp.c

boogiepop · November 12, 2024, 12:03am

Ahh that old one, yes but all of those fixes are also in ffmpeg-rockchip. It is no more in a comparible form with ffmpeg-rockchip, even though it works, it utilized lots lots of hacks and was basically doing everthing itself without relying ffmpeg, rga nor mpp, may be thats the reason it works so better not use that even though it works.

avaf · November 12, 2024, 12:10am

Maybe the hstride is wrong. It is a pity you burned your board, but i hope @nyanmisaka can reproduce it.

boogiepop · November 12, 2024, 12:19am

Btw the 2nd picture in the encoding artifacts seems like a dma sync issue, speculatively with rga

avaf · November 12, 2024, 12:28am

To reproduce:

~/rockchip/ffmpeg/ffmpeg-rockchip/ffmpeg -f v4l2 -input_format nv12 -framerate 30 -video_size 1920x1080 -i /dev/video11 -c:v h264_rkmpp -qp_init 22 -movflags frag_keyframe+empty_moov+faststart -b:v 4000K -vprofile main -level:v 4.2 -vf format=yuv420p -r 25 -bufsize 600k cam-1.h264 -y

and then play back the stream.

nyanmisaka · November 12, 2024, 7:13am

In my tests, ffmpeg received 1080p nv12 input from the following sources, and after applying the software format=yuv420p filter, it encoded fine and there is no green line on my end.

The versions of mpp and librga used are

1080p video files (mp4, mkv, raw h264, raw hevc…)

# Prepare
cd ~/
curl -OL https://repo.jellyfin.org/jellyfish/media/jellyfish-15-mbps-hd-h264.mkv

# Software Decode H.264 file (yuv420p) -> Hardware Encode
ffmpeg -i ~/jellyfish-15-mbps-hd-h264.mkv -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/1.mp4

# Hardware Decode H.264 file (nv12) -> RGA2 convert (yuv420p) -> Hardware Encode
ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime -i ~/jellyfish-15-mbps-hd-h264.mkv \
-vf scale_rkrga=format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/2.mp4

FFmpeg’s built-in test sources at 1080p -f lavfi testsrc=s=1920x1080,format=nv12

# Raw YUV from lavfi (yuv420p) -> Hardware Encode
ffmpeg -f lavfi -i testsrc=s=1920x1080,format=yuv420p -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/3.mp4

# Raw YUV from lavfi (nv12) -> Software Convert (yuv420p) -> Hardware Encode
ffmpeg -f lavfi -i testsrc=s=1920x1080,format=nv12 -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/4.mp4

1080p Raw YUV files

# Prepare
ffmpeg -f lavfi -i testsrc=s=1920x1080,format=nv12 -t 5 -y ~/nv12.yuv
ffmpeg -f lavfi -i testsrc=s=1920x1080,format=yuv420p -t 5 -y ~/yuv420p.yuv

# Raw YUV file (yuv420p) -> Hardware Encode
ffmpeg -pix_fmt yuv420p -s 1920x1080 -i ~/yuv420p.yuv -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/5.mp4

# Raw YUV file (nv12) -> Software Convert (yuv420p) -> Hardware Encode
ffmpeg -pix_fmt nv12 -s 1920x1080 -i ~/nv12.yuv -vf format=yuv420p -c:v h264_rkmpp -qp_init 22 -y /tmp/6.mp4

1080p HDMI RX input
(Can’t test as of writing this, but I tested it in both 1080p and 4k not long ago)

According to searching, /dev/video11 seems to be RK-ISP used by the camera. I don’t have the hardware to verify it, and can’t say for sure what quirks it contains.