[FFmpeg] Introduce FFmpeg-Rockchip for hyper fast video transcoding via CLI

#1

Through this ffmpeg-rockchip fork, users can obtain the fastest hardware transcoding and filtering speed on the Rockchip platform via CLI (command line). And the code style has been kept as consistent as possible with the upstream FFmpeg, making it easy to read and maintain.

This project was originally part of jellyfin/jellyfin-ffmpeg, but I have now separated it for community use and issue collection.

ffmpeg-rockchip

This project aims to provide full hardware transcoding pipeline in FFmpeg CLI for Rockchip platforms that support MPP (Media Process Platform) and RGA (2D Raster Graphic Acceleration). This includes hardware decoders, encoders and filters. A typical target platform is RK3588/3588s based devices.

Hightlights

  • MPP decoders support up to 8K 10-bit H.264, HEVC, VP9 and AV1 decoding
  • MPP decoders support producing AFBC (ARM Frame Buffer Compression) image
  • MPP decoders support de-interlace using IEP (Image Enhancement Processor)
  • MPP decoders support allocator half-internal and pure-external modes
  • MPP encoders support up to 8K H.264 and HEVC encoding
  • MPP encoders support async encoding, AKA frame-parallel
  • MPP encoders support consuming AFBC image
  • RGA filters support image scaling and pixel format conversion
  • RGA filters support image cropping
  • RGA filters support image transposing
  • RGA filters support blending two images
  • RGA filters support async operation
  • RGA filters support producing and consuming AFBC image
  • Zero-copy DMA in above stages

How to use

The documentation is available on the Wiki page of this project.

Codecs and filters

Decoders/Hwaccel

 V..... av1_rkmpp            Rockchip MPP (Media Process Platform) AV1 decoder (codec av1)
 V..... h263_rkmpp           Rockchip MPP (Media Process Platform) H263 decoder (codec h263)
 V..... h264_rkmpp           Rockchip MPP (Media Process Platform) H264 decoder (codec h264)
 V..... hevc_rkmpp           Rockchip MPP (Media Process Platform) HEVC decoder (codec hevc)
 V..... mpeg1_rkmpp          Rockchip MPP (Media Process Platform) MPEG1VIDEO decoder (codec mpeg1video)
 V..... mpeg2_rkmpp          Rockchip MPP (Media Process Platform) MPEG2VIDEO decoder (codec mpeg2video)
 V..... mpeg4_rkmpp          Rockchip MPP (Media Process Platform) MPEG4 decoder (codec mpeg4)
 V..... vp8_rkmpp            Rockchip MPP (Media Process Platform) VP8 decoder (codec vp8)
 V..... vp9_rkmpp            Rockchip MPP (Media Process Platform) VP9 decoder (codec vp9)

Encoders

 V..... h264_rkmpp           Rockchip MPP (Media Process Platform) H264 encoder (codec h264)
 V..... hevc_rkmpp           Rockchip MPP (Media Process Platform) HEVC encoder (codec hevc)
 V..... mjpeg_rkmpp          Rockchip MPP (Media Process Platform) MJPEG encoder (codec mjpeg)

Filters

 ... overlay_rkrga     VV->V      Rockchip RGA (2D Raster Graphic Acceleration) video compositor
 ... scale_rkrga       V->V       Rockchip RGA (2D Raster Graphic Acceleration) video resizer and format converter
 ... vpp_rkrga         V->V       Rockchip RGA (2D Raster Graphic Acceleration) video post-process (scale/crop/transpose)

Important

  • Rockchip BSP/vendor kernel is necessary, 5.10 and 6.1 are two tested versions.
  • For the supported maximum resolution and FPS you can refer to the datasheet or TRM.
  • User MUST be granted permission to access these device files.
# DRM allocator
/dev/dri

# DMA_HEAP allocator
/dev/dma_heap

# RGA filters
/dev/rga

# MPP codecs
/dev/mpp_service

# Optional, for compatibility with older kernels and socs
/dev/iep
/dev/mpp-service
/dev/vpu_service
/dev/vpu-service
/dev/hevc_service
/dev/hevc-service
/dev/rkvdec
/dev/rkvenc
/dev/vepu
/dev/h265e

Todo

  • Support MPP MJPEG/JPEG decoder
  • Support MPP VP8 video encoder
  • …

Acknowledgments

@hbiyik @HermanChen @rigaya

12 Likes
#2

Thank you and all other contributors for this nice package. I was already checking it out in jellyfin and it’s working great. Big step forward for the board!!

1 Like
#3

From code quality pov,this is actually quite mainlinable. Do you think it makes sense to pr this to ffmpeg?

#4

Good job, I am looking forward to testing this when you push the code:

  • Support MPP MJPEG/JPEG decoder and encoder
#5

I made a dedicated post for your Jellyfin Docker so it can be found more easily: Jellyfin and FFMPEG with MPP HW Acceleration is available now

#6

This is feasible, but the code review will take a long time or even get bogged down because I’m sure the ffmpeg devs don’t have any RK3588 devices. RGA is another tricky spot, its only publicly available official repo airockchip/librga does not contain source code, which can be very problematic.

#7

MJPEG/JPEG in MPP uses a different interface from video codec, namely MppTask. RK devs mentioned that they would consider making this change, but didn’t say an ETA.

#8

I’ll merge my jellyfin-ffmpeg PR when I feel this is stable enough. If nothing else this will be available in Jellyfin 10.9.

2 Likes
#9

Just built, building into my OpenCV now. Will test later, probably monday.

#10

I doubt OpenCV supports the MPP and RGA in ffmpeg, you may need additional programming to take advantage of these newly added codecs and filters.

#11

Is this supposed to work for hardware decoding with mpv etc when installed? Trying to use parameters --hwdec=rkmpp(-copy) or --vd=h264_rkmpp gives

[ffmpeg/video] h264_rkmpp: MPP doesn’t support codec ‘h264’ with pix_fmt ‘(null)’
Could not open codec.

#12

Not currently supported. rkmpp in MPV is incompatible with the new hwcontext_rkmpp in this ffmpeg. The MPV needs to be modified to accommodate this.

This ffmpeg-rockchip is best to be used via the CLI or as a library, while for browsers and players it is recommended to use hbiyik’s fork.

#13

Great work! I’ve been using hbiyik’s ffmpeg for quite some time now.
What’s the difference between scale_rkrga and vpp_rkrga?
Which one is faster, more performant for things like frigate? If I want to downscale a 4k video to 1080p.

#14

What’s the difference between scale_rkrga and vpp_rkrga?

Please take a look at the Wiki page.

Which one is faster, more performant for things like frigate? If I want to downscale a 4k video to 1080p.

In terms of decoding, the two speeds are about the same. In terms of encoding, this is twice as fast as before and can achieve more complex filter combinations.

#15

ffmpeg-rockhip using

./ffmpeg -init_hw_device rkmpp=hw -filter_hw_device hw -i rtsp://camera_stream -vf hwupload,scale_rkrga=w=1920:h=1080:format=yuv420p -c:v hevc_rkmpp -y /tmp/tmp.mp4

eats 130-140% CPU, probably because of encode as the scaler doesn’t allow for -c:v copy

while ffmpeg from hbiyik allows -c:v copy and eats 30-35% CPU

#16

This fork does not use libyuv from google, so hwupload is more expensive.

Encoders -c:v {h264,hevc}_rkmpp and -c:v copy are incompatible. The former is re-encoding, and the latter is just copying the stream.

Edit:
Additionally, you should refer to the VideoTranscode chapter to enable -hwaccel rkmpp -hwaccel_output_format drm_prime hardware decoding. Currently your command line is software decoding + hardware filter + hardware encoding, which is not optimal.

Also, on rk3588, unless you have reason to use yuv420p, it is more efficient to use nv12, which is determined by the RGA3 hardware capabilities.

2 Likes
#17

I would like to copy the stream, downsize it and output it in yuv420p for frigate, all in hardware. most of it can be done with hbiyik’s ffmpeg. why doesn’t it work with ffmpeg-rockchip?

[hevc_rkmpp @ 0x55c15a57d0] MPP doesn’t support codec ‘hevc’ with pix_fmt ‘yuvj420p’
[vist#0:0/hevc @ 0x55c15154e0] Error while opening decoder: Function not implemented
[vost#0:0/libx264 @ 0x55c1851500] Error initializing a simple filtergraph
Error opening output file /tmp/tmp.mp4.
Error opening output files: Function not implemented

input stream is

Input #0, rtsp, from ‘rtsp://camera1’:
Metadata:
title : Media Presentation
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: hevc (Main), yuvj420p(pc), 3840x2160 [SAR 1:1 DAR 16:9], 12 fps, 12.50 tbr, 90k tbn
Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

#18

Please apply this commit and try again, I just pushed a fix for YUVJ420P, which is about to be deprecated by upstream.

#19

./ffmpeg -hwaccel rkmpp -i rtsp://camera1 -vf hwupload,scale_rkrga=w=1920:h=1080 -y /tmp/tmp.mp4

Impossible to convert between the formats supported by the filter ‘Parsed_scale_rkrga_1’ and the filter ‘auto_scale_0’
[vf#0:0 @ 0x55aa4f3710] Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while filtering: Function not implemented
[out#0/mp4 @ 0x55aa4c7930] Nothing was written into output file, because at least one of its streams received no packets.

./ffmpeg -hwaccel rkmpp -i rtsp://camera1 -vf hwupload,scale_rkrga=w=1920:h=1080 -c:v copy -y /tmp/tmp.mp4

[vost#0:0/copy @ 0x557c3c8980] Filtergraph ‘hwupload,scale_rkrga=w=1920:h=1080’ was specified, but codec copy was selected. Filtering and streamcopy cannot be used together.
Error opening output file /tmp/tmp.mp4.
Error opening output files: Function not implemented

#20

This ffmpeg CLI syntax is incorrect, a common sense is that scaling cannot be used with -c:v copy. Scaling modifies the video source file, so it must be re-encoded.

./ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime -i rtsp://camera1 -vf scale_rkrga=w=1920:h=1080:format=yuv420p -c:v hevc_rkmpp -b:v 4M -y /tmp/tmp.mp4

1 Like