ROCK 2A/F Video accelerate

[ScreenSoft]

Will have board of Roc 2A or 2F support of hardware accelerated video decoding on linux?
By Gstreamer or FFmpeg.

[nyanmisaka]

You can check this post. Rock 2A/2F uses RK3528, which is also supported by MPP and RGA.

[FFmpeg] Introduce FFmpeg-Rockchip for hyper fast video transcoding via CLI

Through this ffmpeg-rockchip fork, users can obtain the fastest hardware transcoding and filtering speed on the Rockchip platform via CLI (command line). And the code style has been kept as consistent as possible with the upstream FFmpeg, making it easy to read and maintain. This project was originally part of jellyfin/jellyfin-ffmpeg, but I have now separated it for community use and issue collection. ffmpeg-rockchip This project aims to provide full hardware transcoding pipeline in FFmpeg …

[ScreenSoft]

Thanx, works but only for yuv pixel format.

I need video player in java application.

I try gstreamer, works but after add caps acceleration is disabled.
video/x-raw,format=BGRx
or
video/x-raw,format=xRGB

By ffmpeg is failing because of “-pix_fmt rgb24”

ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime  -i ~/input.mp4 \
 -c:v h264_rkmpp -pix_fmt rgb24 -f image2pipe -

 ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime  -i ~/input.mp4 \
 -c:v hvec_rkmpp -pix_fmt rgb24 -f image2pipe -

Question:
How can i use hardware acceleration + read frames as RGB or BGR

[nyanmisaka]

It sounds like you want to convert the decoded YUV image to RGB24 and output it in rawvideo muxer. If so, then you don’t need an encoder -c:v, but instead use the RGA filter.

ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime  -i ~/input.mp4 -an -sn \
 -vf scale_rkrga=format=rgb24,hwmap,format=rgb24 -f rawvideo -

[ScreenSoft]

FHD video (1920x1080) of 30 FPS
-> decoding at 55FPS
That is great

But

UHD video (3840x2160) of 30 FPS
-> decoding at 14FPS
That can not be used.

UHD video sample

Any sugesstions?

ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime  -i ~/sample.mp4 -sn -an \
 -vf scale_rkrga=format=rgb24,hwmap,format=rgb24 -f rawvideo - | cat > /dev/null
ffmpeg version b81c3bf Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga --enable-shared --enable-nonfree --enable-zlib
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/rock/sample.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.76.100
  Duration: 00:00:08.04, start: 0.000000, bitrate: 2689 kb/s
  Stream #0:0[0x1](eng): Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, bt709, progressive), 3840x2160, 2683 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      handler_name    : ?Mainconcept Video Media Handler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_rkmpp) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #0, rawvideo, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf60.16.100
  Stream #0:0(eng): Video: rawvideo (RGB[24] / 0x18424752), rgb24(pc, bt709/unknown/unknown, progressive), 3840x2160, q=2-31, 5966001 kb/s, 29.97 fps, 29.97 tbn (default)
    Metadata:
      handler_name    : ?Mainconcept Video Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 rawvideo
[out#0/rawvideo @ 0xaaaae5bbc110] video:5856300kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
frame=  241 fps= 14 q=-0.0 Lsize= 5856300kB time=00:00:08.00 bitrate=5990860.3kbits/s speed=0.472x

[nyanmisaka]

Because using ffmpeg this way is just a demo target for debugging.

Muxer -f rawvideo - means 3840x2160x3x8 bytes memory copy per second, you should access these functions through libav interface to avoid copying.

[ScreenSoft]

How can i do that?
I mean, any preffered binding for java / kotlin ?
I am kotlin programmer + i can some things put together in shell , but C / C++ only basics.

[nyanmisaka]

Connecting the hardware decoder and video renderer and achieving 0-copy is a pretty low-level task. I’m not sure how to do it in java either. Since you are familiar with kotlin/java, why not install the Android image and use MediaCodec there, Android should take care of everything.

[ScreenSoft]

Option which i know

• FFmpeg (not fast enought because pipeline)
• Gstreamer try multiple variants but RGB by RGA seems to be not working
  • full speed YUV
    • gst-launch-1.0 filesrc location=$src ! parsebin ! mppvideodec ! appsink
  • 0.5 FPS RGB
    • gst-launch-1.0 filesrc location=$src ! qtdemux ! mppvideodec ! videoconvert ! 'video/x-raw,format=RGB' ! appsink
• OpenCV compilation everitime fail on avcodec_get_context_defaults3
  • Rock 5B working, no HW acceleration but cpu is fast enought

Android can not be used because infinite problems with display setup on LED Board, TV …

[nyanmisaka]

There is indeed a ffmpeg4j wrapper project may be useful for you.

But you have to handle avcodec, avfilter etc manually.

GitHub - Manevolent/ffmpeg4j: A Java OOP FFmpeg wrapper around the JavaCPP...

[ChenJaly]

Hi @ScreenSoft,
For the RGA question, please see if it is too low because the librga version is too low:sudo apt list -a librga2

[ScreenSoft]

librga2/rk3528a-bullseye,now 2.2.0-1 arm64 [installed,automatic]
librga2/rk3528a-bullseye 2.1.0-1 arm64

[ScreenSoft]

Somehow it working at least for FHD .
@nyanmisaka Thax for help.

I open next task Rock 2A - Turn off Display, but that is more about firmware as it self.