Recently I promoted my Orion O6 to home-server status. As a result, I needed my Jellyfin instance to not hog the entire CPU. Despite Vulkan looking perfectly fine, I could not get Jellyfin to utilize my AMD RX6600 GPU. Because I was able to utilize it to accelerate my llama.cpp instance, I was pretty sure it wasn’t a Vulkan or kernel problem. After days of troubleshooting, I was able to get it working.
Jellyfin uses a custom build of ffmpeg, however the official aarch64 deb does not support VAAPI, which is the only way to accelerate AMD cards on linux that it supports. I assume this is because aside from the Orion, very few aarch64 devices have pcie slots.
I finally got it working by compiling Jellyfin-ffmpeg with the appropriate flag ( --enable-vaapi I think). Once compiled, replace the stock jellyfin-ffmpeg folder contents with the contents of the build folder.
Once restarted, this should enable hardware encode and decode on the AMD gpu.
Not sure if it matters, but i am using stock Ubuntu Noble with the 6.16 kernel and bleeding edge Mesa drivers.
I can provide screenshots or more info as requested, but hopefully this helps someone!