Best Media Server for Rock5b?

Hello all,
I would like to install a Media-Server on the Rock5b that uses the hardware acceleration (transcoding h.264/h.265, etc.) of the SoC. I have installed Debian 11 and Jellyfin on a (fast) SD-card (290MB/s read, 260MB/s write). However, here the SoC already reaches its limits with ONE 4k stream (WITHOUT transcoding) and the video on the client is very jerky.

My question: What alternatives are there? Is Debian the right OS?
My intentions: A media server to stream videos and music to the local network and Internet.

Note: I think that no matter what software I use, the transcoding always happens via ffmpeg and here is the bottleneck. Or what do you think?

Armbian (A small Linux for ARM boards) claims they have Kodi working with 3D acceleration: https://www.armbian.com/rock-5b/

To enable kodi hardware decode you have to run kodi under gbm or wayland!

Does anyone have experience with this? And what does gbm or wayland mean?

Plex server is the best media server that does a good job with transcoding

yesterday I installed armbian on rock 5b (and on Orange pi5) and kodi works very satisfactorily, iptv works also movies. on rock 5b I followed these steps:
https://forum.armbian.com/topic/24802-kodi-for-rk35xx-510-legacy-kernel/