Rock 4 SE: Video playback is slow and freezes constantly over Jellyfin, Plex as well as network storage drive

I bought a Rock 4 Se recently and have been trying to setup a home media server but so far I have not been able to achieve a smooth playback experience despite spending a lot of time on this.

Gadgets I am using:

  • Rock 4 SE (4GB Ram)
  • Toshiba 2TB external HDD
  • 32GB Sandisk Micro SD card
  • Macbook Pro, Nvidia Shield, iPhone 12 Pro and Pixel 4 as clients for playback
  • Ethernet cable (50Mbps upload and download speed)

The problem is that video playback keeps freezing. I am the only one using it so there is max 1 stream at a time. These are the observations

  • I tried to install Jellyfin, Plex and turned off transcoding and made sure that direct play is happening without transcoding but no difference
  • I made my HDD a network storage and tried to play the videos directly using VLC/Kodi from other clients. Using TCPTrack I observed the transfer/upload speed when playing video on other clients and the speed I am getting is not more than 1MB/s.
  • I tried to do a file transfer from the network storage to my Mac using SFTP and got speed of around 40MB/s

I know there is no comparison between a Macbook pro and Rock 4 SE but I just tried to check what happens if I use the Mac as the server and the playback on other clients was butter smooth so I think my internet speed and HDD are not the issue.

I am expecting that the Rock 4 SE will be able to play 1 stream at a time without transcoding without any issues. Is there some configuration issue or other issue which maybe causing this? What am I missing? Will appreciate any suggestions to fix this. Thanks!