It seems that the rx_delay setting was wrong.
This solved my problem temporarily:
echo 0x28 0x20 > /sys/class/net/eth0/device/driver/fe300000.ethernet/rgmii_delayline
Does anyone know how to make this permanent?
Credits to the @sfo for his info in this thread:
Edit:
For now I just created a cron job @reboot:
@reboot echo 0x28 0x20 > /sys/class/net/eth0/device/driver/fe300000.ethernet/rgmii_delayline
Still curious if there is a better way to fix this, and if these settings are optimal or not. The default 0x28 0x11 works fine on my Gbit lan, but on my 100 Mbit lan it just won’t work for about 3 out of 10 RockPis.