Did some more digging: I flashed an old (known-working) image that uses U-Boot 2017.09-00022-gf18abd08ab-dirty (Aug 29 2022 - 14:20:35 +0200) and simply replaced both boot and root partitions with my own Kernel (v5.15.88) / Bullseye rootfs. In this configuration ethernet works as expected.
So this seems to be an U-Boot issue and not a mainline kernel issue.