I found instructions for recompiling the kernel here: https://wiki.radxa.com/Rock5/guide/build-kernel-on-5b
When I did that (enabling the ATI radeon graphics module), the kernel compiled okay, but when I tried switching to it, the Rock 5 would no longer boot (the blue LED would flash in a pattern with two blinks, and no display output would happen).