SD card and eMMC worked all the time.
The plan was to ship with a bootloader inside the SPI flash prepopulated at the factory to allow booting from NVMe as well. And few weeks ago this has been added to u-boot (the bootloader): https://github.com/radxa/u-boot/commits/stable-5.10-rock5
So in theory it should work and since they added NVMe over here a NVMe SSD should be all that is needed.