This is because spi uboot has priority set to nvme. This is IMHO wrong. Removable media should take precedence. @balbes-150 for some time in their uboot has been making “USB SD … etc” priority. That’s right IMHO
Now you can remove the boot flag from the partition on nvme.