Can anyone help me understand the boot order?

#1

I noticed something that surprised me with the boot order of my rock5b that I can’t really explain and was hoping someone might be able to offer insight.

I have my rock5b’s SPI NOR programmed currently with the release version of the u-boot bootloader to enable NVMe booting. I have my NVMe drive flashed with the official debian release from radxa’s download. I have two SD cards, one 16GB programmed with the official debian image and one 32GB with the official ubuntu image.

If I try to boot with the Debian SD card with the Debian NVMe then the boot order seems to default to the installation on the SD card which I can see by the files that I’ve placed in the default user’s folders. This is generally what I would expect since the SD card can be removed easily while the NVMe is much more inconvenient, however if someone was trying to make a highly secure environment to prevent user tampering it would be the reverse.

Now, if I do the same thing and I put in the Ubuntu SD card instead of the Debian one, instead of it booting from the SD card it boots from the NVMe drive first. I know the Ubuntu SD card works because if I unplug the NVMe drive it will boot from it, but it is booting from the NVMe card if both are installed.

Can anyone explain to me the reason that there is a different boot order for these two SD cards? My understanding is that in both cases the u-boot in the SPI is handling the boot process, so it seems to me like it should make the same choice one way or the other between NVMe and SD cards, but that’s not whats happening. I want to understand this so that I know how to modify u-boot or whatever to achieve a consistent boot order. Does anyone have any insight they could offer?

#2

I saw this with Android when I loaded it to eMMC and an SD card at the same time.

Most of this, I think, has to do with the initialization order of the media: the probe order for media in Linux seems to be SD, MMC, NVME - in particular, I’ve seen SD initialized as mmcblk0 and eMMC as mmcblk1 if both are available.

Since the volumes are found by UUID, with two copies of the image, the first block device found with that UUID becomes your rootfs.

#3

You;re right. That has to be the source of this issue. I tried to change the UUID on the Debian NVMe so that it isn’t the same as the one on Debian SD by running 'tune2fs -U random /dev/nvme0n1p2" to get a new random UUID then making changes to /extlinux/extlinux.conf on /dev/nvme0n1p1 and /etc/fstab on /dev/nvme0n1p2 to change it to booting from this new UUID and it now boots from NVMe before either the Debian or the Ubuntu SD card.

Thank you for helping me understand what’s going on. Unfortunately I would rather it boot from the SD card in both cases instead of the NVMe as I’m using this as my own development toy not a secure, locked-down boot system, so I’ll have to both customize the u-boot loader to change its default boot order and ensure that my NVMe drive also has its own unique UUID, not just a clone of the one of my SD cards to achieve this.

#4

Not that it matters much for the rest of the story, but I do not believe that is true. I think if there is a boot loader on the SD card, that takes preference over the one in the SPI NOR flash.

I had a similar issue where I was getting confused that it would not boot from SD card, but it turned out the u-boot on my SD card would choose to boot from NVMe instead of from SD.

If you’d like to know more, please see [SOLVED] Boot problem (my own fault; not USB PD related)

1 Like
#5

You may be correct about this. If it was loading the u-boot binary from the SPI NOR then I should see the u-boot console and the prompt to allow me to press CTRL-C on the console to interrupt the booting since I installed the debug version there, but I don’t see that when I have the SD card in.

In that case I am confused about why the machine wouldn’t boot Debian from NVMe until I installed the u-boot into the SPI NOR. If its booting off the SD card and ending up running off the NVMe then where does the SPI NOR come into the equation? Why was it even needed?

#6

I also had/have that confusion. I think at some point the u-boot on the SD card was not able to load the /boot on NVMe, but I’m not sure about that because I wasn’t paying attention when I installed a distro on NVMe and I knew that I needed to update SPI, so maybe I didn’t even try.

Later on I wanted not to boot from NVMe and even with SD card I was unable to do that. In the end only a modified u-boot on the SD-card allowed to ignore the NVMe.

#7

What I found was interesting
A debian SD card would boot NVME Debian, Ubuntu, Arbian
same setup
An Ubuntu SD card would boot the SD card

#8

Is there any way via hardware switch etc., to disable SD memory card boot - at least by making sure SPI (or eMMC) always get precedence over SD?

@matt911 @jack SD boot is indeed a security problem