Need help with NVMe boot on Rock 5B

#1

I have the release revision (r1.42?) of Rock 5B with 16GB RAM, and finally have time to try it out after a year or so. I want to install openFyde, and am trying it out. uSD card boot is fine, and my SSD (Intel H20) is detected okay. Try the install option, and it ended okay as well. Shutdown, reboot, and it won’t boot from NVMe.

I tried a couple time reinstalling to the SSD (it can see the 32GB / 27GiB Optane portion only, but it is fine) and it won’t help. Eventually, I reflash the SPI with debug console, run Ubuntu Budgie live CD on a spare machine to look into what’s up during Rock 5B boot. It hangs within uBoot, so I just run the “reset” command so to have the boot message show up again, as quoted below:

DDR Version V1.08 20220617
LPDDR4X, 2112MHz
......
U-Boot SPL board init
U-Boot SPL 2017.09-gbf47e8171f4-220414-dirty #stephen (Jun 07 2023 - 17:56:02)
Trying to boot from MMC2
......
Trying to boot from MMC1
......
Trying to boot from MTD2
Trying fit image at 0x4000 sector
## Verified-boot: 0
......
Jumping to U-Boot(0x00200000) via ARM Trusted Firmware(0x00040000)
Total: 500.164 ms

so the boot process is able to load U-Boot from SPI, but then it cannot find anything to boot from…

Model: Radxa ROCK 5B
PreSerial: 2, raw, 0xfeb50000
DRAM:  15.7 GiB
......
Net:   No ethernet found.
Hit key to stop autoboot('CTRL+C'):  0
pcie@fe150000: PCIe Linking... LTSSM is 0x1
......
pcie@fe150000: PCIe Linking... LTSSM is 0x2
pcie@fe150000: PCIe-0 Link Fail
Device 0: unknown device

note: the above LTSSM message appeared 40+ times with different values of 0x1, 0x2 and 0xa

Is it the NVMe SSD not properly supported, so the PCIe-0 init failed and Rock 5B thus failed boot from the SSD? I intended to use Intel Optane SSD (either M10 16GB, H20 32GB+512GB, or P1600X 118GB) with my Rock 5B, and I wonder if the current bootloader only support specific NVMe SSDs only at boot? Again, after booting into either openFyde or Debain, there is no problem detecting and using the Optane SSD at all.

#2

Indeed some nvme drives are not supported, before You move system to nvme just try to mount it on system started with sd/eMMC and check if read/write is ok. In some cases there is just not enough power to run system from nvme. Also try different nvme drive to be sure

#3

As I can actually access the NVMe SSD using live mode in both openFyde (my target OS) and Radxa’s Debian (b39) booting from uSD, I think the SSD should be fine, just that it cannot be found with uboot? That’s what I suspect; I tried using 12V8A adapter to power up my Rock 5B before I try serial console, and it won’t boot either.

#4

yep I was gonna say this … seems like that nvme is not supported

#5

Some devices fails on bigger operations like massive read/write when more energy is needed, but this also should be visible on system started from sd. Just make sure that it’s working as additional drive and is able to perform needed operations.
You can also try to boot from sd card but set root partition on nvme, it will tell You if problem lies on bootloader and nvme support, also You will get system working exacty like the one from nvme only.

#6

I think being able to write ~8GB of data is “a big enough” operation in most cases to tell that the SSD works. as I’d mentioned, the NVMe SSD works okay if I boot live mode OS using uSD, and I did the basic install which wrote ~8GB of data to it, all went fine.

What I am thinking is that the NVMe code inside current bootloader does not support my Intel Optane SSD, and perhaps many other SSDs as well. It is like in the old days when NVMe was not supported in UEFI yet on many older boards, NVMe SSD could not be used to boot on these systems, but they will work fine after OS boot. It’s not that same as my other SSD that’s not supported at all, which use a Realtek controller.

#7

I just tested again with the few SSDs I have…
Intel H10 : cannot be found by uboot
Intel H20 : cannot be found by uboot
Lite-on T11: cannot be found by uboot (it uses Phison PS5008)
Intel M10: can be found by uboot

pcie@fe150000: PCIe Linking... LTSSM is 0x1
pcie@fe150000: PCIe Linking... LTSSM is 0x6
pcie@fe150000: PCIe Linking... LTSSM is 0x210022
pcie@fe150000: PCIe Linking... LTSSM is 0x210023
pcie@fe150000: PCIe Link up, LTSSM is 0x230011
pcie@fe150000: PCIE-0: Link up (Gen3-x2, Bus0)
pcie@fe150000: invalid flags type!

Device 0: Vendor: 0x8086 Rev: K3110310 Prod: PHBT829405L9016D
            Type: Hard Disk
            Capacity: 13736.0 MB = 13.4 GB (28131328 x 512)
... is now current device
** No partition table - nvme 0 **

so it is uboot not being able to recognize my other SSDs. I am using 12V3A USB PD to DC 5.5 cable to power up my Rock 5B this time, so it should not be a power issue at boot. So far power usage is around 3W only (my USB charger has per-port watt meter) for all SSDs I’d tried

now I wonder if there would ever be update to uboot for Rock 5B that would provide better NVMe support?

#8

the related console message I get from booting openFyde on uSD:

from uBoot

pcie@fe150000: PCIe-0 Link Fail
Device 0: unknown device
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:c...
Found U-Boot script /boot/boot.scr.uimg

from kernel

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
......
[    0.230964] rk-pcie fe150000.pcie: invalid prsnt-gpios property in node
......
[    0.247589] rk-pcie fe150000.pcie: missing legacy IRQ resource
[    0.247614] rk-pcie fe150000.pcie: IRQ msi not found
[    0.247625] rk-pcie fe150000.pcie: use outband MSI support
[    0.247636] rk-pcie fe150000.pcie: Missing *config* reg space
[    0.247660] rk-pcie fe150000.pcie: host bridge /pcie@fe150000 ranges:
[    0.247688] rk-pcie fe150000.pcie:      err 0x00f0000000..0x00f00fffff -> 0x00f0000000
[    0.247708] rk-pcie fe150000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[    0.247731] rk-pcie fe150000.pcie:      MEM 0x00f0200000..0x00f0ffffff -> 0x00f0200000
[    0.247747] rk-pcie fe150000.pcie:      MEM 0x0900000000..0x093fffffff -> 0x0900000000
[    0.247781] rk-pcie fe150000.pcie: Missing *config* reg space
[    0.247817] rk-pcie fe150000.pcie: invalid resource
......
[    0.453164] rk-pcie fe150000.pcie: PCIe Linking... LTSSM is 0x3
......
[    0.535387] rk-pcie fe150000.pcie: PCIe Link up, LTSSM is 0x230011
[    0.536579] rk-pcie fe150000.pcie: PCI host bridge to bus 0000:00
......
[    0.550583] pci 0000:01:00.0: [8086:09ad] type 00 class 0x010802
......
[    0.569371] pcieport 0000:00:00.0: PME: Signaling with IRQ 152
[    0.569536] pcieport 0000:00:00.0: AER: enabled with IRQ 153
[    0.569762] nvme nvme0: pci function 0000:01:00.0
[    0.569810] nvme 0000:01:00.0: enabling device (0000 -> 0002)

read / write to nvme0 is okay: idle at 4W, with dd read at 9W, didn’t test write this time; LTSSM under uBoot is [ 0x1, 0x2 or 0xa ], while under kernel it is only 0x3 and then link is up, dunno if it make a difference?

#9

If you can interrupt uboot ctrl+c you can manually issue the command to find devices

pci enum

It will start the pcie link training again that was what the LTSSM is doing. See if it recognises the drive after a couple of attempts. Should see something like this:

pcie@fe150000: PCIe Linking... LTSSM is 0x1                                     
pcie@fe150000: PCIe Linking... LTSSM is 0x210023                                
pcie@fe150000: PCIe Link up, LTSSM is 0x230011                                  
pcie@fe150000: PCIE-0: Link up (Gen3-x4, Bus0)                                  
pcie@fe150000: invalid flags type!

After which see if the nvme drive is found

nvme scan
nvme info

To continue the boot type

boot

#10

Tried before as well and it is as expected, that “nvme scan” returned nothing, since the PCIe link training in u-boot has failed so it will not be detected correctly. Instead, the NVMe SSD is

Device 0: unknown device

The same happens to other onboard devices like RTL8125B and USB2 / USB3 controller, though it is not my main concern for now (i.e. the board cannot do USB boot, nor pxe boot), while all these are functional when linux completed boot up.

Just in case, running

pci enum

will return

pcie@fe150000: failed to find reset-gpios property
#11

In terms of (max?) power usage :
Intel H20 = 3.3V@2.8A = 9.24W
Intel H10 = 3.3V@2.5A = 8.25W
Intel M10 = 3.3V@1.2A = 3.96W
Lite-On T11 = 3.3V@1.6A = 5.28W

I don’t have any other spare NVMe SSD that I can test, but say if NVMe power consumption is to be taken into consideration, then perhaps 3.3V@1.5A is the maximum that Rock 5B can accept as bootable drive, no matte what power source (PD or DC12V) it uses? If that’s true, then it would be a stupid limitation.

Note: my guess is that Intel H10 and M10 should have the same controller for its Optane, so if M10 works, H10 should also work, but it does not, that means power consumption may be a concern here, at least for u-boot NVMe start-up.

#12

You may try to find some small, cheap nvme that is known to work and test that one to see if it’s some power or software issue. Also hybrid boot from sd+nvme almost always work, so You can have almost every nvme as system drive and almost ready only, small sd card that is used to just spin up system. I’m doing such thing on almost every board when initially there are problems getting nvme to work.
Yesterday I had similar case with Pi5 + nvme hat and kioxia BG4. It shows up in system but was just unstable. I noticed that there are some reports for problems with nvme on firmware before december, after update it was ok but still could not boot, but after all updates it is now working with just kernel on sd card. Seems stable. At some point they may fix that, as far as I could see it’s same thing as on Rock5 with PD negotiation and high power peak needed for nvme to boot.
Good luck :slight_smile:

#13

This would indicate the dts that uboot picked up doesn’t have the reset-gpio . Possibly a cause of the problem as the nvme drive should be reset before link training.

I’m using a MTFDHBA256TCK which rated 3.3V@2.5A and that seems to be detected in uboot. Power supply is 12v@5A and a dc to type c connector.
dc_usb_c

#14

I see that Radxa has a brief tutorial on how to build u-boot, do you think it is good for me to try building it by myself, and try to enable reset-gpios in this custom build? is it hard to do? I also want to try a few more things like enabling DSI and split the PCIe 3.0 x4 into x2+x2

https://wiki.radxa.com/Rock5/guide/build-u-boot-on-5b

And say if I get it work okay with u-boot, do I also need to do kinda the same with a custom kernel, or somehow deduce the dtb and put it somewhere in rootfs as overlay? not familiar with ARM platform at all, and my career had pushed me to the enterprise software side instead of embedded, so I just know very little, more like considering myself as hobbyist.

Yep, I have these type-c to dc5.5 adapters as well, and I am using a 12V USB-C -> DC5.5 to test out my Rock 5B for the time being, which I think can do up to 3A at least. These items are quite cheap on China’s Taobao, so I bought a few “just in case” previously :slight_smile:

#15

With kernel on uSD is a okay-ish solution, though I have no idea how it could be done… I know how to modify grub only :rofl:

Actually, Pi is also fine (I still have my Pi 4 4GB sitting on my desk collecting dust) as I know where the uuid strings are located in the FAT32 boot partition, but not with openFyde :sweat_smile:

Guess if I want to do it similarly at least for the time being, I need to do at least copying the kernel partition over to the uSD, and perhaps the “hidden configuration partition”… maybe simply deleting the corresponding partitions on NVMe and uSD can do the trick, I dunno, but could be… nowadays bootloaders just look for uuids, so deleting will make it not possible to confuse the boot process, though it also means I would need to find a spare uSD to host the kernel… anyway, worth considering when I have spare time.

#16

I just found that as the boot process will load NVMe driver first, my Rock 5B running openFyde will mount NVMe partition first, so I don’t even need to delete anything on the uSD. I doubled checked that all mount points are using NVMe partitions, while the uSD ones are under “removable”.

Not the way it should be, but I would live with it for the time being.

#17

This is simple, just burn image to sd card and nvme. If You don’t have adapter or free slot on computer it’s also possible to just boot system from sd and empty nvme and dd everything to nvme. After restart You should have both sd card and nvme partitions visible, but sd card one is mounted to /
Then just modify uuid of root partition in configuration from sd card to nvme, reboot and You should start from sd, but system running from nvme.