NVMe problems on rock5

#1

After update of bootloader I was able to start rock5 with nvme drive connected (more info here)

I quickly found out that my nvme does not show, problematic drive is: HP EX900 (250G), it should be 4x 3.0
Also tested other drives and have no problem with kioxia 256GB - KBG40ZNS256G - it should be bit slower but still 4x 3.0 - this one come up correctly as /dev/nvme0n1 (yes, it’s small 2230 drive)

I’m inspecting now dmesg but nothing suspicious, on working drive there are correct messages about it. on hp nothing about nvme or any error. I already checked few images, but nothing changes.

Both drive works perfectly via usb adapter.
Any ideas?

1 Like
Rock5 does not work on most PD power supplies
Radxa Rock5 RK3588 SBC: PCIe lanes clarification
JMB585 problems
Rock5 does not work on most PD power supplies
#2

does the hp drive need higher ammounts of power than the kioxia

#3

According to specs this particular kioxia use 3,7W on active when this hp is rated at 2.9W
I don’t expect those problems, drive is not detected, on power problems whole system would restart or hang, and this will probably occur on heavy usage, not on idle. Also there is no error message that something is wrong with this drive.

Rather some nmve are not compatible, I never saw this drive on such list. I tried few kernels and builds and it’s not detected on all of them.

#4

it mabe sounds funny but try to clean the contacts even if it had worked in an caddy

#5

Can you see the drive in lspci? If it does then the nvme driver may failed to load for this drive, it it does not then dmesg should have some PCIe link failed info

#6

I am not using m.2 NVMe SSD yet (but have it installed), and no it is not detected, not even in lspci, nor there is nvme.ko anywhere in the /lib/modules directory. I am trying Radxa’s official Ubuntu at the moment.

BTW, mine is an old leftover I pull from my desktop after upgrade: ADATA XPG SX6000 (RTS5760)

#7

If this is the case I think there should be some rk-pcie logs in dmesg indicating that the PCIe link is failed. Can you check that?

#8

Hi dominik,
Thank you for your info.
I have a KBG40ZNS256G NVMe and my Rock 5B will be here within two or three weeks.
It was shipped just this morning.
I’m looking forward to recieving the board.

#9

See if it would help then…

Basically only two PCIe ports are show via lspci

0002:20:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0002:21:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
0004:40:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 04)

in dmesg, each port will have init message just like this (filtered)

[ 6.033697] rk-pcie fe190000.pcie: invalid prsnt-gpios property in node
[ 6.033712] rk-pcie fe190000.pcie: Looking up vpcie3v3-supply from device tree
[ 6.034455] rk-pcie fe190000.pcie: missing legacy IRQ resource
[ 6.034481] rk-pcie fe190000.pcie: IRQ msi not found
[ 6.034487] rk-pcie fe190000.pcie: use outband MSI support
[ 6.034496] rk-pcie fe190000.pcie: Missing config reg space
[ 6.034512] rk-pcie fe190000.pcie: host bridge /pcie@fe190000 ranges:
[ 6.034548] rk-pcie fe190000.pcie: err 0x00f4000000…0x00f40fffff -> 0x00f4000000
[ 6.034594] rk-pcie fe190000.pcie: IO 0x00f4100000…0x00f41fffff -> 0x00f4100000
[ 6.034618] rk-pcie fe190000.pcie: MEM 0x00f4200000…0x00f4ffffff -> 0x00f4200000
[ 6.034632] rk-pcie fe190000.pcie: MEM 0x0a00000000…0x0a3fffffff -> 0x0a00000000
[ 6.034677] rk-pcie fe190000.pcie: Missing config reg space
[ 6.034714] rk-pcie fe190000.pcie: invalid resource

And then you can see only two ports init okay, and one failed (filtered)

[ 5.806572] rk-pcie fe170000.pcie: PCIe Link up, LTSSM is 0x130011
[ 5.806947] rk-pcie fe170000.pcie: PCI host bridge to bus 0002:20
[ 6.296738] rk-pcie fe190000.pcie: PCIe Link up, LTSSM is 0x130011
[ 6.297028] rk-pcie fe190000.pcie: PCI host bridge to bus 0004:40
[ 8.360131] rk-pcie fe150000.pcie: PCIe Link Fail
[ 8.360157] rk-pcie fe150000.pcie: failed to initialize host

Seems like the “Linking” process failed (filtered)

[ 5.676639] rk-pcie fe150000.pcie: PCIe Linking… LTSSM is 0x3
[ 5.702178] rk-pcie fe150000.pcie: PCIe Linking… LTSSM is 0x3
[ 5.728846] rk-pcie fe150000.pcie: PCIe Linking… LTSSM is 0x3
[ 5.755490] rk-pcie fe150000.pcie: PCIe Linking… LTSSM is 0x2
[ 5.782175] rk-pcie fe150000.pcie: PCIe Linking… LTSSM is 0x100003
[ 5.808983] rk-pcie fe150000.pcie: PCIe Linking… LTSSM is 0x2
[ 5.835667] rk-pcie fe150000.pcie: PCIe Linking… LTSSM is 0x100003
[ 5.862174] rk-pcie fe150000.pcie: PCIe Linking… LTSSM is 0x2
[ 5.888887] rk-pcie fe150000.pcie: PCIe Linking… LTSSM is 0x200003
[ 8.479108] rk-pcie fe150000.pcie: PCIe Link Fail
[ 8.479133] rk-pcie fe150000.pcie: failed to initialize host

#10

Looks like the Link trainning is failed, a very rockchip problem(RK3399 boards also have this issue). I cannot offer further help and we need someone familiar with rockchip’s pcie controller to debug this.

#11

Later today I’ll have more time for this issue, i’ll post my dmesg differences on working and missing nvme.

BTW: there is some issue with UART console - doing dmesg on console causes something wrong about buffer, I’ll get output stripped and need to enter additional chars to get tail of it. I’m not yet sure if it’s client issue (no such problems on rock3) or something with rock5.

#12

for the console try screen or putty both work pretty well

#13

could it be a problem with the dtb? Since the ones w/o problem are the PCIe 2.0 lanes, and perhaps PCIe 3.0 is not activated correctly? I want to stay as hobbyist, so I won’t go deeper.

#14

i dont think so because also pcie 3 ssds worked

#15

so only some m.2 SSDs won’t work, right? like mine (NVMe PCIe 3.0 x2)

#16

when my rock 5b is here i will test all m.2 pcie drives i have

1 Like
#17

Ok, I tried today latest radxa rock5 cli images and both drives are not visible in system :slight_smile:
not much difference in dmesg for three cases (hp, kioxia and none), I can’t confirm now if kioxia worked on some of radxa earlier images or was it my build. Also the fact it showed in system doesn’t necessary mean that it was stable, I need to go back there and retest it.

digging in previous logs,
dmesg part from HP:

[    4.777677] rk-pcie fe150000.pcie: invalid prsnt-gpios property in node
[    4.777687] rk-pcie fe150000.pcie: Looking up vpcie3v3-supply from device tre e
[    4.777757] rk-pcie fe170000.pcie: invalid prsnt-gpios property in node
[    4.777762] rk-pcie fe170000.pcie: Looking up vpcie3v3-supply from device tre e
[    4.783323] rk-pcie fe170000.pcie: missing legacy IRQ resource
[    4.783339] rk-pcie fe170000.pcie: IRQ msi not found
[    4.783345] rk-pcie fe170000.pcie: use outband MSI support
[    4.783351] rk-pcie fe170000.pcie: Missing *config* reg space
[    4.783364] rk-pcie fe170000.pcie: host bridge /pcie@fe170000 ranges:
[    4.783388] rk-pcie fe170000.pcie:      err 0x00f2000000..0x00f20fffff -> 0x0 0f2000000
[    4.783404] rk-pcie fe170000.pcie:       IO 0x00f2100000..0x00f21fffff -> 0x0 0f2100000
[    4.783421] rk-pcie fe170000.pcie:      MEM 0x00f2200000..0x00f2ffffff -> 0x0 0f2200000
[    4.783433] rk-pcie fe170000.pcie:      MEM 0x0980000000..0x09bfffffff -> 0x0 980000000
[    4.783462] rk-pcie fe170000.pcie: Missing *config* reg space
[    4.783494] rk-pcie fe170000.pcie: invalid resource
[    4.783706] rk-pcie fe150000.pcie: missing legacy IRQ resource
[    4.783722] rk-pcie fe150000.pcie: IRQ msi not found
[    4.783727] rk-pcie fe150000.pcie: use outband MSI support
[    4.783732] rk-pcie fe150000.pcie: Missing *config* reg space
[    4.783776] rk-pcie fe150000.pcie: host bridge /pcie@fe150000 ranges:
[    4.783798] rk-pcie fe150000.pcie:      err 0x00f0000000..0x00f00fffff -> 0x0 0f0000000
[    4.783813] rk-pcie fe150000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x0 0f0100000
[    4.783830] rk-pcie fe150000.pcie:      MEM 0x00f0200000..0x00f0ffffff -> 0x0 0f0200000
[    4.783842] rk-pcie fe150000.pcie:      MEM 0x0900000000..0x093fffffff -> 0x0 900000000
[    4.783875] rk-pcie fe150000.pcie: Missing *config* reg space
[    4.783902] rk-pcie fe150000.pcie: invalid resource
[    4.989617] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    4.989623] rk-pcie fe150000.pcie: PCIe Linking... LTSSM is 0x0
[    5.015140] rk-pcie fe150000.pcie: PCIe Linking... LTSSM is 0x1
[    5.015151] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.041818] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.041830] rk-pcie fe150000.pcie: PCIe Linking... LTSSM is 0x0
[    5.068481] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.068487] rk-pcie fe150000.pcie: PCIe Linking... LTSSM is 0x0
[    5.095153] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.095158] rk-pcie fe150000.pcie: PCIe Linking... LTSSM is 0x0
[    5.329563] rk-pcie fe190000.pcie: invalid prsnt-gpios property in node
[    5.329623] rk-pcie fe190000.pcie: Looking up vpcie3v3-supply from device tre e
[    5.335910] rk-pcie fe190000.pcie: missing legacy IRQ resource
[    5.335963] rk-pcie fe190000.pcie: IRQ msi not found
:
[    5.335980] rk-pcie fe190000.pcie: use outband MSI support
[    5.335999] rk-pcie fe190000.pcie: Missing *config* reg space
[    5.336048] rk-pcie fe190000.pcie: host bridge /pcie@fe190000 ranges:
[    5.336123] rk-pcie fe190000.pcie:      err 0x00f4000000..0x00f40fffff -> 0x0 0f4000000
[    5.336173] rk-pcie fe190000.pcie:       IO 0x00f4100000..0x00f41fffff -> 0x0 0f4100000
[    5.336227] rk-pcie fe190000.pcie:      MEM 0x00f4200000..0x00f4ffffff -> 0x0 0f4200000
[    5.336265] rk-pcie fe190000.pcie:      MEM 0x0a00000000..0x0a3fffffff -> 0x0 a00000000
[    5.336378] rk-pcie fe190000.pcie: Missing *config* reg space
[    5.336464] rk-pcie fe190000.pcie: invalid resource
[    5.598599] rk-pcie fe190000.pcie: PCIe Link up, LTSSM is 0x130011
[    5.598815] rk-pcie fe190000.pcie: PCI host bridge to bus 0004:40
[    7.695364] rk-pcie fe150000.pcie: PCIe Link Fail
[    7.695381] rk-pcie fe150000.pcie: failed to initialize host
[    7.711883] rk-pcie fe170000.pcie: PCIe Link Fail
[    7.711903] rk-pcie fe170000.pcie: failed to initialize host

grep for rkcpie and nvme on kioxia:

[    5.346619] rk-pcie fe190000.pcie: invalid prsnt-gpios property in node
[    5.346644] rk-pcie fe190000.pcie: Looking up vpcie3v3-supply from device tree
[    5.347557] rk-pcie fe190000.pcie: missing legacy IRQ resource
[    5.347577] rk-pcie fe190000.pcie: IRQ msi not found
[    5.347584] rk-pcie fe190000.pcie: use outband MSI support
[    5.347592] rk-pcie fe190000.pcie: Missing *config* reg space
[    5.347617] rk-pcie fe190000.pcie: host bridge /pcie@fe190000 ranges:
[    5.347661] rk-pcie fe190000.pcie:      err 0x00f4000000..0x00f40fffff -> 0x00f4000000
[    5.347688] rk-pcie fe190000.pcie:       IO 0x00f4100000..0x00f41fffff -> 0x00f4100000
[    5.347710] rk-pcie fe190000.pcie:      MEM 0x00f4200000..0x00f4ffffff -> 0x00f4200000
[    5.347721] rk-pcie fe190000.pcie:      MEM 0x0a00000000..0x0a3fffffff -> 0x0a00000000
[    5.347766] rk-pcie fe190000.pcie: Missing *config* reg space
[    5.347798] rk-pcie fe190000.pcie: invalid resource
[    5.611256] rk-pcie fe190000.pcie: PCIe Link up, LTSSM is 0x130011
[    5.611517] rk-pcie fe190000.pcie: PCI host bridge to bus 0004:40
[    7.671362] rk-pcie fe170000.pcie: PCIe Link Fail
[    7.671381] rk-pcie fe170000.pcie: failed to initialize host
[    4.975678] rk-pcie fe150000.pcie: PCIe Linking... LTSSM is 0x0
[    4.975685] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.001208] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.027877] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.054541] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.057886] rk-pcie fe150000.pcie: PCIe Link up, LTSSM is 0x230011
[    5.058064] rk-pcie fe150000.pcie: PCI host bridge to bus 0000:00
[    5.081210] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.097962] nvme nvme0: pci function 0000:01:00.0
[    5.098011] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[    5.107877] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.134586] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.161377] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.187948] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.343374] nvme nvme0: 8/0/0 default/read/poll queues
[    5.348519]  nvme0n1:dxz
[    5.346619] rk-pcie fe190000.pcie: invalid prsnt-gpios property in node
[    5.346644] rk-pcie fe190000.pcie: Looking up vpcie3v3-supply from device tree
[    5.347557] rk-pcie fe190000.pcie: missing legacy IRQ resource
[    5.347577] rk-pcie fe190000.pcie: IRQ msi not found
[    5.347584] rk-pcie fe190000.pcie: use outband MSI support
[    5.347592] rk-pcie fe190000.pcie: Missing *config* reg space
[    5.347617] rk-pcie fe190000.pcie: host bridge /pcie@fe190000 ranges:
[    5.347661] rk-pcie fe190000.pcie:      err 0x00f4000000..0x00f40fffff -> 0x00f4000000
[    5.347688] rk-pcie fe190000.pcie:       IO 0x00f4100000..0x00f41fffff -> 0x00f4100000
[    5.347710] rk-pcie fe190000.pcie:      MEM 0x00f4200000..0x00f4ffffff -> 0x00f4200000
[    5.347721] rk-pcie fe190000.pcie:      MEM 0x0a00000000..0x0a3fffffff -> 0x0a00000000
[    5.347766] rk-pcie fe190000.pcie: Missing *config* reg space
[    5.347798] rk-pcie fe190000.pcie: invalid resource
[    5.611256] rk-pcie fe190000.pcie: PCIe Link up, LTSSM is 0x130011
[    5.611517] rk-pcie fe190000.pcie: PCI host bridge to bus 0004:40
[    7.671362] rk-pcie fe170000.pcie: PCIe Link Fail
[    7.671381] rk-pcie fe170000.pcie: failed to initialize host

@enoch - this is probably thing You talked about. There are others rk-pcie errors but device is enabled and it shows in system (I’ll test if its stable or not, what speed can achieve)

BTW: power problems with nvme - I also tested that - supplying not enough power will restart board on nvme initialization causing bootloop.

#18

not all sbc reboot if 3.3 volts for the m.2 slot is missing because it is often a seperate buck converter but this one reboots

#19

Some update:
I tried several power sources and images and none of them work with HP EX900 nvme, I tried few of them and all works perfectly with rock3a board (I never had any problem with them on any board). I also checked few system images including latest from built by radxa and others, most with similar error. :frowning:
On the other hand my small kioxia works. This one is linked and can be used for tests, but because of size I cannot mount it here.
Any chances and ideas to make this nvme to work or should I change all of them to something else?
This drive was on all compatibility lists for few other sbcs and it is reasonable priced. There is high chance that more things are not compatible :frowning:

#20

looks like I have the same problem,

Loaded the spi bootloader
can boot from sd card but not from nvme ssd
the ssd is just missing
when trying to boot from nvme the network lights flash for a bit then goes off, looks like it tries to boot and then dies.

don’t have same problem with rock4c and rock3a