Rock pi 3 and NVMe

#1

Hey,

is the NVMe expected to be working ?
When I try to start rock pi 3 with my NVMe SSD I get the following boot log:

[ 1.416751] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x0
[ 1.416780] rk-pcie 3c0800000.pcie: PCIe Linking… LTSSM is 0x3
[ 2.216868] .
[ 2.430249] rk-pcie 3c0800000.pcie: PCIe Link up, LTSSM is 0x230011
[ 2.430297] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x0
[ 2.430760] rk-pcie 3c0800000.pcie: PCI host bridge to bus 0002:20
[ 2.430841] pci_bus 0002:20: root bus resource [bus 20-2f]
[ 2.430898] pci_bus 0002:20: root bus resource [??? 0x380000000-0x3807fffff flags 0x0] (bus address [0x80000000-0x807fffff])
[ 2.430950] pci_bus 0002:20: root bus resource [io 0x100000-0x1fffff] (bus address [0x80800000-0x808fffff])
[ 2.431002] pci_bus 0002:20: root bus resource [mem 0x380900000-0x3bfffffff] (bus address [0x80900000-0xbfffffff])
[ 2.431115] pci 0002:20:00.0: [1d87:3566] type 01 class 0x060400
[ 2.431186] pci 0002:20:00.0: reg 0x10: [mem 0x00000000-0x3fffffff 64bit]
[ 2.431213] pci 0002:20:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[ 2.431338] pci 0002:20:00.0: supports D1 D2
[ 2.431353] pci 0002:20:00.0: PME# supported from D0 D1 D3hot
[ 2.437831] pci 0002:20:00.0: Primary bus is hard wired to 0
[ 2.437866] pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
[ 2.438070] pci 0002:21:00.0: [1cc1:8201] type 00 class 0x010802
[ 2.438208] pci 0002:21:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 2.438761] pci 0002:21:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8 GT/s x2 link at 0002:20:00.0 (capable of 31.504 Gb/s with 8 GT/s x4 link)
[ 2.450554] pci_bus 0002:21: busn_res: [bus 21-2f] end is updated to 21
[ 2.450597] pci 0002:20:00.0: BAR 0: no space for [mem size 0x40000000 64bit]
[ 2.450638] pci 0002:20:00.0: BAR 0: failed to assign [mem size 0x40000000 64bit]
[ 2.450649] pci 0002:20:00.0: BAR 8: assigned [mem 0x380900000-0x3809fffff]
[ 2.450658] pci 0002:20:00.0: BAR 6: assigned [mem 0x380a00000-0x380a0ffff pref]
[ 2.450673] pci 0002:21:00.0: BAR 0: assigned [mem 0x380900000-0x380903fff 64bit]
[ 2.450724] pci 0002:20:00.0: PCI bridge to [bus 21]
[ 2.450733] pci 0002:20:00.0: bridge window [mem 0x380900000-0x3809fffff]
[ 2.452665] pcieport 0002:20:00.0: Signaling PME with IRQ 113
[ 2.453211] pcieport 0002:20:00.0: AER enabled with IRQ 114
[ 2.454244] nvme nvme0: pci function 0002:21:00.0
[ 2.454386] nvme 0002:21:00.0: enabling device (0000 -> 0002)
[ 3.230205] .
[ 3.443612] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x0
[ 4.120201] vendor storage:20190527 ret = -1
[ 4.243531] .
[ 4.456938] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x0
[ 5.256867] .
[ 5.470244] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x0
[ 6.270197] .
[ 6.483403] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x1
[ 7.283530] .
[ 7.496909] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x0
[ 8.296864] .
[ 8.510243] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x1
[ 9.310195] .ready
[ 9.311348] sd 0:0:0:0: [sda] 488378645 4096-byte logical blocks: (2.00 TB/1.82 TiB)
[ 9.311414] sd 0:0:0:0: [sda] 16384-byte physical blocks
[ 9.369347] sd 0:0:0:0: [sda] Write Protect is off
[ 9.369415] sd 0:0:0:0: [sda] Mode Sense: 4f 00 00 00
[ 9.369870] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUA
[ 9.370551] sd 0:0:0:0: [sda] Optimal transfer size 268431360 bytes not a multiple of physical block size (16384 bytes)
[ 9.388977] sda: sda1
[ 9.391115] sd 0:0:0:0: [sda] Attached SCSI disk
[ 9.523547] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x0
[ 10.536920] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x0
[ 11.550244] rk-pcie 3c0000000.pcie: PCIe Link Fail
[ 11.550311] rk-pcie 3c0000000.pcie: failed to initialize host
[ 122.986865] nvme nvme0: Device not ready; aborting initialisation
[ 122.986936] nvme nvme0: Removing after probe failure status: -19

#2

What is the brand of your NVME SSD?

We test NVME SSD of Kingston, KingBank. And they works.

#3

@Stephen I got this same issue with pcie linking, but without nvme attached. I can’t even get a shell for login and I tried with both img.

#4

lspci output
01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)

#5

Can you try these commands to check if NVME SSD is recognized.

sudo su
echo 1 > echo 1 > /sys/bus/pci/rescan
lsblk
#6

root@rock3a:~# echo 1 > /sys/bus/pci/rescan
root@rock3a:~# lspci
0002:20:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3566 (rev 01)
0002:21:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)
root@rock3a:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
└─sda1 8:1 0 1.8T 0 part
mmcblk1 179:0 0 119.3G 0 disk
├─mmcblk1p1 179:1 0 512M 0 part
└─mmcblk1p2 179:2 0 118.8G 0 part /

#7

Can you check the setting of serial tool according to the guide, https://wiki.radxa.com/Rock3/dev/serial-console?

Also you can access via ssh.

#8

@Stephen any suggestion related to the NVME issue?

#9

I just tried nvme with rock 3A and it worked just out of box, interestingly it showed as nvme (not sda).
What image have You used for system? What power supply?

#10

I tried a different power supply and seems it is working.

Thanks for the tip

#11

I’m glad that it helped :slight_smile: Two most frequent mistakes about sbc-s and nvme are not enough power and cooling. Power issues are easiest to discover because they are quickly noticable.
Consider benchmarking Your NVME and compare results with Internet. You should get stable read/write speeds and it should not get too hot. I found out that some cables and extenders can cause up to 10x slower speeds and Your NVME is worst than SD :wink:

#12

In my Rock3A worked after some updates and upgrades. Make sure to have enought power and some heatsink on rock3A (CPU mostly) , mine was getting hot while I was setting up the nvme. Make sure to make a filesystem and mount after it.

#13

here it is the benchmark that I got:

rock@rock3a:~$ sudo hdparm -t --direct /dev/nvme0n1

/dev/nvme0n1:
Timing O_DIRECT disk reads: 1930 MB in 3.00 seconds = 642.29 MB/sec
rock@rock3a:~$ sudo hdparm -t --direct /dev/nvme0n1

/dev/nvme0n1:
Timing O_DIRECT disk reads: 1736 MB in 3.01 seconds = 577.10 MB/sec
rock@rock3a:~$ sudo hdparm -t --direct /dev/nvme0n1

/dev/nvme0n1:
Timing O_DIRECT disk reads: 1856 MB in 3.00 seconds = 618.37 MB/sec
rock@rock3a:~$ sudo hdparm -t --direct /dev/nvme0n1

/dev/nvme0n1:
Timing O_DIRECT disk reads: 1776 MB in 3.00 seconds = 591.97 MB/sec
rock@rock3a:~$ cat /sys/class/thermal/thermal_zone0/temp
48888

#14

benchmark for the 4 cpu’s:

rock@rock3a:~$ sudo sysbench --test=cpu --cpu-max-prime=22750 --num-threads=4 run
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
WARNING: --num-threads is deprecated, use --threads instead
sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 4
Initializing random number generator from current time

Prime numbers limit: 22750

Initializing worker threads…

Threads started!

CPU speed:
events per second: 1342.09

General statistics:
total time: 10.0017s
total number of events: 13426

Latency (ms):
min: 2.95
avg: 2.98
max: 3.53
95th percentile: 2.97
sum: 39988.67

Threads fairness:
events (avg/stddev): 3356.5000/1.50
execution time (avg/stddev): 9.9972/0.00

rock@rock3a:~$ cat /sys/class/thermal/thermal_zone0/temp
48888

Good speed :smiley:

#15

Why sysbench is not the best tool to use.

Instead of hdparm rather use iozone (which is “CrystalDiskMark for Linux”)

I got 1100Mb/s + with rk3399 (3y ago)
https://forum.armbian.com/topic/7498-nanopc-t4/page/2/?tab=comments#comment-59548

For proper benchmarking (and comparison) check this tool:

#16

Thanks @igorp. So this is the output from the benchmark:

sbc-bench v0.7.8 Radxa ROCK 3 Model A (Wed, 20 Oct 2021 15:04:20 +0300)

Distributor ID: Debian
Description: Debian GNU/Linux bookworm/sid
Release: oldstable-updates
Codename: sid
Architecture: arm64

/usr/bin/gcc (Debian 11.2.0-9) 11.2.0

Uptime: 15:04:20 up 2:00, 1 user, load average: 1.27, 0.51, 0.19

Linux 4.19.193-11-rockchip-g986565dfb5d0 (rock3a) 10/20/21 aarch64 (4 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
1.26 0.00 0.49 0.49 0.00 97.76

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
mmcblk1 4.90 87.59 228.72 0.00 631110 1647932 0
nvme0n1 6.84 1038.02 688.87 34706.81 7478832 4963216 250059100

           total        used        free      shared  buff/cache   available

Mem: 3.7Gi 262Mi 2.1Gi 6.0Mi 1.3Gi 3.4Gi
Swap: 0B 0B 0B

##########################################################################

Checking cpufreq OPP:

Cpufreq OPP: 1992 Measured: 1999.884/2000.925/2000.082
Cpufreq OPP: 1800 Measured: 1836.946/1819.244/1836.528
Cpufreq OPP: 1608 Measured: 1738.289/1712.416/1741.064
Cpufreq OPP: 1416 Measured: 1501.373/1519.124/1517.928
Cpufreq OPP: 1104 Measured: 1169.803/1169.525/1169.446
Cpufreq OPP: 816 Measured: 810.697/811.653/810.707
Cpufreq OPP: 600 Measured: 586.683/594.771/593.663
Cpufreq OPP: 408 Measured: 402.377/398.610/405.011

##########################################################################

tinymembench v0.4.9 (simple benchmark for memory throughput and latency)

==========================================================================
== Memory bandwidth tests ==
== ==
== Note 1: 1MB = 1000000 bytes ==
== Note 2: Results for ‘copy’ tests show how many bytes can be ==
== copied per second (adding together read and writen ==
== bytes would have provided twice higher numbers) ==
== Note 3: 2-pass copy means that we are using a small temporary buffer ==
== to first fetch data into it, and only then write it to the ==
== destination (source -> L1 cache, L1 cache -> destination) ==
== Note 4: If sample standard deviation exceeds 0.1%, it is shown in ==
== brackets ==

C copy backwards : 1843.1 MB/s (0.7%)
C copy backwards (32 byte blocks) : 1842.5 MB/s (0.3%)
C copy backwards (64 byte blocks) : 1555.7 MB/s (0.7%)
C copy : 2788.4 MB/s
C copy prefetched (32 bytes step) : 1718.8 MB/s (0.3%)
C copy prefetched (64 bytes step) : 2320.7 MB/s (0.7%)
C 2-pass copy : 2273.1 MB/s (0.2%)
C 2-pass copy prefetched (32 bytes step) : 1382.7 MB/s (0.9%)
C 2-pass copy prefetched (64 bytes step) : 1694.5 MB/s
C fill : 7591.4 MB/s
C fill (shuffle within 16 byte blocks) : 7598.8 MB/s
C fill (shuffle within 32 byte blocks) : 7592.9 MB/s
C fill (shuffle within 64 byte blocks) : 7588.8 MB/s

standard memcpy : 2781.6 MB/s
standard memset : 7597.8 MB/s (0.3%)

NEON LDP/STP copy : 2785.8 MB/s
NEON LDP/STP copy pldl2strm (32 bytes step) : 2281.3 MB/s
NEON LDP/STP copy pldl2strm (64 bytes step) : 2708.8 MB/s
NEON LDP/STP copy pldl1keep (32 bytes step) : 2010.7 MB/s
NEON LDP/STP copy pldl1keep (64 bytes step) : 2827.8 MB/s
NEON LD1/ST1 copy : 2789.3 MB/s
NEON STP fill : 7597.5 MB/s
NEON STNP fill : 3508.2 MB/s (1.6%)
ARM LDP/STP copy : 2790.3 MB/s
ARM STP fill : 7594.1 MB/s
ARM STNP fill : 3519.1 MB/s (1.8%)

==========================================================================
== Memory latency test ==
== ==
== Average time is measured for random memory accesses in the buffers ==
== of different sizes. The larger is the buffer, the more significant ==
== are relative contributions of TLB, L1/L2 cache misses and SDRAM ==
== accesses. For extremely large buffer sizes we are expecting to see ==
== page table walk with several requests to SDRAM for almost every ==
== memory access (though 64MiB is not nearly large enough to experience ==
== this effect to its fullest). ==
== ==
== Note 1: All the numbers are representing extra time, which needs to ==
== be added to L1 cache latency. The cycle timings for L1 cache ==
== latency can be usually found in the processor documentation. ==
== Note 2: Dual random read means that we are simultaneously performing ==
== two independent memory accesses at a time. In the case if ==
== the memory subsystem can’t handle multiple outstanding ==
== requests, dual random read has the same timings as two ==
== single reads performed one after another. ==

block size : single random read / dual random read
1024 : 0.0 ns / 0.0 ns
2048 : 0.0 ns / 0.0 ns
4096 : 0.0 ns / 0.0 ns
8192 : 0.0 ns / 0.0 ns
16384 : 0.6 ns / 1.2 ns
32768 : 4.0 ns / 3.9 ns
65536 : 9.5 ns / 13.4 ns
131072 : 12.2 ns / 16.1 ns
262144 : 14.5 ns / 17.5 ns
524288 : 17.9 ns / 20.5 ns
1048576 : 90.4 ns / 134.2 ns
2097152 : 129.1 ns / 171.1 ns
4194304 : 149.1 ns / 183.8 ns
8388608 : 166.2 ns / 198.8 ns
16777216 : 175.1 ns / 207.9 ns
33554432 : 181.9 ns / 215.5 ns
67108864 : 185.7 ns / 219.3 ns

##########################################################################

OpenSSL 1.1.1l, built on 24 Aug 2021
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 181973.35k 535150.72k 1037047.81k 1357585.41k 1493532.67k 1504258.73k
aes-128-cbc 181198.26k 529799.30k 1031916.29k 1355136.00k 1491544.75k 1502549.33k
aes-192-cbc 172896.31k 474824.43k 841981.95k 1044140.71k 1122533.38k 1128617.30k
aes-192-cbc 172974.63k 475152.19k 841860.78k 1044843.18k 1123592.87k 1129387.35k
aes-256-cbc 167650.78k 432737.17k 724494.42k 871275.18k 926179.33k 930245.29k
aes-256-cbc 167458.08k 435879.34k 726858.67k 872499.54k 927058.60k 931059.03k

##########################################################################

##########################################################################

Compression:
Decompression:
Total:

##########################################################################

Testing clockspeeds again. System health now:

Time CPU load %cpu %sys %usr %nice %io %irq Temp
System health while running 7-zip multi core benchmark:

Checking cpufreq OPP:

Cpufreq OPP: 1992 Measured: 1995.655/1985.492/1996.396
Cpufreq OPP: 1800 Measured: 1832.753/1832.629/1832.545
Cpufreq OPP: 1608 Measured: 1737.111/1735.710/1738.177
Cpufreq OPP: 1416 Measured: 1516.252/1516.235/1514.705
Cpufreq OPP: 1104 Measured: 1168.533/1168.388/1168.996
Cpufreq OPP: 816 Measured: 811.394/812.241/811.085
Cpufreq OPP: 600 Measured: 590.970/593.097/595.548
Cpufreq OPP: 408 Measured: 402.626/403.914/402.813

##########################################################################

System health while running tinymembench:

Time CPU load %cpu %sys %usr %nice %io %irq Temp
15:04:27: 1992MHz 1.25 2% 0% 1% 0% 0% 0% 57.2°C
15:06:27: 1992MHz 1.06 25% 0% 25% 0% 0% 0% 60.6°C
15:08:27: 1992MHz 1.00 25% 0% 25% 0% 0% 0% 58.9°C

System health while running OpenSSL benchmark:

Time CPU load %cpu %sys %usr %nice %io %irq Temp
15:08:57: 1992MHz 1.00 3% 0% 2% 0% 0% 0% 58.9°C
15:09:07: 1992MHz 1.00 25% 0% 25% 0% 0% 0% 59.4°C
15:09:17: 1992MHz 1.00 25% 0% 24% 0% 0% 0% 61.7°C
15:09:27: 1992MHz 1.00 25% 0% 25% 0% 0% 0% 62.2°C
15:09:37: 1992MHz 1.00 25% 0% 24% 0% 0% 0% 62.2°C
15:09:47: 1992MHz 1.00 25% 0% 25% 0% 0% 0% 62.8°C
15:09:57: 1992MHz 1.00 25% 0% 24% 0% 0% 0% 60.0°C
15:10:07: 1992MHz 1.00 25% 0% 25% 0% 0% 0% 60.6°C
15:10:17: 1992MHz 1.00 25% 0% 24% 0% 0% 0% 62.8°C
15:10:27: 1992MHz 1.00 25% 0% 24% 0% 0% 0% 60.6°C
15:10:37: 1992MHz 1.00 25% 0% 25% 0% 0% 0% 60.6°C

System health while running 7-zip single core benchmark:

System health while running 7-zip multi core benchmark:

##########################################################################

Linux 4.19.193-11-rockchip-g986565dfb5d0 (rock3a) 10/20/21 aarch64 (4 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
2.48 0.00 0.47 0.47 0.00 96.58

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
mmcblk1 4.74 83.20 225.32 0.00 632006 1711560 0
nvme0n1 6.49 984.56 653.39 32919.36 7478832 4963216 250059100

           total        used        free      shared  buff/cache   available

Mem: 3.7Gi 260Mi 2.1Gi 6.0Mi 1.3Gi 3.4Gi
Swap: 0B 0B 0B

Architecture: aarch64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: ARM
Model name: Cortex-A55
Model: 0
Thread(s) per core: 1
Core(s) per cluster: 4
Socket(s): -
Cluster(s): 1
Stepping: r2p0
CPU max MHz: 1992.0000
CPU min MHz: 408.0000
BogoMIPS: 48.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

#17

Hi, did you try ssd with 2T storage space? I try choose large cold ssd