I was wondering if anyone knew if the Rock 5 supported NVME drives greater than 2TB in capacity on its PCIe 3.0 x4 slot.
I see some 4TB drives available with PCIe 3.0 and 4.0 interfaces (hoping the PCIe 4.0 ones would downgrade to PCIe 3.0 speeds as is standard).
I’m using a Rock Pi 4 as a compact low power file server/small app server in a small RV, and upgrading storage space along with CPU and memory would be nice.
Thanks!
NVMe has been designed from the ground up to overcome limitations of AHCI/SATA (where with LBA48 the highest addressable capacity is 131072 TiB or 128 PiB). So once you want to use NVMe drives 65.536 larger than the 2TB thing you want to use now only then I would start to think about capacity limitations.
PCIe 3.0 and 4.0 are specification revisions. The speeds are defined as Gen1, Gen2 and so on and PCIe link training is part of the PCIe specs.
As such the PCIe 3.0 controller in RK3588 allowing for Gen3 link speeds with 4 lanes max (x4) in combination with a PCIe4/Gen4 capable SSD will negotiate Gen3 speeds with as much lanes as available. With dusty contacts negotiated link speed might even be lower so it’s always a great idea to use lspci -vv after installing a NVMe SSD to see what has been negotiated.
BTW: on paper a Gen4 x2 SSD and a Gen3 x4 should perform somewhat similar when looking at theoretical transfer speeds. But in a slot behind a PCIe 3 controller that maxes out at Gen3 link speed the difference becomes obvious since the Gen4 capable SSD ends up with half the speed since this results in a Gen3 x2 device.
As such: when buying SSDs those supporting x4 connections are the better choice if the plan is to connect it to older controllers only capable of lower speeds.
Thanks!
Perhaps I should ask a slightly different question since certain NVMe SSDs are reported to have problems on Rock 4 SBCs:
Has anyone used specific NVMe drives greater than 2TB in capacity working without problems on the Rock 5, and if so which model(s)?
I’ve been wondering how to determine the link speed - thank you for the lspci -vv pointer. Do you know of a good reference describing how to interpret the results to check the speed?
I think my Rock 4 with a Samsung 970 EVO Plus 2TB may be suffering from degraded negotiated link speed (after Gen2 enabled):
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (downgraded), Width x2 (downgraded)Rock 4 is based on RK3399 which is only capable of Gen2 x4. As such Speed 5GT/s (downgraded) is to be expected.
Translation of GT/s to PCIe ‘Gen’ speed classes: https://en.wikipedia.org/wiki/PCI_Express#History_and_revisions (with Gen3 the coding switched to something more efficient as such 8 GT/s with Gen3 is pretty much or almost twice as fast as 5 GT/s with Gen2)
Missing two lanes (Width x4 -> Width x2) is not expected (by me) but the only Rock 4 device I own is an early developer sample from 2018 where even the CE logo is printed wrongly so I can’t say much about that.
As for the reports about ‘certain NVMe SSDs having problems’ I better not comment since that’s just like other storage issues in this weird SBC world: people blame technology without understanding it and copy&paste the same disinformation over and over again (as with UAS or ‘USB attached SCSI’ which is blamed for every USB cable problem in this SBC world).
NVMe as protocol that is directly used has no capacity limitation today (unlike some crappy USB enclosures – at least that was a thing with SATA where drives were capped at 2TB in an USB enclosure while happily showing full capacity when used natively).
Neither Radxa nor Rockchip can change anything about this since there’s nowhere special voodoo involved. Radxa is just slapping Rockchip SoCs on their boards and Rockchip just licenses its PCIe controller over there: https://www.synopsys.com/designware-ip/interface-ip/pci-express.html
The TL;DR version is:
[1] RK3588 consists of 5 PCIe controllers to provide PCIe Gen3 with 4 lanes max and Gen2 with 3 lanes max, the latter pinmuxed with SATA/USB3:
After disassembling everything, blowing out and cleaning all the contacts, and replacing the PCIe ribbon cable, my Rock Pi 4 B+ PCIe Width is back to x4 and NVMe speeds are back to normal. So maybe either dirt/dust or the EcoPi Pro HP case bending/damaging the ribbon cable caused the issue.
Thanks @tkaiser for the guidance and info!
I look forward to repeating this all with a Rock Pi 5, and I’m still interested in hearing people’s Rock 5 experiences with specific NVMe drives - especially fastest Gen3 speeds, lowest power consumption, and largest capacity (I understand all capacities should work) used successfully.
Symptoms point more to “and” and not “either/or”. 
At least you experienced PCIe link training first hand (slowing down your PCIe connection between host and SSD to 25%) and would probably now vote for a startup service that checks the following stuff:
lspci)?lspci -vv | grep -E "LnkCap|LnkSta")? If so somehow inform the user about these issuespowersupersave)Something I suggested to Radxa already months ago and still hoping for them to implement since these PCIe link issues are far more common than expected especially once ‘extender cables’ or ‘extender boards’ (like on Khadas boards) are used.
Hi,
my Crucial P3 4TB (CT4000P3SSD8) is running on my Rock 5B without problem so far (with an Armbian image installed on a microSD).
It’s a Crucial P3 ‘not plus’ (so PCIe 3.0, not 4.0) SSD. My usage is focused on storage (served on a Gb/s network) and h24 running server ; so low power consumption is more important for me compared to performance. According to tomshardware’s test, power efficiency of this SSD is very good.
I made two partitions nvme0n1p[x]
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
mtdblock0 31:0 0 16M 0 disk
mmcblk0 179:0 0 59.5G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 58.6G 0 part /var/log.hdd
/
zram0 254:0 0 3.8G 0 disk [SWAP]
zram1 254:1 0 50M 0 disk /var/log
nvme0n1 259:0 0 3.6T 0 disk
├─nvme0n1p1 259:1 0 200G 0 part
└─nvme0n1p2 259:2 0 3.4T 0 part
lspci -vv
0000:01:00.0 Non-Volatile memory controller: Micron/Crucial Technology P2 NVMe PCIe SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Micron/Crucial Technology P2 NVMe PCIe SSD
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 131
Region 0: Memory at f0200000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [d0] MSI-X: Enable+ Count=9 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [e0] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [f8] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [110 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=300us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1e0 v1] Data Link Feature <?>
Capabilities: [200 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [300 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Kernel driver in use: nvmeSo you trust into numbers somewhere on the Internet instead of conducting an own measurement? You know that PCIe peripherals connected to Rockchip SoCs (often? sometimes? always?) not behave like they should do?
Given:
So yes I rather trusted other’s conclusions, those of an ‘unknown website like tomshardware 
’ and saved some of my free time for other topics.
So you have no problem now having a board+SSD combination that consumes magnitudes more in idle than what some testers on x86 reported somewhere on the Internet?
You maybe even think this ‘Android e-waste world’ you entered recently would be comparable to the PC world?
rock 5B board, 8GB version
idle (one user logged in via ssh, htop running, < 0.5% CPU)
=> 2.07 Watts IDLE (5.18v, 0.40A measured at the USB C plug)
with motion running video detection on 5 fullHD network cameras
=> 18-24% cpu load ; (not null but) low nvme disk usage
=> 3.73 Watts LOW LOAD (5.18v, 0.72A measured at the USB C plug, 120s average)
running stress -c 8
=> 99.8% CPU load
=> 8.19 Watts HIGH CPU LOAD (5.15V, 1.59A measured at the USB C plug, 120s average)
=> temperature around 70°C (room temp = 19°C)
my conclusion (based on scenarii that represent my personal usage): it consumes magnitudes less than my previous setup (AsRock J4105 ITX, 4TB 2.5" HDD, 11W idle)
Thank you, first consumption number ever posted with RK3588 + NVMe SSD and it seems we’re not affected by ‘3W more in idle’ reported with RK3568
Could you please post your idle CPU and SSD temperatures?
sensors
nvme smart-log /dev/nvme0n1
sure:
room temp: 19°C
running ssh + tmux + 2 shells:
glanceswatch sudo nvme smart-log /dev/nvme0n1< 2% CPU load
nvme LUKS partition mounted ; no activity
after 95 minutes:
glances shows:
SENSORS
soc_thermal 1 33C
bigcore0_thermal 1 32C
bigcore1_thermal 1 32C
littlecore_thermal 1 32C
center_thermal 1 32C
gpu_thermal 1 31C
npu_thermal 1 32C
and nvme smart-log shows:
Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
temperature : 32°C (305 Kelvin)
Temperature Sensor 1 : 32°C (305 Kelvin)
Temperature Sensor 2 : 33°C (306 Kelvin)
Temperature Sensor 8 : 32°C (305 Kelvin)
Thank you. My use case is similar (storage, low idle consumption) and I also have the latest Armbian installed on a microSD card. However, my temps are higher (room temp is 19°C):
rock5b:~:% sudo nvme smart-log /dev/nvme0n1
Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
critical_warning : 0
temperature : 36 C (309 Kelvin)
available_spare : 100%
available_spare_threshold : 10%
percentage_used : 0%
endurance group critical warning summary: 0
data_units_read : 86,704,199
data_units_written : 22,332,701
host_read_commands : 178,096,067
host_write_commands : 45,981,877
controller_busy_time : 862
power_cycles : 1,488
power_on_hours : 1,539
unsafe_shutdowns : 605
media_errors : 0
num_err_log_entries : 1
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0
rock5b:~:% sensors
gpu_thermal-virtual-0
Adapter: Virtual device
temp1: +35.2°C
littlecore_thermal-virtual-0
Adapter: Virtual device
temp1: +36.1°C
bigcore0_thermal-virtual-0
Adapter: Virtual device
temp1: +35.2°C
tcpm_source_psy_4_0022-i2c-4-22
Adapter: rk3x-i2c
in0: 9.00 V (min = +9.00 V, max = +9.00 V)
curr1: 3.00 A (max = +3.00 A)
npu_thermal-virtual-0
Adapter: Virtual device
temp1: +35.2°C
center_thermal-virtual-0
Adapter: Virtual device
temp1: +35.2°C
bigcore1_thermal-virtual-0
Adapter: Virtual device
temp1: +35.2°C
soc_thermal-virtual-0
Adapter: Virtual device
temp1: +35.2°C (crit = +115.0°C)
rock5b:~:% cat /sys/module/pcie_aspm/parameters/policy
default performance [powersave] powersupersave
rock5b:~:% cat /etc/default/cpufrequtils
ENABLE=true
MIN_SPEED=408000
MAX_SPEED=
GOVERNOR=ondemand
What am I missing? I have a WD SN550 1TB SSD (latest firmware).
