hey @jack i am using a poe that delivers up to 30W per port and no other usb devices connected, just the nvme and ethernet cable
M2.ssd will randomly fail to write on rock3a
What PoE HAT are you using? I suspect the voltage for 5V is dropped so the NVMe is under voltage.
Hey @jack it is a RockPi_PoE_F4L Rock Pi 4, if you point me where I should take the measurement of the voltage, I can double-check it.
@jack
i have the same problem.
this problem has been happend on my 4 sbc.
the proer supply is CoolMaster GX550.
i’m using this way on cron to alleviate this problem.
- check if nvme fs mount has mounted and is rw
- check /dev/ has nvme partition mount point, if not, reboot
- stop all service which usins the nvme fs mount point
- umount all nvme partition
- run fsck
- remount all nvme partition
- restart alll service which had stoped before
thanks for this but, it still is not a solution on the long run.
i wonder if i with emmc the same thing would happen.
Hey jack, even if i plug a QC charger in it the same happens.
To reproduce it faster you can do apt install stress && cd /nvme-path && journalctl -kf & && stress --cpu 8 --vm 8 --hdd 8
hey @jack do you have anywhere the specs of the maximum current allowed for m2 ssd? that is the only thing i can think of, i have different versions at home and both of them with different power requirements
We will reproduce this issue, the design power current for the SSD on ROCK 3A is 5A. Check your SSD’s power consumption. Usually it’s 3.3V 3A peak.
Hey @jack, thanks for the answer!
The one i have home is a crucial with 3.3V 2.5A so, I should still have plenty of power left.
I will try one of those kernel 5.x builds to try and check if their stack have better luck
Hey jack, good news, using armbian (Armbian 22.05.1 Bullseye) with the following kernel:
Linux rock-3a 5.18.0-rk35xx #22.05.1 SMP PREEMPT Sat May 28 08:41:15 UTC 2022 aarch64 GNU/Linux
I’ve managed to run a stress test like the one i shared above for 30 minutes without having the nvme remount.
Smart complained, but so far it is way better:
Device: /dev/nvme0, number of Error Log entries increased from 476187 to 476188
Scratch that, the error came back after a while
Hi, @jaysonsantos
We have a 3A with 23W PoE HAT setup running for some days, we can reproduce this issue now. We need more investigation on this issues. I will update if we have new finding.
hey @jack, thanks for the answer!
i noticed that in armbian with kernel 5.18, it does happen but less frequently.
could this [1] patch be benefical to be backported into armbian?
[1] https://lore.kernel.org/linux-arm-kernel/165459351568.925770.13686160465924068647.b4-ty@sntech.de/T/
1 Like
I notice that the armbian mainline kernel doesn’t have this patch, but my recent work on 5.19 rc1 has this patch involved: https://github.com/amazingfate/build/blob/rockchip64-5.19/patch/kernel/archive/rockchip64-5.19/rk356x-dts-pcie2x1.patch#L13
You can try my 5.19rc1 kernel: https://drive.google.com/drive/folders/1y4fYI87xFvOrChLhlyevPcZxeP8gHBQM?usp=sharing
Did you locate the problem?
Hi, i had found something about this problem, cloud you help me to collect some data?
i need some command result:
nvme get-feature /dev/nvme0 -f 0x0c -H | grep APST
smartctl -a /dev/nvme0
nvme command provided by ubuntu package nvme-cli
smartctl command provided by ubuntu package smartmontools
I also applied those ranges on 5.18 but still no luck, after 7 minutes I thought that it was working but after control-c-ing stress, it then failed.
nvme get-feature /dev/nvme0 -f 0x0c -H | grep APST
Autonomous Power State Transition Enable (APSTE): Enabled
smartctl -a /dev/nvme0
smartctl 7.2 2020-12-30 r5155 [aarch64-linux-5.18.4-rk35xx] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: CT1000P2SSD8
Serial Number: 2139E5D64ED1
Firmware Version: P2CR033
PCI Vendor/Subsystem ID: 0xc0a9
IEEE OUI Identifier: 0x6479a7
Total NVM Capacity: 1,000,204,886,016 [1.00 TB]
Unallocated NVM Capacity: 0
Controller ID: 1
NVMe Version: 1.3
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 6479a7 54e0000205
Local Time is: Sun Jun 19 20:24:24 2022 CEST
Firmware Updates (0x12): 1 Slot, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005e): Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0e): Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size: 64 Pages
Warning Comp. Temp. Threshold: 70 Celsius
Critical Comp. Temp. Threshold: 85 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 3.50W - - 0 0 0 0 0 0
1 + 1.90W - - 1 1 1 1 0 0
2 + 1.50W - - 2 2 2 2 0 0
3 - 0.0700W - - 3 3 3 3 5000 1900
4 - 0.0020W - - 4 4 4 4 13000 100000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 1
1 - 4096 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 45 Celsius
Available Spare: 100%
Available Spare Threshold: 5%
Percentage Used: 0%
Data Units Read: 620,612 [317 GB]
Data Units Written: 3,991,592 [2.04 TB]
Host Read Commands: 5,228,749
Host Write Commands: 31,533,130
Controller Busy Time: 317
Power Cycles: 606
Power On Hours: 1,743
Unsafe Shutdowns: 398
Media and Data Integrity Errors: 471,412
Error Information Log Entries: 476,199
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Error Information (NVMe Log 0x01, 16 of 16 entries)
Num ErrCount SQId CmdId Status PELoc LBA NSID VS
0 476199 0 0x3002 0x4005 0x028 0 0 -
thanks.
i have a solution now. follow this link change-kernel-params to add nvme_core.default_ps_max_latency_us=0 kernel param to disable nvme APST.
i think i should be work.