CM3 + Taco no nvme on recent images

I can’t get nvme to work on any recent radxa-cm3-rpi-cm4-io images.

I know my taco and nvme drive work because I switched the radxa cm3 with an rpi cm4 and used an official raspbian image and it shows up fine.

I’ve tried radxa-cm3-rpi-cm4-io_debian_bullseye_cli_b15 and also building an image with:
./rbuild radxa-cm5-rpi-cm4-io bullseye cli

In all cases the behavior is the same - no nvme. I also tried a latest mainline kernel with ./bsp linux latest, and still no luck.

Output:

root@radxa-cm3-rpi-cm4-io:/home/rock# uname -a
Linux radxa-cm3-rpi-cm4-io 5.10.160-19-rk356x #eeb393dfb SMP Fri Oct 13 04:13:28 UTC 2023 aarch64 GNU/Linux
root@radxa-cm3-rpi-cm4-io:/home/rock# dmesg | grep pcie
[ 13.265251] rk-pcie 3c0000000.pcie: invalid prsnt-gpios property in node
[ 13.265277] rk-pcie 3c0000000.pcie: Looking up vpcie3v3-supply from device tree
[ 13.266243] rk-pcie 3c0000000.pcie: IRQ msi not found
[ 13.266316] rk-pcie 3c0000000.pcie: use outband MSI support
[ 13.266327] rk-pcie 3c0000000.pcie: Missing config reg space
[ 13.266377] rk-pcie 3c0000000.pcie: host bridge /pcie@fe260000 ranges:
[ 13.266424] rk-pcie 3c0000000.pcie: err 0x00f4000000…0x00f40fffff -> 0x00f4000000
[ 13.266464] rk-pcie 3c0000000.pcie: IO 0x00f4100000…0x00f41fffff -> 0x00f4100000
[ 13.266495] rk-pcie 3c0000000.pcie: MEM 0x00f4200000…0x00f5ffffff -> 0x00f4200000
[ 13.266515] rk-pcie 3c0000000.pcie: MEM 0x0300000000…0x033fffffff -> 0x0300000000
[ 13.266572] rk-pcie 3c0000000.pcie: Missing config reg space
[ 13.266705] rk-pcie 3c0000000.pcie: invalid resource
[ 13.474450] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x3
[ 13.499981] rk-pcie 3c0000000.pcie: PCIe Linking… LTSSM is 0x7
[ 13.583315] rk-pcie 3c0000000.pcie: PCIe Link up, LTSSM is 0x130011
[ 13.583610] rk-pcie 3c0000000.pcie: PCI host bridge to bus 0000:00
[ 13.718864] pcieport 0000:00:00.0: PME: Signaling with IRQ 79
[ 13.719285] pcieport 0000:01:00.0: enabling device (0000 -> 0003)
[ 13.719816] pcieport 0000:02:00.0: enabling device (0000 -> 0003)
[ 13.720858] pcieport 0000:02:02.0: enabling device (0000 -> 0003)

root@radxa-cm3-rpi-cm4-io:~# lspci
00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3566 (rev 01)
01:00.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
02:00.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
02:02.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
02:06.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
02:0e.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
03:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

Here is what I get from the taco with an rpi CM4 and official raspbian image:

root@raspberrypi:~# uname -a
Linux raspberrypi 6.1.0-rpi7-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux

root@raspberrypi:~# lspci -vvv
Linux raspberrypi 6.1.0-rpi7-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux

05:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5765DL NVMe SSD Controller (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
Subsystem: Realtek Semiconductor Co., Ltd. RTS5765DL NVMe SSD Controller (DRAM-less)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 34
Region 0: Memory at 600200000 (64-bit, non-prefetchable) [size=16K]
Region 5: Memory at 600204000 (32-bit, non-prefetchable) [size=8K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis+ LTR+ 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable+ Count=9 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [148 v1] Device Serial Number 00-00-00-01-00-4c-e0-00
Capabilities: [158 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [178 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [180 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=60us PortTPowerOnTime=60us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Capabilities: [190 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Kernel driver in use: nvme

Any suggestions? Will an official rbuild target for the cm3+taco be available one day?

Same problem, radxa its very bad support I’ve ever seen

It’s a shame because it is possible - I used the old e23 image and custom dtb from here: Taco + CM3 not seeing m2 SSD nor SATA drives this gets nvme working on taco + cm3

At least that gets me a working board but on a very old image and an old 4.x kernel.

I did spend some time trying to work out the differences between the e23 custom dtb and the one used by the radxa-cm3-rpi-cm4-io images - best I can tell is there is additional 3.3V routing to the nvme slot required but I haven’t been able to get it to work yet.

My current plan if no official image comes and I need to update will be to replace the radxa cm3 with an rpi cm4 on the taco so that I can get supported updates.

nvme support here - https://github.com/radxa-build/radxa-cm3-rpi-cm4-io/issues/3
new overlay here
https://github.com/radxa/overlays/blob/main/arch/arm64/boot/dts/rockchip/overlays/radxa-cm3-radxa-taco-pcie2x1.dts