Howto enable PCI-E on the Rockpi3A M2. E_key?

ROCK 3 Series 3A
1

Hi all, I’ve connected a M.2 e-key to PCI-e + USB adapter. Next I’ve plugged in a memory stick into the USB socket of that M.2 e-key adapter. This works fine, the memory stick is detected on usb 3-1.2, and i can access the files on the memory stick.

I have also connected a PICI-e to 20x sata interface card in the M.2 E-key adapter, and connected a bunch of sata drive to said interface card. Those are not detected at all.

See also the attached dmesg output.dmesg.zip (11.7 KB)

Could it be that the PCI-E lines on the e-key M.2 slote are disabled?
How can I enable those?
How can the file /boot/dtbs/rockchip/rk3568-rock-3a.dtb be changed to enable PCI-e on the E-key M.2 slot?

My hardware:
RockPi 3A version 1.31
M.2 e-key to PCIE+USB adapter: https://nl.aliexpress.com/item/32998283993.html
PCI-E to sata bridge: https://nl.aliexpress.com/item/1005003060227206.html
5 sata hard drives connected to the PCI-E to sata bridge
1 USB sick connected to the USB socket on the M.2 Ekeu to PCI-E and USB socket

My versions:
Manjaro KDE plasma
$ uname -a
Linux cedric-rock 6.0.12-1-MANJARO-ARM #1 SMP PREEMPT Thu Dec 8 14:18:34 UTC 2022 aarch64 GNU/Linux

2

I have downloaded the debian image from here. [1] That image works fine.
When I have access to the machine again, I’ll post the dmesg and the kernel version, and some benchmarks

[1] https://github.com/radxa-build/rock-3a/releases/tag/20221101-0101

3

The image you used is probably below, and the report below assumes the use of that image.

# sha1sum Manjaro-ARM-kde-plasma-rock3a-22.12.img.xz
7eaf0c6aa7b2dda04525370c1f9f4121b6086e05 Manjaro-ARM-kde-plasma-rock3a-22.12.img.xz

Could it be that the PCI-E lines on the E-key M.2 slote are disabled ?
How can I enable those ?

I think your guess is correct.
I have verified that all ‘PCIE-node’ are ‘disabled’ in ‘rk3568-rock-3a.dtb’.

Coincidentally, I have exactly the same Item (M.2 e-key to PCIE+USB adapter) as you.
* But ‘SATA-Card’ is different from the one you have.

Below we report the results before and after the change to ‘rk3568-rock-3a.dtb’.

# xz -d -c ./Manjaro-ARM-kde-plasma-rock3a-22.12.img.xz | dd of=/dev/mmcblk0 bs=8M status=progress oflag=direct iflag=fullblock

[ Original ]

# uname -a
Linux rock64 6.0.12-1-MANJARO-ARM #1 SMP PREEMPT Thu Dec 8 14:18:34 UTC 2022 aarch64 GNU/Linux

# dmesg | grep pcie
-- [none] ---

Below is the result when I change ‘rk3568-rock-3a.dtb’ and enable two ‘PCIE-node’.
For ‘SATA-Card’, the HDD drive is not connected because connecting the power cable is troublesome.

However, the ‘SATA-Card’ connected to ‘M.2: E-Key’ side is detected correctly.
And ‘nvme SSD’ connected to ‘M.2: M-key’ side is also detected correctly at the same time.

[After replace ‘rk3568-rock-3a.dtb’]

# dmesg | grep pcie
[ 0.290455] rockchip-dw-pcie 3c0000000.pcie: host bridge /pcie@fe260000 ranges:
[ 0.290521] rockchip-dw-pcie 3c0000000.pcie: IO 0x00f4100000..0x00f41fffff -> 0x00f4100000
[ 0.290553] rockchip-dw-pcie 3c0000000.pcie: MEM 0x00f4200000..0x00f5ffffff -> 0x00f4200000
[ 0.290573] rockchip-dw-pcie 3c0000000.pcie: MEM 0x0300000000..0x033fffffff -> 0x0000000000
[ 0.290849] rockchip-dw-pcie 3c0000000.pcie: iATU unroll: enabled
[ 0.290864] rockchip-dw-pcie 3c0000000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[ 0.509471] rockchip-dw-pcie 3c0000000.pcie: PCIe Gen.2 x1 link up
[ 0.509732] rockchip-dw-pcie 3c0000000.pcie: PCI host bridge to bus 0000:00
[ 0.523941] pcieport 0000:00:00.0: PME: Signaling with IRQ 26
[ 0.524446] pcieport 0000:00:00.0: AER: enabled with IRQ 26
[ 0.525900] rockchip-dw-pcie 3c0800000.pcie: host bridge /pcie@fe280000 ranges:
[ 0.525950] rockchip-dw-pcie 3c0800000.pcie: IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[ 0.525982] rockchip-dw-pcie 3c0800000.pcie: MEM 0x00f0200000..0x00f1ffffff -> 0x00f0200000
[ 0.526001] rockchip-dw-pcie 3c0800000.pcie: MEM 0x0380000000..0x03bfffffff -> 0x0080000000
[ 0.537821] rockchip-dw-pcie 3c0800000.pcie: iATU unroll: enabled
[ 0.537848] rockchip-dw-pcie 3c0800000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[ 0.749429] rockchip-dw-pcie 3c0800000.pcie: PCIe Gen.3 x2 link up
[ 0.749700] rockchip-dw-pcie 3c0800000.pcie: PCI host bridge to bus 0002:00
[ 0.782698] pcieport 0002:00:00.0: PME: Signaling with IRQ 28
[ 0.783158] pcieport 0002:00:00.0: AER: enabled with IRQ 28

# dmesg | grep ata
[ 0.824772] ata1: FORCE: PHY spd limit set to 3.0Gbps
[ 0.824788] ata1: SATA max UDMA/133 abar m8192@0xf4210000 port 0xf4210100 irq 36
[ 0.824801] ata2: FORCE: PHY spd limit set to 3.0Gbps
[ 0.824808] ata2: SATA max UDMA/133 abar m8192@0xf4210000 port 0xf4210180 irq 37
[ 0.824818] ata3: DUMMY
[ 0.824824] ata4: DUMMY
[ 0.824830] ata5: DUMMY
[ 1.160949] ata1: SATA link down (SStatus 0 SControl 320)
[ 1.491569] ata2: SATA link down (SStatus 0 SControl 320)

# dmesg | grep nvme
[ 0.815965] nvme nvme0: pci function 0002:01:00.0
[ 0.816050] nvme 0002:01:00.0: enabling device (0000 -> 0002)
[ 0.837109] nvme nvme0: allocated 61 MiB host memory buffer.
[ 0.987055] nvme nvme0: 4/0/0 default/read/poll queues

# lspci -vt
-+-[0000:00]---00.0-[01-ff]----00.0 JMicron Technology Corp. JMB58x AHCI SATA controller
  \-[0002:00]---00.0-[01-ff]----00.0 KIOXIA Corporation NVMe SSD Controller BG4

# ls -al /dev/nv*
crw------- 1 root root 248, 0 Jan 26 14:39 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Jan 26 14:39 /dev/nvme0n1

# dd if=/dev/nvme0n1 of=/dev/null bs=8M status=progress iflag=direct
128035676160 bytes (128GB, 119GiB) copied, 94.6728s, 1.4GB/s

Good luck.

4

How did you do that step? Is it just editing in the dtb file itself, or is it editing something else, and then compile it?

Are you willing to post the patch, or the complete files?

5

How did you do that step ?
Is it just editing in the dtb file itself, or is it editing something else, and then compile it ?

Are you willing to post the patch, or the complete files ?


The work procedure is as follows.


// Install device-tree compiler & patch
# pacman -S dtc patch

# dtc --version
Version: DTC 1.6.1

# patch --version
GNU patch 2.7.6

# diff --version
diff (GNU diffutils) 3.8

// Backup
# cp -v /boot/dtbs/rk3568-rock-3a.dtb /boot/dtbs/rk3568-rock-3a.dtb.bak

// Rebuild dtb
# dtc -O dts -I dtb -o rk3568-rock-3a.dts /boot/dtbs/rk3568-rock-3a.dtb
# patch -p1 -i rk3568-rock-3a.dts.diff
# dtc -O dtb -I dts -o rk3568-rock-3a.dtb rk3568-rock-3a.dts

// Replace dtb
# cp -v rk3568-rock-3a.dtb /boot/dtbs/

The work itself is just replacing one file, so I think 5 minutes is enough.

* Important subjects
‘Manjaro’ does not keep old versions.
Older versions are also removed from the server, so it’s impossible to get them later.
(e.g. ‘linux-6.0.12-1-aarch64.pkg.tar.zst’ has already been removed from the server)

Therefore, it is essential to back up the necessary files before proceeding.
If you neglect to do so, it will be impossible to recover if the operation fails.

‘rk3568-rock-3a.dts.diff.zip’ (sha1sum: 2ae88673830191c0c01f5e329c0d1cc815d01bce)
rk3568-rock-3a.dts.diff.zip (1.7 KB)

The ‘patch’ presented here this time has been made to work by fixing the following problem parts and others.
I chose ‘patch format’ because I expected it to be adaptable to ‘other Kernel-Ver’ or ‘other OS (Debian etc)’ depending on your efforts.

[1/1] arm64: dts: rockchip: rk356x: Fix PCIe register and range mappings

Above are the sources of information about the content of this fix.
Unfortunately, this ‘patch’ itself is wrong, as already pointed out in above thread.
And ‘Manjaro - kernel’ seems to take this ‘patch’ as-is.
So, just a simple change doesn’t work properly.
.

===

I would like to split the M.2 M-key PCI-e x2 port of the RockPi3A into 2 separate PCI express 3x1 slots …

Does the kernel understand such splitting ?


See below, there are boards that actually do that.
[PATCH v5 5/5] arm64: dts: rockchip: Add PCIe v3 nodes to BPI-R2-Pro

The device itself (SOC: rk3568), and the ‘Mainline Kernel’ seem to support it,
But the “rock-3a” board doesn’t seem to be able to support it.
* The “rock-3a” board’s hardware is not configured to support it.

The alternative suggested by ‘dominik’ in the ‘rock-5b’ thread is more realistic.
.

6

Finally, my mini PCI-e wifi card connected thought an adapter is detected in Linux 6.2 mainline, but with some issues. “ath10k” driver takes 184 seconds to detect the card, and then “ath” driver 124 seconds more to set the regulatory domain.

Builtin Ethernet stopped working, but I found a patch for the device tree here: https://lore.kernel.org/lkml/[email protected]/T/. Just change in “rk3568-rock-3a.dts” the line

compatible = "ethernet-phy-ieee802.3-c22";

to

compatible = "ethernet-phy-id001c.c916", "ethernet-phy-ieee802.3-c22";