Rock 5 + PEX8725 - dual 10Gbit + dual nvme

dominik · February 11, 2025, 6:23pm

I’m testing Rock 5B with pcie switch, mainly this particular card:

Instead of bifurcation (like on 5B+ or 5 ITX+) it connects to pcie switch (PEX8725) that has 2x AQC113C (10Gbit ethernet) + 2x m.2 4x pcie 3.0 (mainly for nvme, but other cards via extender can be used (I tested it with some like ASM1166).

Card is not very expensive, not much more than m.2 10G AQC cards and we have here PEX switch and two 10G adapters. Easy to find on ali, sometime discounted to about $125.

I think that it’s ultimate NAS card for RK3588

Of course it needs additional 12V power line and some adapter to convert m.2 to regular pcie slot.
Since 6.12 and 6.13 kernel has lot of pcie fixes I’m trying it with it, works on both 5B and 5b+ config.

It somehow WORKS (mostly), there are still some issues (ethernet controller is not detected at start, but easily added via rescan bus),

root@rock-5b-plus:~# lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0000:01:00.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:01:00.1 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
0000:01:00.2 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
0000:01:00.3 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
0000:01:00.4 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
0000:02:01.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:02:02.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:02:08.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:02:09.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:02:0a.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:83:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller (rev 03)
0000:ff:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller (rev 03)
0002:20:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

pcie rescan:

root@rock-5b-plus:~# echo 1 > /sys/bus/pci/rescan

this discovers missing AQC113C:

root@rock-5b-plus:~# lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0000:01:00.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:01:00.1 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
0000:01:00.2 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
0000:01:00.3 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
0000:01:00.4 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
0000:02:01.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:02:02.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:02:08.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:02:09.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:02:0a.0 PCI bridge: PLX Technology, Inc. PEX 8725 24-Lane, 10-Port PCI Express Gen 3 (8.0 GT/s) Multi-Root Switch with DMA (rev ca)
0000:03:00.0 Ethernet controller: Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)
0000:81:00.0 Ethernet controller: Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)
0000:83:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller (rev 03)
0000:ff:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller (rev 03)
0002:20:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

For now only one 10G ethernet is working, dmesg after rescan reports:

one fails to assign
the other one is working and connects at 10G with network,

I’ll experiment more on this, any ideas for those issues? @meco ? @willy ?

willy · February 11, 2025, 9:22pm

Interesting device! I faced comparable issues in the past on a dual-10G Mellanox NIC, both chips behind a PCIe bridge. I seem to remember (it was almost 10 years ago) that the issue was either a limited number of BAR or too small BAR that cannot cover the whole window for all devices behind the bridge, something like having to cover too large a window. But I don’t remember the details, and from what I remember, the limitation was in the PCIe controller so I never addressed it (it was on an armada370 so I didn’t really care, it was essentially for testing and discovery).

You should probably try to plug it into other machines (a PC and other ARM devices if you have any supporting PCIe slots), it could help you sort out what’s caused by hardware and what comes from software.

Regarding the rescan needed to make the aqc113 appear, probably that it takes too long to appear and the system doesn’t discover it instantly. Just guessing of course. It might be possible that the PLX automatically detects the devices surrounding it and needs some time to see links up etc, and is not ready in time at enumeration time. Pure speculation from me, of course.

dominik · February 12, 2025, 9:00am

In fact it was You who inspired me to search for something like this as I remember You also got 4x10G with pcie 2.0 switch tested on Rock 5B (the one that @tkaiser linked in his Rock 5B insights).

I still have some kernel flags to check, mainly realloc support (it helped some on nvidia forums with same errors with TX2 modules). Card is usable for now with rescan and just one ethernet, of course would be awesome to get it fully working

For sure I still have RockPro64 witch is RK3399 with regular size 4x pcie 2.0 slot. For x86 I need to dig a bit, but there have to be something in basement.

This looks like that,
This card on side has few status leds, and they flash several times on init. I’ll check for delay parameters to see what are the options and maybe just try to boot from slow SD to see if anything has changed.

wong_disable · February 12, 2025, 12:02pm

interesting… what is the output when you check for lspci -vvv?

dominik · February 12, 2025, 2:10pm

@wong_disable here it is: https://pastebin.com/LkRKzf0u

@geerlingguy has some BAR issues on CM4 modules and those needed dts change: https://gist.github.com/geerlingguy/9d78ea34cab8e18d71ee5954417429df

on 5B+ this section looks like:

ranges = <0x1000000 0x00 0xf3100000 0x00 0xf3100000 0x00 0x100000 0x2000000 0x00 0xf3200000 0x00 0xf3200000 0x00 0xe00000 0x3000000 0x00 0x4000
0000 0x09 0xc0000000 0x00 0x40000000>;

I’m not sure how can I increase those ranges here,

Also I found here something related to BAR and 4GB variant (this is the one I’m using).I’ll retry with 16G and 32G board i that changes anything.

I recompiled kernel with pci debug info, but not much more than this, few extra messages on dmesg.
Also checked if card can work with just two AQC113 and to my surprise now both won’t get correct BAR and don’t work. Here is the log for rescan and dmesg

meco · February 12, 2025, 6:49pm

That card looks very interesting. You should open an issue about it on https://github.com/HeyMeco/Rockchip-pcie-devices
Also what kernel did you use to run this?

dominik · February 12, 2025, 11:50pm

This is Your site, correct? I already found it, problem is somehow related to GPU on Rock5 attempts
I’ll try to add it there after some additional tests

This was edge kernel (6.13.2-edge-rockchip64), slightly modified config mainly for debug stuff
I think most things worked on earlier kernels,

Many interesting information about pcie linking, examples, debugging commands I found here:
https://xilinx.github.io/pcie-debug-kmap/pciedebug/build/html/docs/PCIe_Common_Issues/index.html#receiver-overflow

For now I’ll try with 32GB version and some nuc x86 box.

meco · February 13, 2025, 11:30am

Did you try with the latest vendor kernel? There has been some changes to PCIe bar handling that might help. Else you could try to build mainline with the PCIe patch from the 3090 doc in my pcie devices repo.

dominik · February 13, 2025, 4:02pm

I built it today and booted Rock5B 32GB with it,

root@rock-5b:~# uname -a
Linux rock-5b 6.1.99-vendor-rk35xx #1 SMP Wed Feb 12 08:55:33 UTC 2025 aarch64 GNU/Linux

lspci reports all four devices and everything is present at boot time (no need to rescan)

0000:03:00.0 Ethernet controller: Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)
0000:81:00.0 Ethernet controller: Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)
0000:83:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller (rev 03)
0000:ff:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller (rev 03)

but all four devices (two nvmes and acq) are not working,

quick grep still reports errors about BAR:
(example for first aqc)

root@rock-5b:~# dmesg | grep 0000:03:00.0
[    5.536556] pci 0000:03:00.0: [1d6a:14c0] type 00 class 0x020000
[    5.536626] pci 0000:03:00.0: reg 0x10: [mem 0x00000000-0x0007ffff 64bit]
[    5.536670] pci 0000:03:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[    5.536713] pci 0000:03:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[    5.536740] pci 0000:03:00.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
[    5.537121] pci 0000:03:00.0: supports D1 D2
[    5.537127] pci 0000:03:00.0: PME# supported from D0 D1 D3hot D3cold
[    5.537300] pci 0000:03:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x2 link at 0000:02:01.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link)
[    5.541790] pci 0000:03:00.0: BAR 4: no space for [mem size 0x00400000 64bit]
[    5.541797] pci 0000:03:00.0: BAR 4: failed to assign [mem size 0x00400000 64bit]
[    5.541805] pci 0000:03:00.0: BAR 0: no space for [mem size 0x00080000 64bit]
[    5.541811] pci 0000:03:00.0: BAR 0: failed to assign [mem size 0x00080000 64bit]
[    5.541819] pci 0000:03:00.0: BAR 6: no space for [mem size 0x00020000 pref]
[    5.541825] pci 0000:03:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
[    5.541833] pci 0000:03:00.0: BAR 2: no space for [mem size 0x00001000 64bit]
[    5.541840] pci 0000:03:00.0: BAR 2: failed to assign [mem size 0x00001000 64bit]
[    5.549835] atlantic: probe of 0000:03:00.0 failed with error -5

rescan adds some information about BAR:

root@rock-5b:~# dmesg | tail
[   31.357773] [GTP-ERR][__do_register_ext_module:79] Module [goodix-fwu] timeout
[  234.164957] r8169 0004:41:00.0: invalid VPD tag 0xff (size 0) at offset 0; assume missing optional EEPROM
[  285.392203] pcieport 0000:02:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 03-80] add_size 200000 add_align 100000
[  285.392254] pcieport 0000:02:09.0: bridge window [mem 0x00100000-0x000fffff] to [bus 83-fe] add_size 200000 add_align 100000
[  285.392284] pcieport 0000:01:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 02-ff] add_size 400000 add_align 100000
[  285.392307] pcieport 0000:00:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 01-ff] add_size 400000 add_align 100000
[  285.392415] pcieport 0000:00:00.0: BAR 8: assigned [mem 0xf0300000-0xf08fffff]
[  285.392439] pcieport 0000:01:00.0: BAR 8: assigned [mem 0xf0300000-0xf08fffff]
[  285.392457] pcieport 0000:02:01.0: BAR 8: assigned [mem 0xf0300000-0xf04fffff]
[  285.392470] pcieport 0000:02:09.0: BAR 8: assigned [mem 0xf0500000-0xf06fffff]

but it will not make them working

how about remove and rescan…

echo 1 > /sys/bus/pci/devices/0000\:03\:00.0/remove
lspci # gone 
echo 1 > /sys/bus/pci/rescan
lspci # added!

dmesg and atlantic reports that it HANGED:

[  668.856287] pci 0000:03:00.0: [1d6a:14c0] type 00 class 0x020000
[  668.856406] pci 0000:03:00.0: reg 0x10: [mem 0x00000000-0x0007ffff 64bit]
[  668.856475] pci 0000:03:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
[  668.856537] pci 0000:03:00.0: reg 0x20: [mem 0x00000000-0x003fffff 64bit]
[  668.856577] pci 0000:03:00.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
[  668.857110] pci 0000:03:00.0: supports D1 D2
[  668.857122] pci 0000:03:00.0: PME# supported from D0 D1 D3hot D3cold
[  668.859099] pci 0000:03:00.0: BAR 4: no space for [mem size 0x00400000 64bit]
[  668.859120] pci 0000:03:00.0: BAR 4: failed to assign [mem size 0x00400000 64bit]
[  668.859136] pci 0000:03:00.0: BAR 0: assigned [mem 0xf0300000-0xf037ffff 64bit]
[  668.859189] pci 0000:03:00.0: BAR 6: assigned [mem 0xf0380000-0xf039ffff pref]
[  668.859204] pci 0000:03:00.0: BAR 2: assigned [mem 0xf03a0000-0xf03a0fff 64bit]
[  668.859495] atlantic 0000:03:00.0: enabling device (0000 -> 0002)
[  669.060410] atlantic: Boot code hanged
[  669.060848] atlantic: probe of 0000:03:00.0 failed with error -110

Something is clearly wrong here, on mainline it brings up system without two 10GB nics, adds them after rescan and one of them gets ip and link

I will try that one later today or at morning. Thanks for this suggestion

dominik · February 17, 2025, 3:43pm

I checked that already on friday
no change regarding failed BAR memory and ACQ113 init, I’ve added several debug lines to be sure that fix has been applied and it logged three times,
Additionally I plugged in yet another cards (two sata adapter to both slots, one regular pcie LSI 8i and JMB58x, the one for a+e slot was detected with no issues, SAS card was not visible until rescan (like 10Gbit ethernets), so mainline kernel has issues with more pci cards than just this particular PLX switch.

Meco, how can I submit those cards to Your database? just by opening #?

meco · February 18, 2025, 1:04am

Yes + mention which board you used it with and which slot you connected to.