This thread seems to focus “only” on 1x4 and 4x1 bifurcation. Looking in particular at the RK3588 TRM Vol.2 page 865, it seems that the PCIe3PHY_GRF_CMN_CON0 register supports in total 5 link modes: 1x4, 2x2, 1x2+2x2, 2x2+1x2 and finally 4x1. Maybe someone from Radxa can confirm this? How can we configure those modes? Is it possible by simply modifying the DTS?
Radxa Rock5 RK3588 SBC: PCIe lanes clarification
With the 5.10 kernel you could try to update the dts/dtsi file with something like this & change PHY_MODE_PCIE_AGGREGATION as required. However I noticed nodes pcie3x4 & pcie3x2 point to the same pcie30phy although I think only one of then can be enabled at anytime. Try it & see if it works!
&pcie30phy {
rockchip,pcie30-phymode = <PHY_MODE_PCIE_AGGREGATION>;
status = "okay";
};
Possible combinations (1x4, 2x2, 1x2+2x2, 2x2+1x2 ) below:
/*
* pcie30_phy_mode[2:0]
* bit2: aggregation
* bit1: bifurcation for port 1
* bit0: bifurcation for port 0
*/
#define PHY_MODE_PCIE_AGGREGATION 4 /* PCIe3x4 */
#define PHY_MODE_PCIE_NANBNB 0 /* P1:PCIe3x2 + P0:PCIe3x2 */
#define PHY_MODE_PCIE_NANBBI 1 /* P1:PCIe3x2 + P0:PCIe3x1*2 */
#define PHY_MODE_PCIE_NABINB 2 /* P1:PCIe3x1*2 + P0:PCIe3x2 */
#define PHY_MODE_PCIE_NABIBI 3 /* P1:PCIe3x1*2 + P0:PCIe3x1*2 */
1 Like
Thanks for the info @mtx512rk, i noted and actually read the trm and rockchip developer guide for pcie to prepare to compile my own kernel at both 6.6 and 5.10 from radxa repo. However, both of them get me either non-booting kernel or kernel with only one of the pcie slot working.
I doubt if the standard rst-gpio pin is correct from the source… If there is any deviation from rock5b, as check the example code, the gpio was quite different for pcie2.0… And the quoted pin for pcie3.0x2 is actually assigned to usb power…
Therefore i ask for clues to fix it…
For 5.10 kernel you would only need to compile the dtb file and copy it across. There is no need to change/enabel pice3x2. If your compiling on rock5b (although slow) the instructions on the wiki do work. However they do seem to rebuild the whole kernel/dtbs and repackage them everytime. I’ve used them when re-configuring pcie for testing a gpu card. To stop it rebuilding from scratch you can change a line in pack-kernel.sh to
cd ${KERNEL_DIR} && make && make -f $ROCKCHIP_BSP_DIR/build/kernel-package.mk kernel-package
If you wanting to cross compile on x86 then you need to export some environment variable to tell it to use arm gcc.
@mtx512rk Thanks a lot for the kernel code info. That confirms the 5 link modes. For the PCIe3.0 controller, the TRM mentions two dedicated PHYs (0 and 1) which work in aggregation mode for 1x4. It seems that the other 4 modes are using PHY0 and PHY1 separately and possibly bifurcate each PHY into 2 links with a single joint clock per PHY. I’m not an expert at PCIe at all, but what may be a limiting factor with the Radxa 5B board is that the x4 slot only provides the 1x4 single clock on standard pins. There’s a second clock signal (CLK_3P/N) provided on pins 44/46 which are usually not connected on any cards.
My interest was in particular support for Google’s dual edge TPU M.2 card (haven’t had the time to try yet) with the possibility to use the other 2 unsued PCIe lanes for other purposes.
@stylon @wong_disable Reviewed the TRM & understand how the bifurcation needs to be configured, its a combination of pcie controllers and the correct pins. Need to look at the schematics/dts to see if its possible or not.
@mtx512rk @stylon A quick update. I’ve finally managed to get the board working with both pcie slots! 
rock@rock-5b:~$ lspci -tvb
-±[0004:40]—00.0-[41]----00.0 Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
±[0002:20]—00.0-[21]----00.0 JMicron Technology Corp. JMB58x AHCI SATA controller
±[0001:10]—00.0-[11]----00.0 Intel Corporation SSD Pro 7600p/760p/E 6100p Series
-[0000:00]—00.0-[01-ff]----00.0 Yangtze Memory Technologies Co.,Ltd Device 0001
Both of the nvme SSD connected to the m-key slot and they are working now.
rock@rock-5b:~$ sudo hdparm -Tt /dev/nvme0n1p1
/dev/nvme0n1p1:
Timing cached reads: 5438 MB in 2.00 seconds = 2720.40 MB/sec
Timing buffered disk reads: 4296 MB in 3.00 seconds = 1431.83 MB/sec
rock@rock-5b:~$ sudo hdparm -Tt /dev/nvme1n1p1
/dev/nvme1n1p1:
Timing cached reads: 3702 MB in 2.00 seconds = 1851.65 MB/sec
Timing buffered disk reads: 2628 MB in 3.00 seconds = 875.51 MB/sec
After inspecting the board and schematic carefully, the problem was the bifurcation board utilized different GPIO pin from the development guide for perst signal.
2 Likes
That’s great ! I presume you had to set PHY_MODE_PCIE_NANBNB mode and set pcie3x4 with 2 lanes and enable pcie3x2 in your dts. However from the 5b schematics I thought you would be missing a reset gpio pin for pcei3x2 unless your is working without that? The driver uses the reset line as part of the pcie initialisation. Which bifurcation board are you using.
Yes, i’ve set the phy_mode_pcie as well as the settings on pcie3x4 and pcie3x2 for getting this working. For the schematic, the rst_gpio would be needed. I’ve set them for both pcie slots. For the additional slot, it actually routed to clkreq pin of original m-slot.
I’m also interested in which bifurcation board you used in particular what it does to the clock signals. AFAIK, you can’t use a single clock lane to drive 2 PCIe slots as the impedance halves if you try to do so. Clocks need to be “split” by an active component. Alternatively, you can use a reference clock chip same or similar to the one you can find in the Radxa 5B schematics (U5).
em… the board is actually a clone from from @Semi , who is generous enough to share us his hardwork from another thread and on altium.
For a better reference, I would hereattach the dts I’ve used which could successfully load both of the PCIe slots.
rk3588-rock-5b.zip (5.1 KB)
It is compiled under Kernel version 5.10.110.
I am currently working on the 10Gbps NIC… after that I would upload an image for this set up
2 Likes
Maybe looking at the device tree for the Friendlyelec CM3588 Nas Kit would help. It has 4 1x M.2 slots without a PCI-E switch.
1 Like
What do you mean? The NC552SFP working just fine, you just need to include drivers for it with kernel recompile, which arguably is simpler than pcie bifur
em… my problems is a little bit complicated after bifurcation.
you may take a look on my other threads and hope you could give me some advise on that 
https://forum.radxa.com/t/pcie-link-downgraded-with-10g-nic/19680