I’m not sure how you plan to debug as the pcie negotiation is done at the hardware level, furthermore your dealing with older NIC cards. Personally I’d try a newer generation of card such as AQC107 which is Gen3 x4 or AQC113.
Pcie link downgraded with 10g NIC?
Actually I have a AQC107 on my windows console… but hey! fun thing happened!
I’ve tried to retrain the pcie with script from here to gen 2
Force PCIe Gen2
And magically it works! now it give me a throughput at 6.26Gbps from iperf3!
3 Likes
Great! You could try forcing it to Gen 3, I think just needs this line changed:
lc2n=$(printf "%08x" $((("0x$lc2" & 0xFFFFFFF0) | 0x2)))
to 0x3 instead of 0x2
2 Likes
unfortunately the gen3 training was failed… ~anyways, i would try to get a samba test as practical case first!~ 
thanks for your paients and guidance!!
A brief test on iperf3 is hereattached for reference
root@rock-5b:~# iperf3 -c 192.168.3.77 -P14
Connecting to host 192.168.3.77, port 5201
[ 5] local 192.168.3.30 port 39942 connected to 192.168.3.77 port 5201
[ 7] local 192.168.3.30 port 39944 connected to 192.168.3.77 port 5201
[ 9] local 192.168.3.30 port 39946 connected to 192.168.3.77 port 5201
[ 11] local 192.168.3.30 port 39948 connected to 192.168.3.77 port 5201
[ 13] local 192.168.3.30 port 39950 connected to 192.168.3.77 port 5201
[ 15] local 192.168.3.30 port 39952 connected to 192.168.3.77 port 5201
[ 17] local 192.168.3.30 port 39954 connected to 192.168.3.77 port 5201
[ 19] local 192.168.3.30 port 39956 connected to 192.168.3.77 port 5201
[ 21] local 192.168.3.30 port 39958 connected to 192.168.3.77 port 5201
[ 23] local 192.168.3.30 port 39960 connected to 192.168.3.77 port 5201
[ 25] local 192.168.3.30 port 39962 connected to 192.168.3.77 port 5201
[ 27] local 192.168.3.30 port 39964 connected to 192.168.3.77 port 5201
[ 29] local 192.168.3.30 port 39966 connected to 192.168.3.77 port 5201
[ 31] local 192.168.3.30 port 39968 connected to 192.168.3.77 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 26.8 MBytes 225 Mbits/sec 0 280 KBytes
[ 7] 0.00-1.00 sec 94.0 MBytes 788 Mbits/sec 0 297 KBytes
[ 9] 0.00-1.00 sec 28.6 MBytes 240 Mbits/sec 0 332 KBytes
[ 11] 0.00-1.00 sec 117 MBytes 982 Mbits/sec 0 280 KBytes
[ 13] 0.00-1.00 sec 80.8 MBytes 677 Mbits/sec 0 280 KBytes
[ 15] 0.00-1.00 sec 28.6 MBytes 240 Mbits/sec 0 280 KBytes
[ 17] 0.00-1.00 sec 56.8 MBytes 476 Mbits/sec 0 280 KBytes
[ 19] 0.00-1.00 sec 26.5 MBytes 223 Mbits/sec 0 280 KBytes
[ 21] 0.00-1.00 sec 116 MBytes 975 Mbits/sec 0 280 KBytes
[ 23] 0.00-1.00 sec 26.6 MBytes 223 Mbits/sec 0 288 KBytes
[ 25] 0.00-1.00 sec 62.9 MBytes 528 Mbits/sec 0 280 KBytes
[ 27] 0.00-1.00 sec 83.8 MBytes 702 Mbits/sec 0 262 KBytes
[ 29] 0.00-1.00 sec 50.8 MBytes 426 Mbits/sec 0 288 KBytes
[ 31] 0.00-1.00 sec 26.3 MBytes 221 Mbits/sec 0 315 KBytes
[SUM] 0.00-1.00 sec 826 MBytes 6.93 Gbits/sec 0
[ 5] 1.00-2.00 sec 52.6 MBytes 442 Mbits/sec 0 280 KBytes
[ 7] 1.00-2.00 sec 39.3 MBytes 329 Mbits/sec 1 297 KBytes
[ 9] 1.00-2.00 sec 110 MBytes 924 Mbits/sec 0 332 KBytes
[ 11] 1.00-2.00 sec 80.8 MBytes 678 Mbits/sec 0 280 KBytes
[ 13] 1.00-2.00 sec 86.0 MBytes 722 Mbits/sec 0 280 KBytes
[ 15] 1.00-2.00 sec 82.1 MBytes 689 Mbits/sec 0 280 KBytes
[ 17] 1.00-2.00 sec 82.5 MBytes 692 Mbits/sec 0 280 KBytes
[ 19] 1.00-2.00 sec 32.8 MBytes 275 Mbits/sec 0 280 KBytes
[ 21] 1.00-2.00 sec 64.1 MBytes 538 Mbits/sec 0 280 KBytes
[ 23] 1.00-2.00 sec 50.7 MBytes 426 Mbits/sec 0 288 KBytes
[ 25] 1.00-2.00 sec 33.2 MBytes 278 Mbits/sec 0 280 KBytes
[ 27] 1.00-2.00 sec 33.1 MBytes 277 Mbits/sec 0 262 KBytes
[ 29] 1.00-2.00 sec 33.4 MBytes 280 Mbits/sec 0 288 KBytes
[ 31] 1.00-2.00 sec 52.7 MBytes 443 Mbits/sec 0 315 KBytes
[SUM] 1.00-2.00 sec 833 MBytes 6.99 Gbits/sec 1
[ 5] 2.00-3.00 sec 48.7 MBytes 408 Mbits/sec 0 280 KBytes
[ 7] 2.00-3.00 sec 93.0 MBytes 780 Mbits/sec 0 297 KBytes
[ 9] 2.00-3.00 sec 150 MBytes 1.26 Gbits/sec 0 332 KBytes
[ 11] 2.00-3.00 sec 74.4 MBytes 624 Mbits/sec 0 280 KBytes
[ 13] 2.00-3.00 sec 108 MBytes 909 Mbits/sec 0 280 KBytes
[ 15] 2.00-3.00 sec 74.4 MBytes 624 Mbits/sec 0 280 KBytes
[ 17] 2.00-3.00 sec 51.4 MBytes 431 Mbits/sec 0 280 KBytes
[ 19] 2.00-3.00 sec 27.6 MBytes 231 Mbits/sec 0 280 KBytes
[ 21] 2.00-3.00 sec 28.2 MBytes 236 Mbits/sec 0 280 KBytes
[ 23] 2.00-3.00 sec 48.1 MBytes 404 Mbits/sec 0 288 KBytes
[ 25] 2.00-3.00 sec 27.5 MBytes 230 Mbits/sec 0 280 KBytes
[ 27] 2.00-3.00 sec 27.6 MBytes 231 Mbits/sec 0 262 KBytes
[ 29] 2.00-3.00 sec 26.4 MBytes 221 Mbits/sec 0 288 KBytes
[ 31] 2.00-3.00 sec 48.6 MBytes 407 Mbits/sec 0 315 KBytes
[SUM] 2.00-3.00 sec 834 MBytes 7.00 Gbits/sec 0
[ 5] 3.00-4.00 sec 52.1 MBytes 437 Mbits/sec 0 280 KBytes
[ 7] 3.00-4.00 sec 29.2 MBytes 245 Mbits/sec 0 297 KBytes
[ 9] 3.00-4.00 sec 29.4 MBytes 246 Mbits/sec 0 332 KBytes
[ 11] 3.00-4.00 sec 84.1 MBytes 706 Mbits/sec 0 280 KBytes
[ 13] 3.00-4.00 sec 85.4 MBytes 716 Mbits/sec 0 280 KBytes
[ 15] 3.00-4.00 sec 86.3 MBytes 724 Mbits/sec 0 280 KBytes
[ 17] 3.00-4.00 sec 85.4 MBytes 716 Mbits/sec 0 280 KBytes
[ 19] 3.00-4.00 sec 28.6 MBytes 240 Mbits/sec 0 280 KBytes
[ 21] 3.00-4.00 sec 162 MBytes 1.36 Gbits/sec 0 280 KBytes
[ 23] 3.00-4.00 sec 52.4 MBytes 440 Mbits/sec 0 288 KBytes
[ 25] 3.00-4.00 sec 28.7 MBytes 241 Mbits/sec 0 280 KBytes
[ 27] 3.00-4.00 sec 28.6 MBytes 240 Mbits/sec 0 262 KBytes
[ 29] 3.00-4.00 sec 28.0 MBytes 235 Mbits/sec 0 288 KBytes
[ 31] 3.00-4.00 sec 52.0 MBytes 437 Mbits/sec 0 315 KBytes
[SUM] 3.00-4.00 sec 832 MBytes 6.99 Gbits/sec 0
[ 5] 4.00-5.00 sec 78.2 MBytes 656 Mbits/sec 0 280 KBytes
[ 7] 4.00-5.00 sec 62.6 MBytes 525 Mbits/sec 0 297 KBytes
[ 9] 4.00-5.00 sec 28.6 MBytes 240 Mbits/sec 0 332 KBytes
[ 11] 4.00-5.00 sec 126 MBytes 1.06 Gbits/sec 0 280 KBytes
[ 13] 4.00-5.00 sec 29.0 MBytes 243 Mbits/sec 0 280 KBytes
[ 15] 4.00-5.00 sec 28.8 MBytes 242 Mbits/sec 0 280 KBytes
[ 17] 4.00-5.00 sec 85.2 MBytes 715 Mbits/sec 0 280 KBytes
[ 19] 4.00-5.00 sec 42.0 MBytes 352 Mbits/sec 0 280 KBytes
[ 21] 4.00-5.00 sec 125 MBytes 1.05 Gbits/sec 0 280 KBytes
[ 23] 4.00-5.00 sec 65.2 MBytes 547 Mbits/sec 0 288 KBytes
[ 25] 4.00-5.00 sec 28.6 MBytes 240 Mbits/sec 0 280 KBytes
[ 27] 4.00-5.00 sec 28.5 MBytes 239 Mbits/sec 0 262 KBytes
[ 29] 4.00-5.00 sec 27.4 MBytes 230 Mbits/sec 0 288 KBytes
[ 31] 4.00-5.00 sec 78.0 MBytes 654 Mbits/sec 0 315 KBytes
[SUM] 4.00-5.00 sec 833 MBytes 6.99 Gbits/sec 0
[ 5] 5.00-6.00 sec 59.3 MBytes 497 Mbits/sec 0 280 KBytes
[ 7] 5.00-6.00 sec 94.6 MBytes 794 Mbits/sec 0 297 KBytes
[ 9] 5.00-6.00 sec 72.5 MBytes 608 Mbits/sec 0 332 KBytes
[ 11] 5.00-6.00 sec 77.1 MBytes 647 Mbits/sec 0 280 KBytes
[ 13] 5.00-6.00 sec 33.7 MBytes 283 Mbits/sec 0 280 KBytes
[ 15] 5.00-6.00 sec 73.2 MBytes 614 Mbits/sec 0 280 KBytes
[ 17] 5.00-6.00 sec 34.3 MBytes 288 Mbits/sec 0 280 KBytes
[ 19] 5.00-6.00 sec 94.2 MBytes 791 Mbits/sec 0 280 KBytes
[ 21] 5.00-6.00 sec 76.7 MBytes 644 Mbits/sec 0 280 KBytes
[ 23] 5.00-6.00 sec 59.5 MBytes 499 Mbits/sec 0 288 KBytes
[ 25] 5.00-6.00 sec 33.6 MBytes 282 Mbits/sec 0 280 KBytes
[ 27] 5.00-6.00 sec 33.6 MBytes 282 Mbits/sec 0 262 KBytes
[ 29] 5.00-6.00 sec 33.0 MBytes 277 Mbits/sec 0 288 KBytes
[ 31] 5.00-6.00 sec 58.5 MBytes 491 Mbits/sec 0 315 KBytes
[SUM] 5.00-6.00 sec 834 MBytes 7.00 Gbits/sec 0
[ 5] 6.00-7.00 sec 50.5 MBytes 424 Mbits/sec 0 280 KBytes
[ 7] 6.00-7.00 sec 154 MBytes 1.29 Gbits/sec 0 297 KBytes
[ 9] 6.00-7.00 sec 78.7 MBytes 660 Mbits/sec 0 332 KBytes
[ 11] 6.00-7.00 sec 77.9 MBytes 654 Mbits/sec 0 280 KBytes
[ 13] 6.00-7.00 sec 28.9 MBytes 242 Mbits/sec 0 280 KBytes
[ 15] 6.00-7.00 sec 78.5 MBytes 658 Mbits/sec 0 280 KBytes
[ 17] 6.00-7.00 sec 71.3 MBytes 598 Mbits/sec 0 280 KBytes
[ 19] 6.00-7.00 sec 29.6 MBytes 248 Mbits/sec 0 280 KBytes
[ 21] 6.00-7.00 sec 77.5 MBytes 650 Mbits/sec 0 280 KBytes
[ 23] 6.00-7.00 sec 50.0 MBytes 419 Mbits/sec 0 288 KBytes
[ 25] 6.00-7.00 sec 28.9 MBytes 242 Mbits/sec 0 280 KBytes
[ 27] 6.00-7.00 sec 28.9 MBytes 242 Mbits/sec 0 262 KBytes
[ 29] 6.00-7.00 sec 29.2 MBytes 244 Mbits/sec 0 288 KBytes
[ 31] 6.00-7.00 sec 50.1 MBytes 420 Mbits/sec 0 315 KBytes
[SUM] 6.00-7.00 sec 834 MBytes 6.99 Gbits/sec 0
[ 5] 7.00-8.00 sec 50.2 MBytes 421 Mbits/sec 0 280 KBytes
[ 7] 7.00-8.00 sec 113 MBytes 952 Mbits/sec 1 297 KBytes
[ 9] 7.00-8.00 sec 66.8 MBytes 561 Mbits/sec 0 332 KBytes
[ 11] 7.00-8.00 sec 79.0 MBytes 663 Mbits/sec 0 280 KBytes
[ 13] 7.00-8.00 sec 33.0 MBytes 277 Mbits/sec 0 280 KBytes
[ 15] 7.00-8.00 sec 78.5 MBytes 659 Mbits/sec 0 280 KBytes
[ 17] 7.00-8.00 sec 113 MBytes 950 Mbits/sec 0 280 KBytes
[ 19] 7.00-8.00 sec 33.0 MBytes 277 Mbits/sec 0 280 KBytes
[ 21] 7.00-8.00 sec 67.0 MBytes 562 Mbits/sec 0 280 KBytes
[ 23] 7.00-8.00 sec 50.3 MBytes 422 Mbits/sec 0 288 KBytes
[ 25] 7.00-8.00 sec 32.9 MBytes 276 Mbits/sec 0 280 KBytes
[ 27] 7.00-8.00 sec 33.0 MBytes 277 Mbits/sec 0 262 KBytes
[ 29] 7.00-8.00 sec 32.7 MBytes 275 Mbits/sec 0 288 KBytes
[ 31] 7.00-8.00 sec 50.6 MBytes 424 Mbits/sec 0 315 KBytes
[SUM] 7.00-8.00 sec 834 MBytes 7.00 Gbits/sec 1
[ 5] 8.00-9.00 sec 59.0 MBytes 495 Mbits/sec 0 280 KBytes
[ 7] 8.00-9.00 sec 61.1 MBytes 513 Mbits/sec 0 297 KBytes
[ 9] 8.00-9.00 sec 96.7 MBytes 811 Mbits/sec 0 332 KBytes
[ 11] 8.00-9.00 sec 95.0 MBytes 797 Mbits/sec 0 280 KBytes
[ 13] 8.00-9.00 sec 30.3 MBytes 255 Mbits/sec 0 280 KBytes
[ 15] 8.00-9.00 sec 96.4 MBytes 809 Mbits/sec 0 280 KBytes
[ 17] 8.00-9.00 sec 61.0 MBytes 512 Mbits/sec 0 280 KBytes
[ 19] 8.00-9.00 sec 30.1 MBytes 253 Mbits/sec 0 280 KBytes
[ 21] 8.00-9.00 sec 93.6 MBytes 785 Mbits/sec 0 280 KBytes
[ 23] 8.00-9.00 sec 59.2 MBytes 497 Mbits/sec 0 288 KBytes
[ 25] 8.00-9.00 sec 30.3 MBytes 255 Mbits/sec 0 280 KBytes
[ 27] 8.00-9.00 sec 30.3 MBytes 255 Mbits/sec 0 262 KBytes
[ 29] 8.00-9.00 sec 30.4 MBytes 255 Mbits/sec 0 288 KBytes
[ 31] 8.00-9.00 sec 59.0 MBytes 495 Mbits/sec 0 315 KBytes
[SUM] 8.00-9.00 sec 833 MBytes 6.98 Gbits/sec 0
[ 5] 9.00-10.00 sec 99.5 MBytes 835 Mbits/sec 0 280 KBytes
[ 7] 9.00-10.00 sec 30.1 MBytes 252 Mbits/sec 0 297 KBytes
[ 9] 9.00-10.00 sec 30.6 MBytes 256 Mbits/sec 0 332 KBytes
[ 11] 9.00-10.00 sec 132 MBytes 1.11 Gbits/sec 0 280 KBytes
[ 13] 9.00-10.00 sec 29.5 MBytes 248 Mbits/sec 0 280 KBytes
[ 15] 9.00-10.00 sec 30.7 MBytes 257 Mbits/sec 0 280 KBytes
[ 17] 9.00-10.00 sec 30.3 MBytes 254 Mbits/sec 0 280 KBytes
[ 19] 9.00-10.00 sec 30.0 MBytes 252 Mbits/sec 0 280 KBytes
[ 21] 9.00-10.00 sec 132 MBytes 1.11 Gbits/sec 0 280 KBytes
[ 23] 9.00-10.00 sec 99.6 MBytes 836 Mbits/sec 0 288 KBytes
[ 25] 9.00-10.00 sec 29.6 MBytes 249 Mbits/sec 0 280 KBytes
[ 27] 9.00-10.00 sec 29.8 MBytes 250 Mbits/sec 0 262 KBytes
[ 29] 9.00-10.00 sec 29.7 MBytes 249 Mbits/sec 0 288 KBytes
[ 31] 9.00-10.00 sec 99.3 MBytes 833 Mbits/sec 0 315 KBytes
[SUM] 9.00-10.00 sec 833 MBytes 6.99 Gbits/sec 0
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 577 MBytes 484 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 576 MBytes 483 Mbits/sec receiver
[ 7] 0.00-10.00 sec 771 MBytes 647 Mbits/sec 2 sender
[ 7] 0.00-10.00 sec 770 MBytes 646 Mbits/sec receiver
[ 9] 0.00-10.00 sec 692 MBytes 581 Mbits/sec 0 sender
[ 9] 0.00-10.00 sec 691 MBytes 580 Mbits/sec receiver
[ 11] 0.00-10.00 sec 944 MBytes 792 Mbits/sec 0 sender
[ 11] 0.00-10.00 sec 943 MBytes 791 Mbits/sec receiver
[ 13] 0.00-10.00 sec 545 MBytes 457 Mbits/sec 0 sender
[ 13] 0.00-10.00 sec 544 MBytes 457 Mbits/sec receiver
[ 15] 0.00-10.00 sec 658 MBytes 552 Mbits/sec 0 sender
[ 15] 0.00-10.00 sec 657 MBytes 551 Mbits/sec receiver
[ 17] 0.00-10.00 sec 671 MBytes 563 Mbits/sec 0 sender
[ 17] 0.00-10.00 sec 670 MBytes 562 Mbits/sec receiver
[ 19] 0.00-10.00 sec 374 MBytes 314 Mbits/sec 0 sender
[ 19] 0.00-10.00 sec 373 MBytes 313 Mbits/sec receiver
[ 21] 0.00-10.00 sec 943 MBytes 791 Mbits/sec 0 sender
[ 21] 0.00-10.00 sec 942 MBytes 791 Mbits/sec receiver
[ 23] 0.00-10.00 sec 562 MBytes 471 Mbits/sec 0 sender
[ 23] 0.00-10.00 sec 561 MBytes 470 Mbits/sec receiver
[ 25] 0.00-10.00 sec 336 MBytes 282 Mbits/sec 0 sender
[ 25] 0.00-10.00 sec 335 MBytes 281 Mbits/sec receiver
[ 27] 0.00-10.00 sec 357 MBytes 300 Mbits/sec 0 sender
[ 27] 0.00-10.00 sec 356 MBytes 299 Mbits/sec receiver
[ 29] 0.00-10.00 sec 321 MBytes 269 Mbits/sec 0 sender
[ 29] 0.00-10.00 sec 320 MBytes 268 Mbits/sec receiver
[ 31] 0.00-10.00 sec 575 MBytes 482 Mbits/sec 0 sender
[ 31] 0.00-10.00 sec 574 MBytes 482 Mbits/sec receiver
[SUM] 0.00-10.00 sec 8.13 GBytes 6.98 Gbits/sec 2 sender
[SUM] 0.00-10.00 sec 8.12 GBytes 6.97 Gbits/sec receiver
iperf Done.
And the speed reach 7Gbps after using jumbo packet.
p.s. Samba image attached, not brilliant indeed, but better than before.
The good news is CPU usage is far from saturated
1 Like
@wong_disable great that you managed to figure it out! could you please kindly provide a link to the bifurcation adapter you used?
@DIYprojectz Board was designed by @Semi and discussed in this thread. You need to get one made using the design files.
1 Like
That’s good one. And as for 3.3v - it’s actually worth checking, but 3.3v usually have no use whatsoever
Thanks!,
, as mentioned by @mtx512rk this wonderful board is came from generous work from @Semi!
I got mine based on semi’s works and my friend in China who is familiar with electronic stuff🙏
It would be a good idea to supplement the board with additional power socket for 3.3V.
just realized that I had made an serious mistake by testing with my sata SSD in workstation…
after shrinking the file size and move to RAM disk, the speed was more realistic for 7Gbps iperf3 result…
2 Likes
Hello all friends!
I am happy to share that I’ve finally overcome the PCIe link downgraded issue!
The problem seems to be with the NIC card.
The card I previously used was a Mellanox CX341A.
After swapping various cards, I can confirm that I can get the PCIe link working with 8GT/s @ 2 lanes with Mellanox CX4411A!
rock@rock-5b:~$ sudo lspci -vvv -s 0000:01:00.0
[sudo] password for rock:
0000:01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Subsystem: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 157
Region 0: Memory at 900000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at f0200000 [virtual] [disabled] [size=1M]
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x2 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [48] Vital Product Data
Product Name: CX4411A- ConnectX-4 LX SFP28
Read-only fields:
[PN] Part number: MCX4411A-ACAN
[EC] Engineering changes: BC
[SN] Serial number: MT1821K18888
[V0] Vendor specific: PCIeGen3 x8
[RV] Reserved: checksum good, 0 byte(s) reserved
End
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.19 GBytes 1.02 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.19 GBytes 1.02 Gbits/sec receiver
[ 7] 0.00-10.00 sec 560 MBytes 470 Mbits/sec 0 sender
[ 7] 0.00-10.00 sec 559 MBytes 469 Mbits/sec receiver
[ 9] 0.00-10.00 sec 864 MBytes 725 Mbits/sec 0 sender
[ 9] 0.00-10.00 sec 864 MBytes 725 Mbits/sec receiver
[ 11] 0.00-10.00 sec 1.30 GBytes 1.12 Gbits/sec 0 sender
[ 11] 0.00-10.00 sec 1.30 GBytes 1.12 Gbits/sec receiver
[ 13] 0.00-10.00 sec 884 MBytes 741 Mbits/sec 0 sender
[ 13] 0.00-10.00 sec 883 MBytes 741 Mbits/sec receiver
[ 15] 0.00-10.00 sec 708 MBytes 594 Mbits/sec 0 sender
[ 15] 0.00-10.00 sec 707 MBytes 593 Mbits/sec receiver
[ 17] 0.00-10.00 sec 960 MBytes 805 Mbits/sec 0 sender
[ 17] 0.00-10.00 sec 959 MBytes 805 Mbits/sec receiver
[ 19] 0.00-10.00 sec 497 MBytes 417 Mbits/sec 0 sender
[ 19] 0.00-10.00 sec 497 MBytes 417 Mbits/sec receiver
[ 21] 0.00-10.00 sec 707 MBytes 593 Mbits/sec 0 sender
[ 21] 0.00-10.00 sec 706 MBytes 593 Mbits/sec receiver
[ 23] 0.00-10.00 sec 746 MBytes 626 Mbits/sec 0 sender
[ 23] 0.00-10.00 sec 745 MBytes 625 Mbits/sec receiver
[ 25] 0.00-10.00 sec 539 MBytes 452 Mbits/sec 0 sender
[ 25] 0.00-10.00 sec 538 MBytes 452 Mbits/sec receiver
[ 27] 0.00-10.00 sec 877 MBytes 735 Mbits/sec 0 sender
[ 27] 0.00-10.00 sec 876 MBytes 735 Mbits/sec receiver
[ 29] 0.00-10.00 sec 1.11 GBytes 954 Mbits/sec 0 sender
[ 29] 0.00-10.00 sec 1.11 GBytes 953 Mbits/sec receiver
[ 31] 0.00-10.00 sec 596 MBytes 500 Mbits/sec 0 sender
[ 31] 0.00-10.00 sec 595 MBytes 499 Mbits/sec receiver
[SUM] 0.00-10.00 sec 11.4 GBytes 9.75 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 11.3 GBytes 9.75 Gbits/sec receiver
Thanks for all the assistance provided!
1 Like
Thats great! BTW was an AQC107 in your collection of tested cards?
Intel x520?
Welp as I mentioned the problem is pcie training and boot time of NIC.
I do have a AQC107 card but it is inside my windows console I am using so I had not yet tested it yet.
I was also planning to try RDMA in the next step so probably not going to use it later 
yes I tested Intel X520 which give me speed of PCIe2 x1
yes it is true that the problem is with pcie training, where training can only go up to Gen2 and beyond that, link equalization fail so it cannot go any further, so I could only say there is some error in the hardware/NIC, which I cannot afford to debug as PCIe debugging devices/software are pricy and simply not worthy for enthusiast IMO.
I was doubting the problem was with the driver, so I’ve search for NIC using newer driver, so I came across this card, which use mlx5 instead of mlx4
1 Like
More correctly it’s on Rockchip side, because pcie root is a bit bullshit. RK3399 for example doesn’t work with any pcie switch because reasons (uboot just fails at pcie init).
em… i do have different experience with RK3399. at first I got the RK3399 working with my CX341 as my first NAS, but this set-up is unstable and at limited speed at 5Gbps.
According to Rockchip PCIE developer guide, it sometime need as high as 3.8V at the 3.3V pcie supply to get some PCIE device to work stablely. In a recent release of Armbain they seems had solution to solve the long outstanding issue of PCIE on RK3399 platform but it seems not working for me 
Anyways, for this time, I suspect the root cause of the problem is from the OCP to PCIE adaptor or the NIC itself.
But as it is working, I would be shifting my focus to get ZFS and OMV installation at the moment 
Passive adapter cant be the reason for failed pcie training btw
Unless he is REALLY low quality
Fun fact, I were unable to make it run with Connectx-3 pro (40gb), it simply was not showing in lspci
it was really a simple adaptor, even without a proper power supply up to spec…
anyways it is resolved now