Pcie link downgraded with 10g NIC?

#6

After a closer look to the lspci, it seems to me that it is the pcie port that is downgraded and limiting the pcie link speed.

lspci -vv

root@rock-5b:~# lspci -vv
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 168
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: f0200000-f03fffff [size=2M]
Prefetchable memory behind bridge: 0000000900000000-00000009007fffff [size=8M]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=16/32 Maskable+ 64bit+
Address: 00000000fe670040 Data: 0000
Masking: fffffeff Pending: 00000000
Capabilities: [70] Express (v2) Root Port (Slot-), MSI 08
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x2 (downgraded)
TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
RootCap: CRSVisible-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR+
10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled, ARIFwd+
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
Vector table: BAR=4 offset=00020000
PBA: BAR=4 offset=00028000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
RootCmd: CERptEn- NFERptEn- FERptEn-
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsg 9
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Capabilities: [148 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: LaneErr at lane: 2 3
Capabilities: [190 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1- L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us
L1SubCtl2: T_PwrOn=10us
Capabilities: [1d0 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Capabilities: [2d0 v1] Vendor Specific Information: ID=0006 Rev=0 Len=018 <?>
Kernel driver in use: pcieport

0000:01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
** Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3]**
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 168
Region 0: Memory at f0200000 (64-bit, non-prefetchable)
Region 2: Memory at 900000000 (64-bit, prefetchable)
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Vital Product Data
Product Name: CX341A - ConnectX-3 SFP+
Read-only fields:
[PN] Part number: MCX341A-XCEN
[EC] Engineering changes: A6
[SN] Serial number: MT1422U01485
[V0] Vendor specific: PCIe Gen3 x8
[RV] Reserved: checksum good, 0 byte(s) reserved
Read/write fields:
[V1] Vendor specific: N/A
[YA] Asset tag: N/A
[RW] Read-write area: 105 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 252 byte(s) free
End
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Vector table: BAR=0 offset=0007c000
PBA: BAR=0 offset=0007d000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 116.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x2 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [148 v1] Device Serial Number f4-52-14-03-00-99-85-9c
Capabilities: [18c v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Kernel driver in use: mlx4_core
** Kernel modules: mlx4_core`**

So far, I had :-

  1. set the apsm = off / performance
  2. Set cpufreq governor = performance
  3. check the max-link-speed = <3> in dts

none of these works to bring the NIC to 10Gbps

the current iperf3 performance was low.
[SUM] 0.00-10.00 sec 3.79 GBytes 3.26 Gbits/sec 31 sender
[SUM] 0.00-10.00 sec 3.77 GBytes 3.24 Gbits/sec receiver

which is far from 10Gbps…
Is there any thing which could limit the linkspeed of pcie port for rk3588?

#7

After swapping the NIC with nvme, the pcie link speed resumed normal, so the problem should be with the card or driver?

lspci

root@rock-5b:~# lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0000:01:00.0 Non-Volatile memory controller: Yangtze Memory Technologies Co.,Ltd ZHITAI TiPro7000 (rev 01)
0001:10:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0001:11:00.0 Non-Volatile memory controller: Yangtze Memory Technologies Co.,Ltd Device 0001 (rev 03)
0002:20:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0002:21:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

sudo lspci -vvv

root@rock-5b:~# sudo lspci -vvv
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 149
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled]
Memory behind bridge: f0200000-f02fffff [size=1M]
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
Expansion ROM at f0300000 [virtual] [disabled] [size=64K]
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=16/32 Maskable+ 64bit+
Address: 00000000fe670040 Data: 0000
Masking: fffffeff Pending: 00000000
Capabilities: [70] Express (v2) Root Port (Slot-), MSI 08
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x2 (downgraded)
TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
RootCap: CRSVisible-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR+
10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled, ARIFwd+
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
Vector table: BAR=4 offset=00020000
PBA: BAR=4 offset=00028000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
RootCmd: CERptEn- NFERptEn- FERptEn-
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsg 9
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Capabilities: [148 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [190 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1- L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us
L1SubCtl2: T_PwrOn=10us
Capabilities: [1d0 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Capabilities: [2d0 v1] Vendor Specific Information: ID=0006 Rev=0 Len=018 <?>
Kernel driver in use: pcieport

0000:01:00.0 Non-Volatile memory controller: Yangtze Memory Technologies Co.,Ltd ZHITAI TiPro7000 (rev 01) (prog-if 02 [NVM Express])
Subsystem: Yangtze Memory Technologies Co.,Ltd ZHITAI TiPro7000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 139
Region 0: Memory at f0200000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (downgraded), Width x2 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: Upstream Port
Capabilities: [b0] MSI-X: Enable+ Count=66 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
Not readable
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [158 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [178 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [19c v1] Lane Margining at the Receiver <?>
Capabilities: [1b4 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 32, Total VFs: 32, Number of VFs: 0, Function Dependency Link: 00
VF offset: 256, stride: 256, Device ID: 5208
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1f4 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1fc v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Capabilities: [20c v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [244 v1] Data Link Feature <?>
Kernel driver in use: nvme

0001:10:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 140
Bus: primary=10, secondary=11, subordinate=11, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled]
Memory behind bridge: f1200000-f12fffff [size=1M]
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
Expansion ROM at f1300000 [virtual] [disabled] [size=64K]
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=16/32 Maskable- 64bit+
Address: 00000000fe670040 Data: 0000
Capabilities: [70] Express (v2) Root Port (Slot-), MSI 08
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x2, ASPM L1, Exit Latency L1 <16us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x2 (ok)
TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
RootCap: CRSVisible+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR+
10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled, ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
Vector table: BAR=4 offset=00020000
PBA: BAR=4 offset=00028000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
RootCmd: CERptEn- NFERptEn- FERptEn-
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsg 9
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Capabilities: [148 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [180 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1- L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=10us
L1SubCtl2: T_PwrOn=10us
Capabilities: [190 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Kernel driver in use: pcieport

0001:11:00.0 Non-Volatile memory controller: Yangtze Memory Technologies Co.,Ltd Device 0001 (rev 03) (prog-if 02 [NVM Express])
Subsystem: Yangtze Memory Technologies Co.,Ltd Device 1e49
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 172
Region 0: Memory at f1200000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x2 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable+ Count=16 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00002100
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [158 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [178 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [180 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Kernel driver in use: nvme

hdparm -Tt /dev/nvme1n1p2

root@rock-5b:~# hdparm -Tt /dev/nvme1n1p2

/dev/nvme1n1p2:
Timing cached reads: 7294 MB in 2.00 seconds = 3650.15 MB/sec
Timing buffered disk reads: [ 1454.590902] blk_update_request: I/O error, dev nvme1n1, sector 6300160 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
read(2097152) returned 471040 bytes
[ 1454.596752] blk_update_request: I/O error, dev nvme1n1, sector 6302616 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1454.596767] Buffer I/O error on dev nvme1n1p2, logical block 754803, async page read

hdparm -Tt /dev/nvme0n1p1

root@rock-5b:~# hdparm -Tt /dev/nvme0n1p1

/dev/nvme0n1p1:
Timing cached reads: 5310 MB in 2.00 seconds = 2656.15 MB/sec
Timing buffered disk reads: 4198 MB in 3.00 seconds = 1399.07 MB/sec

The nvme disk 1 seems to have some problem when testing, but it should be sufficient to test the pcie slot.

So, what should I do to tune or get the mlx4 to work at 8GT/s pcie gen3 mode… I am really run out of idea…:dizzy_face:

#8

Is the NIC an M.2 card or are you using a M.2 to PCIE adapter with a cable?

#9

the bifurcation board get m2 slot into two m2 slot. I’ve get a m2 slot to PCIE slot adapter for the NIC.
for the second Nvme test, I plug it into the PCIE slot adapter with additional m2-pcie slot card.

As the second nvme was able to reach 8GT/s, I think it is not the problem of hardware adapter in my case.

#10

@wong_disable Link the bifurcation board? Are you sure you’re getting enough power on 3.3V rail?

Most m.2 to PCIe adapters pull 3.3V directly from m.2 slot, it might be a problem if your PCIe cards pull enough from 3.3v and are used simultaneously with an SSD sitting on the very same 3.3v rail from 3A regulator…

#11

Thanks for the suggestion.

Yes, when using the bifurcation board only, it would be insufficient power for the pcie device, especially the power hungry 10G NIC which required 12V indeed.

Therefore, my set up actually included the 12V and 5V DC power supply for the standard PCIE slot…!

IMG_20240118_020115
IMG_20240118_020115.jpg5408×3680 3.47 MB

photo updated

[NAS Image] - OMV7 image for Rock 5B with ZFS, PCIe Bifurcation and 10G NIC
#12

To rule out the m.2 to PCIE adapter isn’t causing the downgrade, it is possible to plug something else into the PCIE slot (maybe wifi or usb expansion board)?

#13

Use pcie switch? :slight_smile:

#14

em… a good idea, i actually got an intel x520 and another mellanox cx341 and they were tested…
for x520, the pcie link became 5GT/s with lane reduced from 2 to one, so the total throughput was the same.
the other cx341 give me exactly the same situation that cap me at 3.3Gbps in iperf3… :cry:

maybe i could try another intel 2.5gbps nic this weekend… but rock5b actually built in with 2.5gbps :sweat_smile:

#15

how much for an pcie switch indeed? here in hongkong it cost nearly the cost of rock5b itself… and nearly 7 times of the cx341 for a 16x to 4x+4x+4x+4x pcie switch in gen4…:cry:

#16

From ConnectX-3 docs it states:

PCI Express (PCIe) Uses PCIe Gen 3.0 (1.1 and 2.0 compatible) through anx4 or x8 edge connector up to 8GT/s

The ConnectX®-3 adapter cards support PCI Express 3.0 (1.1 and 2.0 compatible) through an x4
or x8 edge connector. The following lists the PCIe interface features:
• PCIe Base 3.0 compliant, 1.1 and 2.0 compatible
• 2.5, 5.0, or 8.0GT/s link rate x8
• Auto-negotiates to x8, x4, x2, or x1

Since your using x2 lanes I guess there is no guarantee it should work as it requires 4 or 8 lanes to negotiate correctly. If you remove the bifurcation board and return the m.2 slot to use 4 lanes I would expect the ConnectX-3 to behave better.

#17

hello @mtx512rk, thanks for your sharing.
It seems to me that they are referring to the x4/x8 connector, where we actually do not have x2 connector actually exist in the market.
As I remember, the card work correctly without bifurcation before.

Anyways, I would try to plug the card without bifurcation board under the same image.
but that require recompiling of the kernel to include NIC, which take time :pensive:

#18

em… after taking a second look at the adaptor and the PCIE_x8 config from the manual shared by @mtx512rk, I think it would be worthy to check if the 3.3V has any deal with the additional power connector…
if the 3.3V is solely supplied by the m2 slot, maybe I should get additional 3.3V supply to the board…

PCIE_x8
PCIE_x8.PNG681×666 93.5 KB

1 Like
#19

I’m not convinced the 3.3v will help as your NIC does work but at Gen 1 speeds. I think the issue is with PCIE link training, what happens is that Rock 5b pice controller will first establish a link to the NIC at Gen 1 speed. If successful the pice controller will conduct link equalization processes at higher speeds ie Gen2/Gen3. For some reason this fails either because the quality of link or NICs pcie controller refuses for some reason. With 2 lanes your seeing the link stays a Gen 1 while with 1 lane it moves to Gen 2. Which suggests quality of the link is fine for Gen 2 but the NICs pcie is deciding to keep it at Gen 1 for 2 lanes. Hence my suggestion of trying 4 lanes. With 2 lanes the NIC might be separating TX & RX hence Gen 1 x2 which equates to Gen 2 x1.

#20

em… sometime the insufficient voltage could give funny result, anyways I would just check if the external power is there with multimeter first.

I agree that it is the NIC refuses to link up with higher speed during negotiation, where both of the link and device control say go for 8GT/s, the link status return as 2.5GT/s (downgraded). However, I am not able to get the cause for this.
Am I able to drill into the debug /fault message of the pcie controller of NIC?
As I remember there is a debug module for rk3588, I would try to get it for more debug message.

I believe the quality of link is actually OK for Gen3 8GT/s as I was able to get it by nvme SSD at thread 7. (Timing cached reads: 7294 MB in 2.00 seconds = 3650.15 MB/sec through hdparm)

Nvme seems would consume less power than NIC in the slot, so I doubt if the 3.3V is playing any role here…

1 Like
#21

I’m not sure how you plan to debug as the pcie negotiation is done at the hardware level, furthermore your dealing with older NIC cards. Personally I’d try a newer generation of card such as AQC107 which is Gen3 x4 or AQC113.

#22

Actually I have a AQC107 on my windows console… but hey! fun thing happened!
I’ve tried to retrain the pcie with script from here to gen 2
Force PCIe Gen2
And magically it works! now it give me a throughput at 6.26Gbps from iperf3!

3 Likes
#23

Great! You could try forcing it to Gen 3, I think just needs this line changed:

lc2n=$(printf "%08x" $((("0x$lc2" & 0xFFFFFFF0) | 0x2)))

to 0x3 instead of 0x2

2 Likes
#24

unfortunately the gen3 training was failed… ~anyways, i would try to get a samba test as practical case first!~ :slightly_smiling_face: thanks for your paients and guidance!!

A brief test on iperf3 is hereattached for reference

root@rock-5b:~# iperf3 -c 192.168.3.77 -P14
Connecting to host 192.168.3.77, port 5201
[ 5] local 192.168.3.30 port 39942 connected to 192.168.3.77 port 5201
[ 7] local 192.168.3.30 port 39944 connected to 192.168.3.77 port 5201
[ 9] local 192.168.3.30 port 39946 connected to 192.168.3.77 port 5201
[ 11] local 192.168.3.30 port 39948 connected to 192.168.3.77 port 5201
[ 13] local 192.168.3.30 port 39950 connected to 192.168.3.77 port 5201
[ 15] local 192.168.3.30 port 39952 connected to 192.168.3.77 port 5201
[ 17] local 192.168.3.30 port 39954 connected to 192.168.3.77 port 5201
[ 19] local 192.168.3.30 port 39956 connected to 192.168.3.77 port 5201
[ 21] local 192.168.3.30 port 39958 connected to 192.168.3.77 port 5201
[ 23] local 192.168.3.30 port 39960 connected to 192.168.3.77 port 5201
[ 25] local 192.168.3.30 port 39962 connected to 192.168.3.77 port 5201
[ 27] local 192.168.3.30 port 39964 connected to 192.168.3.77 port 5201
[ 29] local 192.168.3.30 port 39966 connected to 192.168.3.77 port 5201
[ 31] local 192.168.3.30 port 39968 connected to 192.168.3.77 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 26.8 MBytes 225 Mbits/sec 0 280 KBytes
[ 7] 0.00-1.00 sec 94.0 MBytes 788 Mbits/sec 0 297 KBytes
[ 9] 0.00-1.00 sec 28.6 MBytes 240 Mbits/sec 0 332 KBytes
[ 11] 0.00-1.00 sec 117 MBytes 982 Mbits/sec 0 280 KBytes
[ 13] 0.00-1.00 sec 80.8 MBytes 677 Mbits/sec 0 280 KBytes
[ 15] 0.00-1.00 sec 28.6 MBytes 240 Mbits/sec 0 280 KBytes
[ 17] 0.00-1.00 sec 56.8 MBytes 476 Mbits/sec 0 280 KBytes
[ 19] 0.00-1.00 sec 26.5 MBytes 223 Mbits/sec 0 280 KBytes
[ 21] 0.00-1.00 sec 116 MBytes 975 Mbits/sec 0 280 KBytes
[ 23] 0.00-1.00 sec 26.6 MBytes 223 Mbits/sec 0 288 KBytes
[ 25] 0.00-1.00 sec 62.9 MBytes 528 Mbits/sec 0 280 KBytes
[ 27] 0.00-1.00 sec 83.8 MBytes 702 Mbits/sec 0 262 KBytes
[ 29] 0.00-1.00 sec 50.8 MBytes 426 Mbits/sec 0 288 KBytes
[ 31] 0.00-1.00 sec 26.3 MBytes 221 Mbits/sec 0 315 KBytes
[SUM] 0.00-1.00 sec 826 MBytes 6.93 Gbits/sec 0

[ 5] 1.00-2.00 sec 52.6 MBytes 442 Mbits/sec 0 280 KBytes
[ 7] 1.00-2.00 sec 39.3 MBytes 329 Mbits/sec 1 297 KBytes
[ 9] 1.00-2.00 sec 110 MBytes 924 Mbits/sec 0 332 KBytes
[ 11] 1.00-2.00 sec 80.8 MBytes 678 Mbits/sec 0 280 KBytes
[ 13] 1.00-2.00 sec 86.0 MBytes 722 Mbits/sec 0 280 KBytes
[ 15] 1.00-2.00 sec 82.1 MBytes 689 Mbits/sec 0 280 KBytes
[ 17] 1.00-2.00 sec 82.5 MBytes 692 Mbits/sec 0 280 KBytes
[ 19] 1.00-2.00 sec 32.8 MBytes 275 Mbits/sec 0 280 KBytes
[ 21] 1.00-2.00 sec 64.1 MBytes 538 Mbits/sec 0 280 KBytes
[ 23] 1.00-2.00 sec 50.7 MBytes 426 Mbits/sec 0 288 KBytes
[ 25] 1.00-2.00 sec 33.2 MBytes 278 Mbits/sec 0 280 KBytes
[ 27] 1.00-2.00 sec 33.1 MBytes 277 Mbits/sec 0 262 KBytes
[ 29] 1.00-2.00 sec 33.4 MBytes 280 Mbits/sec 0 288 KBytes
[ 31] 1.00-2.00 sec 52.7 MBytes 443 Mbits/sec 0 315 KBytes
[SUM] 1.00-2.00 sec 833 MBytes 6.99 Gbits/sec 1

[ 5] 2.00-3.00 sec 48.7 MBytes 408 Mbits/sec 0 280 KBytes
[ 7] 2.00-3.00 sec 93.0 MBytes 780 Mbits/sec 0 297 KBytes
[ 9] 2.00-3.00 sec 150 MBytes 1.26 Gbits/sec 0 332 KBytes
[ 11] 2.00-3.00 sec 74.4 MBytes 624 Mbits/sec 0 280 KBytes
[ 13] 2.00-3.00 sec 108 MBytes 909 Mbits/sec 0 280 KBytes
[ 15] 2.00-3.00 sec 74.4 MBytes 624 Mbits/sec 0 280 KBytes
[ 17] 2.00-3.00 sec 51.4 MBytes 431 Mbits/sec 0 280 KBytes
[ 19] 2.00-3.00 sec 27.6 MBytes 231 Mbits/sec 0 280 KBytes
[ 21] 2.00-3.00 sec 28.2 MBytes 236 Mbits/sec 0 280 KBytes
[ 23] 2.00-3.00 sec 48.1 MBytes 404 Mbits/sec 0 288 KBytes
[ 25] 2.00-3.00 sec 27.5 MBytes 230 Mbits/sec 0 280 KBytes
[ 27] 2.00-3.00 sec 27.6 MBytes 231 Mbits/sec 0 262 KBytes
[ 29] 2.00-3.00 sec 26.4 MBytes 221 Mbits/sec 0 288 KBytes
[ 31] 2.00-3.00 sec 48.6 MBytes 407 Mbits/sec 0 315 KBytes
[SUM] 2.00-3.00 sec 834 MBytes 7.00 Gbits/sec 0

[ 5] 3.00-4.00 sec 52.1 MBytes 437 Mbits/sec 0 280 KBytes
[ 7] 3.00-4.00 sec 29.2 MBytes 245 Mbits/sec 0 297 KBytes
[ 9] 3.00-4.00 sec 29.4 MBytes 246 Mbits/sec 0 332 KBytes
[ 11] 3.00-4.00 sec 84.1 MBytes 706 Mbits/sec 0 280 KBytes
[ 13] 3.00-4.00 sec 85.4 MBytes 716 Mbits/sec 0 280 KBytes
[ 15] 3.00-4.00 sec 86.3 MBytes 724 Mbits/sec 0 280 KBytes
[ 17] 3.00-4.00 sec 85.4 MBytes 716 Mbits/sec 0 280 KBytes
[ 19] 3.00-4.00 sec 28.6 MBytes 240 Mbits/sec 0 280 KBytes
[ 21] 3.00-4.00 sec 162 MBytes 1.36 Gbits/sec 0 280 KBytes
[ 23] 3.00-4.00 sec 52.4 MBytes 440 Mbits/sec 0 288 KBytes
[ 25] 3.00-4.00 sec 28.7 MBytes 241 Mbits/sec 0 280 KBytes
[ 27] 3.00-4.00 sec 28.6 MBytes 240 Mbits/sec 0 262 KBytes
[ 29] 3.00-4.00 sec 28.0 MBytes 235 Mbits/sec 0 288 KBytes
[ 31] 3.00-4.00 sec 52.0 MBytes 437 Mbits/sec 0 315 KBytes
[SUM] 3.00-4.00 sec 832 MBytes 6.99 Gbits/sec 0

[ 5] 4.00-5.00 sec 78.2 MBytes 656 Mbits/sec 0 280 KBytes
[ 7] 4.00-5.00 sec 62.6 MBytes 525 Mbits/sec 0 297 KBytes
[ 9] 4.00-5.00 sec 28.6 MBytes 240 Mbits/sec 0 332 KBytes
[ 11] 4.00-5.00 sec 126 MBytes 1.06 Gbits/sec 0 280 KBytes
[ 13] 4.00-5.00 sec 29.0 MBytes 243 Mbits/sec 0 280 KBytes
[ 15] 4.00-5.00 sec 28.8 MBytes 242 Mbits/sec 0 280 KBytes
[ 17] 4.00-5.00 sec 85.2 MBytes 715 Mbits/sec 0 280 KBytes
[ 19] 4.00-5.00 sec 42.0 MBytes 352 Mbits/sec 0 280 KBytes
[ 21] 4.00-5.00 sec 125 MBytes 1.05 Gbits/sec 0 280 KBytes
[ 23] 4.00-5.00 sec 65.2 MBytes 547 Mbits/sec 0 288 KBytes
[ 25] 4.00-5.00 sec 28.6 MBytes 240 Mbits/sec 0 280 KBytes
[ 27] 4.00-5.00 sec 28.5 MBytes 239 Mbits/sec 0 262 KBytes
[ 29] 4.00-5.00 sec 27.4 MBytes 230 Mbits/sec 0 288 KBytes
[ 31] 4.00-5.00 sec 78.0 MBytes 654 Mbits/sec 0 315 KBytes
[SUM] 4.00-5.00 sec 833 MBytes 6.99 Gbits/sec 0

[ 5] 5.00-6.00 sec 59.3 MBytes 497 Mbits/sec 0 280 KBytes
[ 7] 5.00-6.00 sec 94.6 MBytes 794 Mbits/sec 0 297 KBytes
[ 9] 5.00-6.00 sec 72.5 MBytes 608 Mbits/sec 0 332 KBytes
[ 11] 5.00-6.00 sec 77.1 MBytes 647 Mbits/sec 0 280 KBytes
[ 13] 5.00-6.00 sec 33.7 MBytes 283 Mbits/sec 0 280 KBytes
[ 15] 5.00-6.00 sec 73.2 MBytes 614 Mbits/sec 0 280 KBytes
[ 17] 5.00-6.00 sec 34.3 MBytes 288 Mbits/sec 0 280 KBytes
[ 19] 5.00-6.00 sec 94.2 MBytes 791 Mbits/sec 0 280 KBytes
[ 21] 5.00-6.00 sec 76.7 MBytes 644 Mbits/sec 0 280 KBytes
[ 23] 5.00-6.00 sec 59.5 MBytes 499 Mbits/sec 0 288 KBytes
[ 25] 5.00-6.00 sec 33.6 MBytes 282 Mbits/sec 0 280 KBytes
[ 27] 5.00-6.00 sec 33.6 MBytes 282 Mbits/sec 0 262 KBytes
[ 29] 5.00-6.00 sec 33.0 MBytes 277 Mbits/sec 0 288 KBytes
[ 31] 5.00-6.00 sec 58.5 MBytes 491 Mbits/sec 0 315 KBytes
[SUM] 5.00-6.00 sec 834 MBytes 7.00 Gbits/sec 0

[ 5] 6.00-7.00 sec 50.5 MBytes 424 Mbits/sec 0 280 KBytes
[ 7] 6.00-7.00 sec 154 MBytes 1.29 Gbits/sec 0 297 KBytes
[ 9] 6.00-7.00 sec 78.7 MBytes 660 Mbits/sec 0 332 KBytes
[ 11] 6.00-7.00 sec 77.9 MBytes 654 Mbits/sec 0 280 KBytes
[ 13] 6.00-7.00 sec 28.9 MBytes 242 Mbits/sec 0 280 KBytes
[ 15] 6.00-7.00 sec 78.5 MBytes 658 Mbits/sec 0 280 KBytes
[ 17] 6.00-7.00 sec 71.3 MBytes 598 Mbits/sec 0 280 KBytes
[ 19] 6.00-7.00 sec 29.6 MBytes 248 Mbits/sec 0 280 KBytes
[ 21] 6.00-7.00 sec 77.5 MBytes 650 Mbits/sec 0 280 KBytes
[ 23] 6.00-7.00 sec 50.0 MBytes 419 Mbits/sec 0 288 KBytes
[ 25] 6.00-7.00 sec 28.9 MBytes 242 Mbits/sec 0 280 KBytes
[ 27] 6.00-7.00 sec 28.9 MBytes 242 Mbits/sec 0 262 KBytes
[ 29] 6.00-7.00 sec 29.2 MBytes 244 Mbits/sec 0 288 KBytes
[ 31] 6.00-7.00 sec 50.1 MBytes 420 Mbits/sec 0 315 KBytes
[SUM] 6.00-7.00 sec 834 MBytes 6.99 Gbits/sec 0

[ 5] 7.00-8.00 sec 50.2 MBytes 421 Mbits/sec 0 280 KBytes
[ 7] 7.00-8.00 sec 113 MBytes 952 Mbits/sec 1 297 KBytes
[ 9] 7.00-8.00 sec 66.8 MBytes 561 Mbits/sec 0 332 KBytes
[ 11] 7.00-8.00 sec 79.0 MBytes 663 Mbits/sec 0 280 KBytes
[ 13] 7.00-8.00 sec 33.0 MBytes 277 Mbits/sec 0 280 KBytes
[ 15] 7.00-8.00 sec 78.5 MBytes 659 Mbits/sec 0 280 KBytes
[ 17] 7.00-8.00 sec 113 MBytes 950 Mbits/sec 0 280 KBytes
[ 19] 7.00-8.00 sec 33.0 MBytes 277 Mbits/sec 0 280 KBytes
[ 21] 7.00-8.00 sec 67.0 MBytes 562 Mbits/sec 0 280 KBytes
[ 23] 7.00-8.00 sec 50.3 MBytes 422 Mbits/sec 0 288 KBytes
[ 25] 7.00-8.00 sec 32.9 MBytes 276 Mbits/sec 0 280 KBytes
[ 27] 7.00-8.00 sec 33.0 MBytes 277 Mbits/sec 0 262 KBytes
[ 29] 7.00-8.00 sec 32.7 MBytes 275 Mbits/sec 0 288 KBytes
[ 31] 7.00-8.00 sec 50.6 MBytes 424 Mbits/sec 0 315 KBytes
[SUM] 7.00-8.00 sec 834 MBytes 7.00 Gbits/sec 1

[ 5] 8.00-9.00 sec 59.0 MBytes 495 Mbits/sec 0 280 KBytes
[ 7] 8.00-9.00 sec 61.1 MBytes 513 Mbits/sec 0 297 KBytes
[ 9] 8.00-9.00 sec 96.7 MBytes 811 Mbits/sec 0 332 KBytes
[ 11] 8.00-9.00 sec 95.0 MBytes 797 Mbits/sec 0 280 KBytes
[ 13] 8.00-9.00 sec 30.3 MBytes 255 Mbits/sec 0 280 KBytes
[ 15] 8.00-9.00 sec 96.4 MBytes 809 Mbits/sec 0 280 KBytes
[ 17] 8.00-9.00 sec 61.0 MBytes 512 Mbits/sec 0 280 KBytes
[ 19] 8.00-9.00 sec 30.1 MBytes 253 Mbits/sec 0 280 KBytes
[ 21] 8.00-9.00 sec 93.6 MBytes 785 Mbits/sec 0 280 KBytes
[ 23] 8.00-9.00 sec 59.2 MBytes 497 Mbits/sec 0 288 KBytes
[ 25] 8.00-9.00 sec 30.3 MBytes 255 Mbits/sec 0 280 KBytes
[ 27] 8.00-9.00 sec 30.3 MBytes 255 Mbits/sec 0 262 KBytes
[ 29] 8.00-9.00 sec 30.4 MBytes 255 Mbits/sec 0 288 KBytes
[ 31] 8.00-9.00 sec 59.0 MBytes 495 Mbits/sec 0 315 KBytes
[SUM] 8.00-9.00 sec 833 MBytes 6.98 Gbits/sec 0

[ 5] 9.00-10.00 sec 99.5 MBytes 835 Mbits/sec 0 280 KBytes
[ 7] 9.00-10.00 sec 30.1 MBytes 252 Mbits/sec 0 297 KBytes
[ 9] 9.00-10.00 sec 30.6 MBytes 256 Mbits/sec 0 332 KBytes
[ 11] 9.00-10.00 sec 132 MBytes 1.11 Gbits/sec 0 280 KBytes
[ 13] 9.00-10.00 sec 29.5 MBytes 248 Mbits/sec 0 280 KBytes
[ 15] 9.00-10.00 sec 30.7 MBytes 257 Mbits/sec 0 280 KBytes
[ 17] 9.00-10.00 sec 30.3 MBytes 254 Mbits/sec 0 280 KBytes
[ 19] 9.00-10.00 sec 30.0 MBytes 252 Mbits/sec 0 280 KBytes
[ 21] 9.00-10.00 sec 132 MBytes 1.11 Gbits/sec 0 280 KBytes
[ 23] 9.00-10.00 sec 99.6 MBytes 836 Mbits/sec 0 288 KBytes
[ 25] 9.00-10.00 sec 29.6 MBytes 249 Mbits/sec 0 280 KBytes
[ 27] 9.00-10.00 sec 29.8 MBytes 250 Mbits/sec 0 262 KBytes
[ 29] 9.00-10.00 sec 29.7 MBytes 249 Mbits/sec 0 288 KBytes
[ 31] 9.00-10.00 sec 99.3 MBytes 833 Mbits/sec 0 315 KBytes
[SUM] 9.00-10.00 sec 833 MBytes 6.99 Gbits/sec 0

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 577 MBytes 484 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 576 MBytes 483 Mbits/sec receiver
[ 7] 0.00-10.00 sec 771 MBytes 647 Mbits/sec 2 sender
[ 7] 0.00-10.00 sec 770 MBytes 646 Mbits/sec receiver
[ 9] 0.00-10.00 sec 692 MBytes 581 Mbits/sec 0 sender
[ 9] 0.00-10.00 sec 691 MBytes 580 Mbits/sec receiver
[ 11] 0.00-10.00 sec 944 MBytes 792 Mbits/sec 0 sender
[ 11] 0.00-10.00 sec 943 MBytes 791 Mbits/sec receiver
[ 13] 0.00-10.00 sec 545 MBytes 457 Mbits/sec 0 sender
[ 13] 0.00-10.00 sec 544 MBytes 457 Mbits/sec receiver
[ 15] 0.00-10.00 sec 658 MBytes 552 Mbits/sec 0 sender
[ 15] 0.00-10.00 sec 657 MBytes 551 Mbits/sec receiver
[ 17] 0.00-10.00 sec 671 MBytes 563 Mbits/sec 0 sender
[ 17] 0.00-10.00 sec 670 MBytes 562 Mbits/sec receiver
[ 19] 0.00-10.00 sec 374 MBytes 314 Mbits/sec 0 sender
[ 19] 0.00-10.00 sec 373 MBytes 313 Mbits/sec receiver
[ 21] 0.00-10.00 sec 943 MBytes 791 Mbits/sec 0 sender
[ 21] 0.00-10.00 sec 942 MBytes 791 Mbits/sec receiver
[ 23] 0.00-10.00 sec 562 MBytes 471 Mbits/sec 0 sender
[ 23] 0.00-10.00 sec 561 MBytes 470 Mbits/sec receiver
[ 25] 0.00-10.00 sec 336 MBytes 282 Mbits/sec 0 sender
[ 25] 0.00-10.00 sec 335 MBytes 281 Mbits/sec receiver
[ 27] 0.00-10.00 sec 357 MBytes 300 Mbits/sec 0 sender
[ 27] 0.00-10.00 sec 356 MBytes 299 Mbits/sec receiver
[ 29] 0.00-10.00 sec 321 MBytes 269 Mbits/sec 0 sender
[ 29] 0.00-10.00 sec 320 MBytes 268 Mbits/sec receiver
[ 31] 0.00-10.00 sec 575 MBytes 482 Mbits/sec 0 sender
[ 31] 0.00-10.00 sec 574 MBytes 482 Mbits/sec receiver
[SUM] 0.00-10.00 sec 8.13 GBytes 6.98 Gbits/sec 2 sender
[SUM] 0.00-10.00 sec 8.12 GBytes 6.97 Gbits/sec receiver

iperf Done.

And the speed reach 7Gbps after using jumbo packet.

p.s. Samba image attached, not brilliant indeed, but better than before.
The good news is CPU usage is far from saturated
%E8%9E%A2%E5%B9%95%E6%93%B7%E5%8F%96%E7%95%AB%E9%9D%A2%202024-01-19%20010524

1 Like
#25

@wong_disable great that you managed to figure it out! could you please kindly provide a link to the bifurcation adapter you used?