1st quick check for networking. It’s a RTL8125BG connected via PCIe Gen2 x1:
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
In TX direction everything is fine even when letting iperf3 run on a little core:
root@rock-5b:/home/rock# taskset -c 5 iperf3 -c 192.168.83.63
Connecting to host 192.168.83.63, port 5201
[ 5] local 192.168.83.113 port 47964 connected to 192.168.83.63 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 272 MBytes 2.28 Gbits/sec 41 573 KBytes
[ 5] 1.00-2.00 sec 281 MBytes 2.36 Gbits/sec 0 701 KBytes
[ 5] 2.00-3.00 sec 281 MBytes 2.36 Gbits/sec 0 728 KBytes
[ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec 0 735 KBytes
[ 5] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec 0 738 KBytes
[ 5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 741 KBytes
[ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec 0 745 KBytes
[ 5] 7.00-8.00 sec 281 MBytes 2.36 Gbits/sec 0 748 KBytes
[ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec 0 749 KBytes
[ 5] 9.00-10.00 sec 281 MBytes 2.36 Gbits/sec 0 749 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec 41 sender
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec receiver
iperf Done.
root@rock-5b:/home/rock# taskset -c 1 iperf3 -c 192.168.83.63
Connecting to host 192.168.83.63, port 5201
[ 5] local 192.168.83.113 port 47968 connected to 192.168.83.63 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 227 MBytes 1.90 Gbits/sec 244 479 KBytes
[ 5] 1.00-2.00 sec 282 MBytes 2.37 Gbits/sec 0 689 KBytes
[ 5] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec 0 730 KBytes
[ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec 0 744 KBytes
[ 5] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec 0 749 KBytes
[ 5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 755 KBytes
[ 5] 6.00-7.00 sec 281 MBytes 2.36 Gbits/sec 0 759 KBytes
[ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec 0 762 KBytes
[ 5] 8.00-9.00 sec 281 MBytes 2.36 Gbits/sec 0 765 KBytes
[ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec 0 768 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.69 GBytes 2.31 Gbits/sec 244 sender
[ 5] 0.00-10.00 sec 2.69 GBytes 2.31 Gbits/sec receiver
iperf Done.
But in RX direction performance sucks with defaults. IRQs 133 and 149 are on little cores. And even when moving them to big cores I’m stuck at 510 Mbits/sec. Also the weird cpufreq behaviour (even with performance cpufreq governor clocking the cores as low as 408 MHz) doesn’t help.
This is some stuff for deeper investigations. But let’s check PCIe powermanagement first:
root@rock-5b:/home/rock# cat /sys/module/pcie_aspm/parameters/policy
default performance [powersave] powersupersave
Well, that doesn’t look right. Let’s switch to default et voilà:
Even on a little core 2.5GbE link saturated:
root@rock-5b:/home/rock# taskset -c 1 iperf3 -R -c 192.168.83.63
Connecting to host 192.168.83.63, port 5201
Reverse mode, remote host 192.168.83.63 is sending
[ 5] local 192.168.83.113 port 47980 connected to 192.168.83.63 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 272 MBytes 2.29 Gbits/sec
[ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 4.00-5.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 5.00-6.00 sec 279 MBytes 2.34 Gbits/sec
[ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 8.00-9.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec sender
[ 5] 0.00-10.00 sec 2.73 GBytes 2.34 Gbits/sec receiver
iperf Done.