RockPi4 and 10Gbe SFP+

Good time of day everyone.
Since we have information that RK3588 will be used in RockPi5 - after a bit of googling i have discovered, that it have PCIE 2.0 AND PCIe 3.0 which leads us to possibility of having two m.2 slots (which one of possible way how RockPi5 will be). Which lead me to question, since we already have SATA hat is it possible to connect SFP+ 10 Gbe adapter.
I have tested two adapters (both are emulex):

The problem with Dell is that it takes eternity to loads up, so both Armbian and Ubuntu can’t initialize properly because of it and fails with Segmentation Fault, but HP is working. The problem with Dell is that it’s not only NIC, but also iSCSI initiator and he have a lot of extra stuff, so Armbian and Ubuntu most likely just timeout (if someone know how to recompile kernel with increased timeout - i’m happy to check this).
As for m.2 to PCIe - I’ve used this one

That’s HP output:
root@rockpi-4c:/home/rock# lspci

00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd RK3399 PCI Express Root Port
01:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
01:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
01:00.2 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
01:00.3 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
01:00.4 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
01:00.5 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
01:00.6 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
01:00.7 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)

So it does see it in pcie, but ifconfig shows me nothing. Also instead of 2 lines it shows me a way too many.
That leads me to thinking that there is no driver for it, so i found out this one.
One thing led to another and after a bit of digging I have discovered that I need to build kernel from scratch (since driver/Network/Ethernet/Emulex/be2net is not included by default in Kernel).
There is two ways, or to use vendor or to use armbian, since both of them are detecting HP - it’s all the same.
P.S. There was one problem that i met - for some reason there possible bad connection between m.2 to PCIe and m.2 expansion board, so after I reattached everything lspci showed my nothing, until i reattached m.2 to PCIe once again.

So after rebuilding image with driver included and forcing on my 10gbe switch 10 gbe advertisement - i was able to get 10 Gbe
2021-03-28%2012_03_17-root%40rockpi-4c_%20_home_rock
So testing (between Windows iperf3 and Ubuntu Focal (armbian) iperf3 showed me following (with gen1 pcie and default GH, changing to gen2 and 2.0 GHz not really changed anything)
2021-03-28%2022_40_16-root%40rockpi-4c_%20_home_rock

Next is testing 1TB Samsung 870 EVO over samba:
So default Samsung 870 EVO is showing me following in CrystalDiskMark (6 passes with 64 GB files with 0 sec pause) connected to PC by SATA (NTFS):
image
Connect to PC by USB 3.0 adapter (NTFS)
image
And following with fio (connected by USB 3.0 adapter)

fio --loops=5 --size=64GiB --filename=/mnt/fiotest.tmp --stonewall --ioengine=libaio --direct=1 \
--name=Seqread --bs=1m --rw=read \
--name=Seqwrite --bs=1m --rw=write 
Run status group 0 (all jobs):
   READ: bw=315MiB/s (331MB/s), 315MiB/s-315MiB/s (331MB/s-331MB/s), io=358GiB (384GB), run=1161457-1161457msec

Run status group 1 (all jobs):
  WRITE: bw=301MiB/s (316MB/s), 301MiB/s-301MiB/s (316MB/s-316MB/s), io=358GiB (384GB), run=1215237-1215237msec

And that’s connection over SMB (so limiting factor is USB 3.0 adapter for reading).
image

Just for reference that connection over 1 Gbe
0%20x64%20%5BADMIN%5D%20_REAL_

What is the CPU load when using iperf3 to run bandwidth test on RockPi4?

Around 98% if i remember correctly. Basically the limitations for iperf is indeed CPU

Hi Dante4

The NC552SFP you are testing is an eight lane (x8) PCI Express (PCIe) with 2 ports 10 Gigabit network, but RK3399 only support PCIe x4 lane. so the NC552SFP only one port can be used in RockPi4?

Limitation in case of rk3399 is not pcie lines, but cpu, you may use two ports, but cpu with just one wasn’t able to utilize all network bandwidth. Thats why i look forward to rock 5

I’m more interested in NAT forwarding performance of RockPi4 with NC552SFP, if two port can used for 1xWAN an 1xLAN port, I think it is able to utilize all network bandwidth。