Rock5 as Samba file share

#1

Since we finally have pcie 3.0 x4 lanes (and 2.1 x1 lane), I have decided to expand my RockPi4 storage to Rock5. While I’m still waiting for a few parts, namely

Here is first tests of network&storage performance. At first I was testing speed with FIO, but it’s results always lower than diskspd or real performance, so I have choosed to use diskspd. Both U.2 NVMe drives in mdadm raid0

The graphs:
Reading:

image
image.png1658×813 13.3 KB

Writing:
image
image.png1658×782 13.1 KB

The raw data:
smb tests.zip (47.2 KB)

The scheme right now

The NIC is connected by using M.2 e-key (x1 pcie 2.1) to x8 pcie adapter

image
image.jpg1081×751 175 KB

iperf3 between Windows & Rock5

root@rock-5b:/home/rock# iperf3 -s

Server listening on 5201

Accepted connection from 192.168.1.43, port 50063
[ 5] local 192.168.1.45 port 5201 connected to 192.168.1.43 port 50064
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 376 MBytes 3.16 Gbits/sec 0 4.01 MBytes
[ 5] 1.00-2.00 sec 368 MBytes 3.08 Gbits/sec 0 4.01 MBytes
[ 5] 2.00-3.00 sec 368 MBytes 3.08 Gbits/sec 0 4.01 MBytes
[ 5] 3.00-4.00 sec 368 MBytes 3.08 Gbits/sec 0 4.01 MBytes
[ 5] 4.00-5.00 sec 368 MBytes 3.08 Gbits/sec 0 4.01 MBytes
[ 5] 5.00-6.00 sec 368 MBytes 3.08 Gbits/sec 0 4.01 MBytes
[ 5] 6.00-7.00 sec 368 MBytes 3.08 Gbits/sec 0 4.01 MBytes
[ 5] 7.00-8.00 sec 366 MBytes 3.07 Gbits/sec 0 4.01 MBytes
[ 5] 8.00-9.00 sec 368 MBytes 3.08 Gbits/sec 0 4.01 MBytes
[ 5] 9.00-10.00 sec 368 MBytes 3.08 Gbits/sec 0 4.01 MBytes
[ 5] 10.00-10.02 sec 6.25 MBytes 3.02 Gbits/sec 0 4.01 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.02 sec 3.60 GBytes 3.09 Gbits/sec 0 sender

root@rock-5b:/home/rock# iperf3 -c 192.168.1.43
Connecting to host 192.168.1.43, port 5201
[ 5] local 192.168.1.45 port 40424 connected to 192.168.1.43 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 357 MBytes 2.99 Gbits/sec 816 1.06 MBytes
[ 5] 1.00-2.00 sec 358 MBytes 3.00 Gbits/sec 0 1.45 MBytes
[ 5] 2.00-3.00 sec 362 MBytes 3.04 Gbits/sec 15 1.23 MBytes
[ 5] 3.00-4.00 sec 360 MBytes 3.02 Gbits/sec 1 970 KBytes
[ 5] 4.00-5.00 sec 362 MBytes 3.04 Gbits/sec 0 1.37 MBytes
[ 5] 5.00-6.00 sec 361 MBytes 3.03 Gbits/sec 1 1.13 MBytes
[ 5] 6.00-7.00 sec 361 MBytes 3.03 Gbits/sec 10 851 KBytes
[ 5] 7.00-8.00 sec 362 MBytes 3.04 Gbits/sec 0 1.30 MBytes
[ 5] 8.00-9.00 sec 361 MBytes 3.03 Gbits/sec 3 1.05 MBytes
[ 5] 9.00-10.00 sec 362 MBytes 3.04 Gbits/sec 0 1.45 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.52 GBytes 3.03 Gbits/sec 846 sender
[ 5] 0.00-10.00 sec 3.51 GBytes 3.01 Gbits/sec receiver

iperf Done.

The target scheme

image
image.png982×625 302 KB

Also thanks @tkaiser for advice regarding HELIOS lantest. Here is also results from it:

image

The m.2 to SATA is working with switch. The power consumption is much lesser than i expected.

IMG_20220908_165544
IMG_20220908_165544.jpg1846×4000 3.04 MB
IMG_20220908_165637
IMG_20220908_165637.jpg4000×1846 3.03 MB

2 Likes
Use ROCK 5 as NAS?
#2

because of all ssds/ pcie card / sbc has connect ground together, so you should use only one power supply to offer power

#3

Easy since ‘real performance’ since Windows 7 at least when copying with Windows Explorer is characterized by

  • auto-tuning block sizes
  • using parallel streams

https://www.helios.de/web/EN/support/TI/157.html

For your iperf tests… can you repeat them with server on the Windows PC and then

  • taskset -c 5 iperf3 -c 192.168.1.43
  • taskset -c 5 iperf3 -R -c 192.168.1.43 (reverse direction)

Also do you see interrupts being bottlenecked by cpu0 (checking /proc/interrupts afterwards and in a separate run checking while benchmarking with atop what’s going on)?

Of course testing with Helios LanTest would also be welcomed :slight_smile:

#4

That would explain fio vs explorer, but no fio vs diskspd. There is (average) ~7% difference based on 20 tests

Just a quick reminder, I’m bottlenecked by x1 pcie 2.1 lane, so 3.0Gb/s sounds true enough.

#5

Erm, no? The ground is still on power cables.
And It’s not high speed system, where I need to sync everything to reach max performance.

#6

Especially the 1st three numbers (not sequential transfer speeds) are super low and results variation (the orange triangles) is extremely high. You might want to compare with the 10GbE SMB numbers here if you want: https://github.com/openmediavault/openmediavault/issues/101#issuecomment-468270197 (made with macOS but shouldn’t matter since Microsoft should still be able to fabricate a better SMB client than Apple).

As for diskspd performance and iperf3 numbers. I was neither aware that your NIC card sits in the Key E slot nor that Explorer and diskspd numbers differed. Anyway: without knowing CLI parameters (count of parallel/asynchronous threads for example) it’s a bit pointless to compare and I’m still interested in /proc/interrupts output to get a better understanding of RK’s 5.10 BSP kernel and PCIe.

1 Like
#7

Well, connection wholly explained on scheme:
image
and as for diskspd cli - it’s all here

image
image.png1628×288 23.2 KB

And as for

the first run is a few hundreds ms, but later it’s becoming ~5s

#8

That took a bit more time than I expected, but I finally collected all necessary materials

IMG_20221203_193207
IMG_20221203_193207.jpg1732×2616 1.03 MB

2x A4 plasticine board
7x A5 plasticine board
3x DC-DC converted 20V to 5v based on XL4015
2x PD trigger for 20V5A and 1x PD trigger for 12V3A
2x QB826 as UPS to power up all of this
1x ANM24PE16 (PLX 8748)
2x JMB585
10x Samsung 870 EVO 1tb
3x noctua fans (one for board, one for DC-DC, one for ANM24PE16)
A lot of screws M3x5
A lot of Molex–2xMolex and Molex–2xSata_power

After a bit of soldering and drilling

:slight_smile:

IMG_20221208_225039
IMG_20221208_225039.jpg3456×1840 1.26 MB
IMG_20221208_225043
IMG_20221208_225043.jpg3208×1844 1.04 MB
IMG_20221209_004056
IMG_20221209_004056.jpg1846×4000 1.1 MB

This project (almost) finished.

IMG_20221209_022722
IMG_20221209_022722.jpg1840×3688 1.53 MB
IMG_20221209_022444
IMG_20221209_022444.jpg1840×3112 949 KB

image
image.jpg1846×4000 1.6 MB
image
image.png1069×490 14.1 KB

There two downsides -
first - ANM24PE16 requires cooling, it’s getting very hot very fast
second - there is no good way of monitor QB826

1 Like
#9

So 1 year and 3 months later, the NAS with 10 ssd drives powered by Power delivery trigger is still working just fine.

As for kernel it’s a bit tricky, since I dont update it because of NC552SFP that is connected to the top M.2 slot.

Right now its working by using mdadm raid5 from 10ssd drives. The roles are:

  • SMB share for personal usage
  • NFS share for vcenter
  • Torrent node
  • Nextcloud server
  • Decentralised java based image hosting server

The only problem I met is MOLEX cables, it’s actually better to use sata power cables because so far, there were a few situations (like 4 so far) where because of molex connection the ssd (from 1 to 4 ssd) were cut off, since they were powered off.

So mdadm raid had a bit of hiccup, but --force assemble always fixed it (and fsck.ext4).

#10

Hi There, fyi, I’m using 2x2,5gbe and smb 3.1.1 to get full 5gbe connection. It’s working wonderful with single connection - better than trunk port i used before… Bestimage

1 Like
#11

Welp, not bad at all :slight_smile:

But in my case I still have windows 7 and some other legacy stuff, so I’m gonna failover to smb2 anyway, not even taking in account that I only have csr305 and all 4 ports are taken

1 Like
#12

great to see your work! @Dante4
i was also using the hp 10gbps nic with rock 5b before as my nas setup and it’s really fast! (it gave me 7xxMB/s using [5B ram disk] <> [nvme SSD on desktop] file transfer in samba）. However, as the data in ramdisk is flashed jn each reboot and size is limited. I’ve came up with the idea of pcie bifurcation to separate two pcie3x2 slots for 10G NIC and nvme SSD in hope to get a decent 10Gbps NAS.

1 Like
#13

android
Screenshot_20240112_181108_com.gspace.android.jpg2376×1104 664 KB
android
Screenshot_20240112_180455_com.gspace.android.jpg2376×1104 1.04 MB