[NAS Image] - OMV7 image for Rock 5B with ZFS, PCIe Bifurcation and 10G NIC

Hello friends!

WIth the helps from this forum, radxa and skillful enthusiast, I had finally baked my OMV image for usage of NAS.


Transfer from SSD to SSD through samba with ARC of ZFS


Power consumption ranging from 50W peak(dual HDD start spinning from sleep) to 17W idle power (most of the time). During read/write to both HDD and SSD measured a consumption of ~35W

In short, you could try the image here.

  • the image is unstable and you are on your own :sweat_smile: If it set your house on fire, your cat dancing, your sandwich drop on floor, you have been warned.
  • The image was based on Radxa’s kernel 5.10 image, modified for PCIe3 2x + 2x bifurcation at m2 slot, upgraded to Bookworm with OMV7 and installed with ZFS
  • NIC tested are base on mellanox mlx4 and mlx5 driver, intel x520, boardcom be2net driver. 10Gbps iperf3 throughput is not gauranteed.
  • Yes, I’ve messed up the apt library when making ZFS works with this board. – to be fixed

I’ve tested the image for few weeks and it works so far.
If you run into problem, I would be glad if you could let me know.

Regards,
Vasco

3 Likes

Since radxa release the Rock 5B, I had been considering its ability to work as NAS to replace my RK3399 NAS used for years.

To make the most from the board, I would like to have:-

  • 10Gbps network ability
  • RAID 1 3.5" HDD with fast SSD to saturate the bandwidth
  • potential usage of ZFS
  • installation of OMV for NAS usage.
  • small footprint as a NAS

However, it took me nearly a year to get all these done :sweat_smile:


The whole set-up is housed inside a 5x 3.5" HDD cage, including Rock 5B, 2x 3.5 HDD, 1x 2280 nvme ssd, 10G NIC w/adaptor, 12V -5V regulated power supply and PWM-controlled 12V cooling fan – one 12V dc supply cable, one 10G spf+ cable and that’s it

First of all is the bandwidth. Rock 5B provided a 4 lanes PCIe 3.0 m-key m2 slot which in theory providing a bandwidth of 8GT/s x4, a total of 32GT/s, which is roughly 4GB/s if saturated.
There is another 1 lane PCIe 2.1 e-key m2 slot which I use it for SATA extension board for the 2x 3.5" HDD at 500MB/s.

To have a decent NAS with 10G NIC, I hope this board can work with ~ 1GB/s during file transfer.
The only hope to get two 1GB/s file transfer working would be bifurcation of PCIe 3.0 lanes form x4 into x2 + x2, which should be supported by RK3588 SoC.

This turn out to be the easiest part, as @Semi was generous enough to share us his hardwork from another thread and on altium.
Altium
Thread

However, to get the bifurcation work, the armbian path is not working since their mainline kernel is too advanced and not working with the bifurcation. After repeated trial of kernel recompilation, the kernel can only recognized the device in the first two lanes of the bifurcation board, this lead me back to Radxa official image.
With this image at kernel 5.10, we need to modify the Device Tree Sources to tell the kernel the PCIe 3.0 phy is running at bifurcation mode.
For a better reference, I would hereattach the dts I’ve used which could successfully load both of the PCIe slots.

rk3588-rock-5b.zip (5.1 KB)

For manual editing, I’ve altered ./linux/arch/arm64/boot/dts/rockchip/rk3588-rock-5b.dts
You will need to modify the following nodes:-

&pcie30phy {
rockchip,pcie30-phymode = <PHY_MODE_PCIE_NANBNB>; /* 3 modes available, x4, x2+x2, x1+x1+x1+x1*/
status = “okay”;
};

&pcie3x2 {
reset-gpios = <&gpio4 RK_PB4 GPIO_ACTIVE_HIGH>; /* check your gpio pin for prst*/
vpcie3v3-supply = <&vcc3v3_pcie30>;
rockchip,perst-inactive-ms = <1000>;
supports-clkreq;
max-link-speed = <3>; /* PCIe speed */
status = “okay”;
};

&pcie3x4 {
num-lanes = <2>; /* limit lanes for 2x+2x bifurcation*/
reset-gpios = <&gpio4 RK_PB6 GPIO_ACTIVE_HIGH>; /* check your gpio pin for prst*/
vpcie3v3-supply = <&vcc3v3_pcie30>;
rochchip,perst-inactive-ms = <1000>;
supports-clkreq;
max-link-speed = <3>;
status = “okay”;
};

Now we have both the PCIe slots detected in kernel. Please note that the current bifurcation board does not support NVME SSD to operate simutaneously in both slots. (disk will drop out when copying data between them) I suspected it could not provide sufficent 3.3V power for both SSD to operate and additional power shall be required if you want to do so.

Nevertheless, we need an adapter to fit PCIe slot NIC into m-key m2 slot.


This is the adapter I’ve used in this setup

Please note that the adapter actually require additional power input for 12V through the small 4pin plugs while drawing 3.3V directly from Rock5B. The NIC won’t be detected without 12V inputs.

Now we need to choose the NIC. I’ve tested quite a few 10NIC on hands, including Mellanox CX341 (mlx4), CX4411 (mlx5), Intel X520 (ixgbe) and Boardcom Blade Engine 3 (be2net). Most of them have bandwidth/ compatibility problem due to PCIe (Details refer to this thread) except CX4411 in my setup.
All these kernel modules were included in the NAS image.


For the NIC, I take some second hand OCP 10G NICs at good price, but make sure you use a proper adapter for OCP to PCIe.


Always choose the one with 5V converter and make sure no 12V goes into 5V. I’ve burnt a CX4411 for that :cry:

3 Likes

This seems to be complete project :slight_smile:
I’m also interested in such NAS, now completing components needed for all of those and plan cooling as well as some compact case for everything.
I also plan to get 10G nic at some point, that obviously needs seni board :slight_smile:
I’m not yet there at software level, I could not get good compact hardware RAID adapter card and have to use software RAID.

1 Like

This project is far from complete :sweat_smile:
I would like to log the components used and the development of this NAS.
I am squeezeing time to beef up the content in the threads but my work get rather busying recently.

Hope it could help the built of your NAS~ :grin:

There is always something to add or modify, as You mentioned kernel still needs some work on mainline to get biurcation ready there.
Wha about cooling? anything acitive? Anything for hard drives?
I’m investigating if hotswap for sata is possible on my setup. I’ll have 4 big 3.5 sata drives on ROCK 5B and next 4 on backup NAS based on ROCK 3B :slight_smile: And I’ll need to create my own case.

what 10G nic are You using now?

em… To my best knowledge, the current mainline kernel had no bifucation implementation through the DTS up to this moment, be it the latest 6.10 or the 6.1 SDK from Rockchip, so we had to wait and see.

The official image from Radxa was based on Bullseye, which only support OMV6, so I had manually upgraded it to Bookworm for tasting OMV7. It’s a beta version but I have no adverse experience so far. :crazy_face:

For the cooling, the 5x3.5 HDD case I used actually built with one 12cm case fan slot.
As I’ve mentioned in the thread, I’ve use the 12cm case fan with PWM controller module for a balance of cooling and auscutic noise.
The fan is sit at the back of the enclosure blowing air out, covering all components inside the cage, including the rock5B (only with heatsink), the two HDDs, the nvme SSD, the 10G NIC (only with heatsink), the SATA card and the power regulators for power supply (to be discussed in thread later on).


The Blow-them-all fan! :crazy_face:

For the 10G NIC I finally used, please refer to thread 2# :wink:

Uh lala…

You get an Armbian userland/image with 5.10 BSP kernel in Bookworm flavour: http://fi.mirror.armbian.de/dl/rock-5b/archive/

And then all you need to do is to run a modification of my initial OMV install script (part of Armbian’s build system back then to bake ready-to-use optimized OMV images for a variety of SBC): https://github.com/OpenMediaVault-Plugin-Developers/installScript

1 Like

Nothing beats custom made case. GReat work with ZFS btw!

2 Likes

OMG I am glad to see you here tkasier!

I noted the armbian is the best way to install OMV and it was my first choice indeed.
However, since there is quite a number of modifications I need on the kernel, (NIC driver, PCIe bifurcation, ZFS).

My original plan was to use the armbian built script. (I used it for building my Nanopi Neo4 image with 2.5G NIC, but the recent pcie fix on rk3399 is failed in my case btw). However, I was only able to use the mainline kernel source, which proved failed in PCIe bifurcation. When I try to retrivie the legacy kernel source of 5.10.x through the built script, I was always having verification error for 3 times and then the built script exit itself. I spent weeks on the script again and again but each time it failed at the download part so I switched to radxa official image. I would be glad if you could take a look to see if the data got any problem? :thinking:

Your 9xSSD array NAS is impressive!
Is that you are using USB step-up converter to get 12V for the SATA power cables?
What is your power scheme and power consumption for the 9 disks setup? :open_mouth:

x10 actually

And no, it’s actually step-down from 20v to 5v, since I’m using zmi25000 (as UPS) to power the whole system, so the purple chip is QC trigger for 20V then it’s step down to 5V10A for all the sata

As for power consumption, hm, never actually measured it for the whole system, but the battery is enough for 9-14 hours of powering the whole system

1 Like

Can it achieve full 10gb in transfer ?

I am sure HDD will be the bottleneck right ?

you may refer to the picture of the file transfer at my first post in the topic :wink:
It is 986 MB/s, back and forth are similar in speed.

As I mentioned, I got a nvme SSD alongside with 10gbps NIC

1 Like

to enable PCIe bifurcation, you did it by applying patch to kernel or u-boot? if u-boot, any downloadable u-boot image or patch file(s)?

Hello Enoch Leung!

The PCIe bifurcation was actually done through modifying the dts according to the PCIe developer guide from rockchip. It would take some effort to learn the correct pins to assign for rstn in pcie from the hardware schematics.

Since I use it for NAS with 10Gbps NIC, I just modify the dts and recompile the kernel with the required modules. So there is no patches / u-boot images.

However, there should have some debs for replacing the kernel. I could try to find them out if that is what you need.

I would also try to get the kernel patched for ZFS and ksmbd for better NAS performance.

u-boot would also use dts, and I want to update u-boot anyway to make it see my SSD, which is not detected at the moment and thus not bootable on my 5B. so, it is “easier” for me to just do PCIe bifurcation when u-boot loads. besides, I want to use openfyde and would like to NOT recompile kernel, hopefully it won’t override the dts defined by u-boot when kernel loads.

may you share the part you’d changed in dts? since you are using 5B as well, it should also work for me :slight_smile:

I am not sure if the dts part contribute to your u-boot case.

Usually the SSD case is detected or not. In some rare case, for e.g. you are using Intel Octane M20 which basically combining two storage device on the same m.2 pcb, it would require bifurcation to recognize both octane memory and ssd correctly.

For the bifurcation, of course I am happy to share! :slight_smile:

In my case, I’ve edited
./linux/arch/arm64/boot/dts/rockchip/rk3588-rock-5b.dts

You will need to modify the following nodes:-

&pcie30phy {
rockchip,pcie30-phymode = <PHY_MODE_PCIE_NANBNB>; /* 3 modes available, x4, x2+x2, x1+x1+x1+x1*/
status = “okay”;
};

&pcie3x2 {
reset-gpios = <&gpio4 RK_PB4 GPIO_ACTIVE_HIGH>; /* check your gpio pin for prst*/
vpcie3v3-supply = <&vcc3v3_pcie30>;
rockchip,perst-inactive-ms = <1000>;
supports-clkreq;
max-link-speed = <3>; /* PCIe speed */
status = “okay”;
};

&pcie3x4 {
num-lanes = <2>; /* limit lanes for 2x+2x bifurcation*/
reset-gpios = <&gpio4 RK_PB6 GPIO_ACTIVE_HIGH>; /* check your gpio pin for prst*/
vpcie3v3-supply = <&vcc3v3_pcie30>;
rochchip,perst-inactive-ms = <1000>;
supports-clkreq;
max-link-speed = <3>;
status = “okay”;
};

For those who wana play with a 10Gbps NAS but have no access to bifurcation board, you may consider Radxa Rock 5B Plus!

I’ve recently learnt that Radxa would soon release this board with tons of updates and native dual m2 PCIe slots with bifurcation built in! :star_struck:

1 Like

a small update~ the apt was finally fixed - sort of… can at least do some install n update without losing zfs module

ksmbd was so built for those who wana squeeze the performance :wink:

oh btw, i managed to got some photos of the rock 5b plus! let’s take a look :crazy_face:


:shushing_face:

6 Likes

Looks good,
Now let’s wait for “more competitive” pricing information :smiley:
Obviously it’s great when we need two slots and wanted to use wifi (and/or lte card). I’m also curious about ddr5 extra speed, somebody just reviewed orange pi 5 plus and failed to show that.

wow: also @tkaiser reported his insights about R5 ITX, seems that it will take some time to get anything more from DDR5

1 Like