I’ve got a fork of Talos Linux (a secure API-driven distro for Kubernetes) for the Rock 5 series:
This is unofficial, and I’ve got no affiliation with Talos/Sideros other than thinking they’re awesome. Please report any issues on my fork on GitHub if they’re Rock 5-related
I’ve adjusted the kernel defconfig as needed to get Kubernetes & Talos happy, feel free to PR general improvements – I’d love to get the kernel & module filesize down.
There are various changes to Talos itself to work around the current U-Boot & kernel support. Hopefully, these changes will slowly fade away until this fork can cease to exist; Talos already includes support for some Rock devices.
I’m using this for a home lab, so adjust your support expectations accordingly. (That said, everything is already running via GitHub Actions, so I’ll likely automate rebasing onto latest upstream in the near future.)
My current cluster has 3x Rock 5B (16GB) workers. I’ll look at getting 5A support included as well once feasible, but I don’t have that hardware right now (nor does anyone else )
My control plane is currently a single node older Intel Mac Mini that’s out of macOS support. I’ll likely switch this to a Rock 5A once those are available for reduced power consumption (though the Mac Mini is pretty good in this regard)
Few more details on my setup currently: 3x Rock 5Bs with 64 GB eMMC (Talos) and SK hynix P31 NVMe (currently using TopoLVM…which I also have arm64 images for now )
This setup is how Talos recommends running for SBCs (and in general): Talos gets its own entire device to partition and update as it wishes and then you (optionally) use a separate device for CSI storage. Given that, I have not messed with booting from NVMe at all
The Rock 5Bs have all been super stable, the only hiccup I’ve had since getting everything set up was the Radxa kernel swapped /dev/mmcblk0 and /dev/mccblk1 from being eMMC&SD respectively to SD&eMMC briefly and then back to eMMC&SD
(Also, I’m using the 30W Radxa USB-C PD adapters on all of them without issue)
Good stuff. I have been using Flatcar Linux for my production k8s deployments for awhile and I’ve always been aware of Talos but never took a good look at it.
Now that it’s the only minimal immutable OS for the rk3588 (thanks to your fork) I have to get it running to see if I like it.
TopoLVM is new to me but it’s still local storage correct? You’ve given me something to do this weekend.
Yes, TopoLVM is for local storage. I chose it for simplicity at the moment
FWIW I originally set up Mayastor (…and built arm64 images for that too) but it runs a busy loop on one core by design and to top things off, having huge pages enabled doesn’t play nice with common Postgres configs
If you run into trouble, beyond posting in this thread, feel free to open an issue on GitHub or ping me in the Radxa Discord (same username)
Just tried Talos on the Rock 5 but I couldn’t get it to boot. During initial boot I got a “Waiting 1 second for USB” message and it just hung there forever.
That example is using the latest published Collabora U-Boot build from https://hub.docker.com/r/milas/rock5-u-boot/tags?name=collabora and has a placeholder for an image with a Linux kernel in the format that the Talos installer expects (this is one thing that milas/rock5-toolchain generates)
Building a Docker image with the Collabora kernel would be substantially more involved but is doable. I don’t have that set up and don’t have any plans to do so imminently. At some point it’ll just be more practical to get EFI and everything working with upstream Talos and mainline Linux.
This may be a non-issue because if I wait long enough the boot process continues normally (about 3 minutes).
It turns out to be a quirk with one of my eMMC modules. If I have a NVMe drive installed I get the message after initial ramdisk loads:
“Waiting 1 second for USB”
The issue doesn’t happen if the NVMe is absent OR if I use another one of my eMMC modules. They all came from Allnetchina too. Strange.
Now to get Cilium running without kube proxy. That’s a challenge since extra kernel build flags are required and I’m learning this build environment as I have time.
I’ve refactored the builds/workflows a few times to try and keep my own sanity but it’s a lot between all the pieces.
If you clone both milas/rock5-toolchain and milas/rock5-talos, you can modify the Talos kernel config:
$ cd /path/to/rock5-talos/hack/boards/defconfig/
$ /path/to/rock5-toolchain/defconfig.sh
This will overwrite rockchip_linux_defconfig in /home/milas/dev/talos/hack/boards/defconfig
Continue (y/N)? y
...
After a bit, that will pop-up menuconfig, you can make changes, and then the result will be written back to the Talos repo.
As you can tell, I am evaluating several storage backends. How would I go about adding the Talos drbd system extension? Since it’s a kernel module it is tied to the kernel version and of course we are currently stuck on the bsp kernel. Talos builds drbd as part of upstream, so how would I include it in this fork of Talos to use?
Unfortunately, I don’t think it’s practical. Currently, drbd is not in-tree, and I couldn’t get it building against the BSP kernel. I don’t remember the exact error, but IIRC it scans the kernel headers to self-patch the drbd source code and fails due to API differences.
Ok good to know. Upstream kernel can’t get here fast enough.
I had Mayastor running for a few weeks. It was ok. I couldn’t saturate a 2.5Gb link and it keeps one core pegged at 100% by design for latency reasons. I ended up not wanting to use it long term.
TopoLVM looks like what I’ll use until the BSP kernel can go away.
So it looks like kernel 6.5 rc1 may be viable. How much of a lift would it be to get a bootable Talos image? The Talos devs said they only use the LTS kernel which means a long time from now before the rk3588 is officially available.
I was looking to tackle building against 6.5 rc1 myself but I remember there were some U-Boot hacks you had to do.
)
)