Fork of Talos Linux for Rock 5

#1

I’ve got a fork of Talos Linux (a secure API-driven distro for Kubernetes) for the Rock 5 series:

This is unofficial, and I’ve got no affiliation with Talos/Sideros other than thinking they’re awesome. Please report any issues on my fork on GitHub if they’re Rock 5-related :upside_down_face:

image
image.png1468×479 40.5 KB

I’ve adjusted the kernel defconfig as needed to get Kubernetes & Talos happy, feel free to PR general improvements – I’d love to get the kernel & module filesize down.

There are various changes to Talos itself to work around the current U-Boot & kernel support. Hopefully, these changes will slowly fade away until this fork can cease to exist; Talos already includes support for some Rock devices.

I’m using this for a home lab, so adjust your support expectations accordingly. (That said, everything is already running via GitHub Actions, so I’ll likely automate rebasing onto latest upstream in the near future.)

My current cluster has 3x Rock 5B (16GB) workers. I’ll look at getting 5A support included as well once feasible, but I don’t have that hardware right now (nor does anyone else :stuck_out_tongue:)

4 Likes
#2

Thank you for your hard work. Time for me to break out my 4 5Bs and get k8s installed on them.

What are you using for the control plane?

1 Like
#3

My control plane is currently a single node older Intel Mac Mini that’s out of macOS support. I’ll likely switch this to a Rock 5A once those are available for reduced power consumption (though the Mac Mini is pretty good in this regard)

Few more details on my setup currently: 3x Rock 5Bs with 64 GB eMMC (Talos) and SK hynix P31 NVMe (currently using TopoLVM…which I also have arm64 images for now :sweat_smile:)

This setup is how Talos recommends running for SBCs (and in general): Talos gets its own entire device to partition and update as it wishes and then you (optionally) use a separate device for CSI storage. Given that, I have not messed with booting from NVMe at all

The Rock 5Bs have all been super stable, the only hiccup I’ve had since getting everything set up was the Radxa kernel swapped /dev/mmcblk0 and /dev/mccblk1 from being eMMC&SD respectively to SD&eMMC briefly and then back to eMMC&SD

(Also, I’m using the 30W Radxa USB-C PD adapters on all of them without issue)

1 Like
#4

Good stuff. I have been using Flatcar Linux for my production k8s deployments for awhile and I’ve always been aware of Talos but never took a good look at it.

Now that it’s the only minimal immutable OS for the rk3588 (thanks to your fork) I have to get it running to see if I like it.

TopoLVM is new to me but it’s still local storage correct? You’ve given me something to do this weekend. :grin:

#5

I haven’t used it but I know folks like https://github.com/budimanjojo/talhelper, which might be useful

Yes, TopoLVM is for local storage. I chose it for simplicity at the moment

FWIW I originally set up Mayastor (…and built arm64 images for that too) but it runs a busy loop on one core by design and to top things off, having huge pages enabled doesn’t play nice with common Postgres configs

If you run into trouble, beyond posting in this thread, feel free to open an issue on GitHub or ping me in the Radxa Discord (same username)

1 Like
#6

Just tried Talos on the Rock 5 but I couldn’t get it to boot. During initial boot I got a “Waiting 1 second for USB” message and it just hung there forever.

Maybe I’ll try another day.

#7

Huh, I haven’t seen that. Were there any other logs leading up to that point?

#8

Hi, I saw your other git repo https://github.com/milas/rock5-toolchain.git
How difficult would it be to change to collaboras kernel/uboot for your talos build?

Just curius given that their Debian build is the only one I can boot on my board.

#9

I just tweaked the Makefile in the rock5-talos repo to make it possible to pass custom kernel/U-Boot:

make installer \
  IMAGE_NAME="rock5-talos" \
  ROCK5_BOARD="rock-5b" \
  PLATFORM="linux/arm64" \
  ROCK5_UBOOT='docker-image://milas/rock5-u-boot:latest-rock-5b-collabora' \
  ROCK5_KERNEL='<your context here>'

That example is using the latest published Collabora U-Boot build from https://hub.docker.com/r/milas/rock5-u-boot/tags?name=collabora and has a placeholder for an image with a Linux kernel in the format that the Talos installer expects (this is one thing that milas/rock5-toolchain generates)

Building a Docker image with the Collabora kernel would be substantially more involved :sweat_smile: but is doable. I don’t have that set up and don’t have any plans to do so imminently. At some point it’ll just be more practical to get EFI and everything working with upstream Talos and mainline Linux.

#10

This may be a non-issue because if I wait long enough the boot process continues normally (about 3 minutes).

It turns out to be a quirk with one of my eMMC modules. If I have a NVMe drive installed I get the message after initial ramdisk loads:

“Waiting 1 second for USB”

The issue doesn’t happen if the NVMe is absent OR if I use another one of my eMMC modules. They all came from Allnetchina too. Strange.

Now to get Cilium running without kube proxy. That’s a challenge since extra kernel build flags are required and I’m learning this build environment as I have time.

#11

Hopefully the new image from https://github.com/milas/rock5-talos/issues/8 helps

I’ve refactored the builds/workflows a few times to try and keep my own sanity but it’s a lot between all the pieces.

If you clone both milas/rock5-toolchain and milas/rock5-talos, you can modify the Talos kernel config:

$ cd /path/to/rock5-talos/hack/boards/defconfig/
$ /path/to/rock5-toolchain/defconfig.sh
This will overwrite rockchip_linux_defconfig in /home/milas/dev/talos/hack/boards/defconfig
Continue (y/N)? y
...

After a bit, that will pop-up menuconfig, you can make changes, and then the result will be written back to the Talos repo.

From there, you can rebuild the Talos kernel:

$ cd /path/to/rock5-talos
$ docker buildx bake kernel --set 'kernel.tags=docker.io/path/to/your-kernel:latest' --push

And then build a new Talos installer image with it:

$ cd /path/to/rock5-talos
$ make installer PUSH=1 IMAGE_NAME=rock5-talos PLATFORM=linux/arm64 ROCK5_KERNEL='docker-image://path/to/your-kernel'
1 Like
#12

Hi there. I have another custom build question.

As you can tell, I am evaluating several storage backends. How would I go about adding the Talos drbd system extension? Since it’s a kernel module it is tied to the kernel version and of course we are currently stuck on the bsp kernel. Talos builds drbd as part of upstream, so how would I include it in this fork of Talos to use?

Thanks!

#13

Unfortunately, I don’t think it’s practical. Currently, drbd is not in-tree, and I couldn’t get it building against the BSP kernel. I don’t remember the exact error, but IIRC it scans the kernel headers to self-patch the drbd source code and fails due to API differences.

You can see as far as I got here: https://github.com/milas/rock5-toolchain/commit/bc8a024713acf39e3fb2b7a94a64b55887f65027 (I went from planning on trying Mayastor, to hoping to try drbd, to settling on TopoLVM and not doing replication at the block layer.)

#14

Ok good to know. Upstream kernel can’t get here fast enough.

I had Mayastor running for a few weeks. It was ok. I couldn’t saturate a 2.5Gb link and it keeps one core pegged at 100% by design for latency reasons. I ended up not wanting to use it long term.

TopoLVM looks like what I’ll use until the BSP kernel can go away.

#15

So it looks like kernel 6.5 rc1 may be viable. How much of a lift would it be to get a bootable Talos image? The Talos devs said they only use the LTS kernel which means a long time from now before the rk3588 is officially available.

I was looking to tackle building against 6.5 rc1 myself but I remember there were some U-Boot hacks you had to do.

#16

It should be pretty straightforward to make a build with a different kernel - the Makefile actually already supports it by setting ROCK5_KERNEL to a local directory or something like docker-image://my-kernel:latest: https://github.com/milas/rock5-talos/blob/5ff0cec10ea4b60372db2d2d8c6d77b0a5a2165e/Makefile#L118-L120

So you could:

  1. Compile 6.5rc1 or Collabora fork
    • Need to generate a config - could probably use upstream Talos as base and make oldconfig?
  2. Copy the kernel image + modules into a new folder in the same format as Talos kernel images
     vmlinuz
 dtb/
   rockchip/
     rk3588-rock-5b.dtb
     overlay/
       rk3588-uart7-m2.dtbo
 lib/modules/
    (reference: https://github.com/milas/rock5-toolchain/blob/3fe53962afcd360db60f20a28b31db53b8b19cae/Dockerfile#L241-L245)
  3. Run make installer ROCK5_KERNEL=/path/to/kernel