Anybody running K3s on Rock 5B. I have just received my Rock 5Bs from Allnet looking forward to having some fun this weekend.
Running K3s on Rock 5B
Why ru posting in Rock 4?
moved to 5B, thanks.
Yes. I’m running a cluster of 5 Rock5bs. Runs k3s like a champ.
I’m running Armbian (not Radxa). You have to configure some iptables stuff before you install docker/containerd but other than that no issues at all. See here.
There is a small hitch if you are planning to run Longhorn for clustered storage as Armbian doesn’t have a required kernel module installed (for required to start the iscsid service, which Longhorn uses to present virtual disks in the CNI). I’m not sure if this is or is not included in the RADXA kernel build. I had to build my own custom image of Armbian to get that turned on (not really that hard, but some would find it daunting). I’m planning to offer a PR to the Armbian team to get that module included in the default build.
1 Like
Thanks @PigLover I am using Armbian and I am already building a custom image to add cloud init so I should be able to include the module require by iscsid. As for the iptable workaround I guess I can add that to my cloud init configs and have done at start.
What are you for sending you cloud-init config payload to your devices?
Oke, so you have no problems so far with Armbian? And which are you using? Because i want to use Armbian Jammy (CLI).
I found this script which you can use to flash your image and load the config but I won’t be using it as my ultimate usecase if you have people which little technical knowledge flashing the images to SD and loading them to devices in the field. So for may case I need to be able to customize the image itself to add the files. I will be trying to use this probably with some modifications.
If you wanna check out how to add cloud init check https://forum.armbian.com/topic/14616-cloud-init/?do=findComment&comment=110444
I am using Jammy minimal so far it is booting and connecting to wifi. I haven’t tried anything else but I will be playing with it a lot this weekend.
gotcha. Yeah for servicing normies in field it’s easier to add a firstboot script to your image tries to automount a fat formatted USB key with a config payload.
I wish cloud-init no cloud had some native support for reading config values from SPI or uboot vars.
On my to-do list to explore some day
Jammy CLI at the moment.
The full setup o k8s works just fine.
If you want to use the Ceph however (bad idea), only the old protocol is supported by the kernel.
Easy enough to build a kernel with Ceph modules using Armbian. Their build environment is really clean and easy enough for any user advanced enough to install k8s w/Ceph.
FWIW I went down another path using microk8s on Radxa Ubuntu. Ok so far, I don’t know why I didn’t hit the missing module issue https://github.com/rockchip-linux/kernel/issues/273 but that’s fixed now anyway.
I’ve tried with armbian and ubuntu to run k3s. K3s starts but pods can’t run for some reason seems to be a cni issue. I did do the iptables updates etc but still the same thing. Whats steps did you follow?
Can you grab the kubelet.log output and share here? Anything in kernel logs about missing modules etc?
I’m not running K3s/Ubuntu, but here’s a defconfig that’s working well for me with Kubernetes (using Flannel as CNI):
You might diff it with your config from Armbian to see if you see any big differences around iptables/networking and try to adjust your Armbian kernel config accordingly. I went through a lot of trial and error originally to adjust the stock Radxa config to be suitable for Talos/Kubernetes.
So currently I start with a fresh armbian install then do the following:
apt update
apt upgrade
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
iptables -F
then run k3s install.
it installs then I get broken pipe from the shell
the logs say
systemd[1]: Caught <SEGV>, dumped core as pid 3895.
Broadcast message from systemd-journald@rock-5b (Wed 2023-07-05 10:45:15 SAST):
systemd[1]: Freezing execution.
Jul 05 10:44:40 rock-5b k3s[3670]: /go/pkg/mod/github.com/k3s-io/kine@v0.10.1/pkg/logstructured/logstructured.go:306 +0x58 ```
the rest of the logs just seems gibirish seems k3s just stuck in a bootloopok I’m almost there but not quite comming right with talos docs:
I flashed the image and it boots
I can list disks using the talosctl tool.
talosctl gen config --with-secrets secrets.yaml --kubernetes-version 1.25.4 my-cluster https://192.168.88.38:6443
I fix the config to reflect the nvme then run
talosctl apply-config --insecure --nodes 192.168.88.38 --file controlplane.yaml
Then it locks me out with auth failure.
How do I hookup the kubeconfig form here?
so got the cluster booting now but dies soon after.
192.168.88.38: user: warning: [2023-07-05T16:11:25.784134023Z]: [talos] created rbac.authorization.k8s.io/v1/ClusterRole/system:coredns {"component": "controller-runtime", "controller": "k8s.ManifestApplyController"}
192.168.88.38: user: warning: [2023-07-05T16:11:26.174739023Z]: [talos] created /v1/ConfigMap/coredns {"component": "controller-runtime", "controller": "k8s.ManifestApplyController"}
192.168.88.38: user: warning: [2023-07-05T16:11:26.595857023Z]: [talos] created apps/v1/Deployment/coredns {"component": "controller-runtime", "controller": "k8s.ManifestApplyController"}
192.168.88.38: user: warning: [2023-07-05T16:11:26.992227023Z]: [talos] created /v1/Service/kube-dns {"component": "controller-runtime", "controller": "k8s.ManifestApplyController"}
192.168.88.38: user: warning: [2023-07-05T16:11:27.379441023Z]: [talos] created /v1/ConfigMap/kubeconfig-in-cluster {"component": "controller-runtime", "controller": "k8s.ManifestApplyController"}
192.168.88.38: user: warning: [2023-07-05T16:11:32.187742023Z]: [talos] task labelNodeAsControlPlane (1/1): done, 1m28.473281576s
192.168.88.38: user: warning: [2023-07-05T16:11:32.188144023Z]: [talos] phase labelControlPlane (19/21): done, 1m28.473736986s
192.168.88.38: user: warning: [2023-07-05T16:11:32.188226023Z]: [talos] phase uncordon (20/21): 1 tasks(s)
192.168.88.38: user: warning: [2023-07-05T16:11:32.188326023Z]: [talos] task uncordonNode (1/1): starting
192.168.88.38: user: warning: [2023-07-05T16:11:32.211565023Z]: [talos] retrying error: node not ready
error streaming results: rpc error: code = Unavailable desc = error reading from server: read tcp 192.168.88.14:55478->192.168.88.38:50000: read: operation timed out ```