Hi @radxaYuntian ,
Perhaps the whole issue was just NetworkManager, let’s see.
Talking to experts I realize that there is a very substantial criticism against NetworkManager out there. NetworkManager is not just problematic, but an absolute and total disaster.
Basically NetworkManager still after all these years does not work. NM does all kinds of weird things. I have seen it do “worse things” than not work, details below. Users easily become blamed for that the networking on their Linux computers malfunctions gravely while the issue is actualy NetworkManager. And this is now in 2025, not 10 years ago and fixed since.
So the first thing many experts do when installing Linux is to purge NM, and instead use the ifupdown package [1] i.e. configuration files in /etc/network/interfaces.d/ (or perhaps alternatively use systemd-networkd [2] or even just their own ifconfig scripts).
Question to you
@RadxaYuntian I had a complete failure yesterday with reinstalling the Rock5B from scratch and I wonder if the problem could be either or also the rsetup / kernel disaster yesterday, or some other problem, instead of or together with NM.
In other words, NM behaved undescribably catastrophic that I wonder if NM is enough to explain the great brokenness, and for this reason I want to ask you if the rsetup/kernel disaster could have contributed.
It is really important that a device like Rock5B actually works and can be trusted to work and I am quite upset at the last 1.5 months of downtime and absolut mess with the SBC.
Please let me know any ideas or suggestions you have.
I’d try this fix
If I would allocate more time to try something more to make this Rock5B-Radxa-Debian 12 Bookworm deployment work at all, I think the first thing I would try would be not to reinstall from scratch and then stay away from rsetup system upgrade altogether (which would otherwise be logical), but instead:
On this deployment which was a fresh install from image, then rsetup-upgraded, and then with subsequent manual kernel downgrade in /boot/extlinux/extlinux.conf to make the onboard Ethernet controller work, I would permanently disable NetworkManager by systemctl disable NetworkManager.service; systemctl mask NetworkManager.service; systemctl stop NetworkManager.service.
Note, NM must be purged altogether in this fashion because if NM is ever running, it risks compromising /etc/resolv.conf and this way destroying all DNS lookups on the device, which is almost as bad as a complete meltdown of the system. Also, when NM is entrusted with DHCP client function, there will be no other software on the device that updates /etc/resolv.conf, so NM’s damage is permanent. I’ll give an example of the incredible destruction caused by NM this way below.
(Note carefully that NM’s configuration option unmanaged-devices=interface-name:eth0;.. under [keyfile] still permits NM to crush the system permanently via /etc/resolv.conf arbitrarily, anytime.)
After disabling NM, do apt install ifupdown isc-dhcp-client, place configuration files for the network interfaces in /etc/network/interfaces/, reboot, and see if the system has now magically become stable.
Postmortem of the disaster install one month ago
As you know one month ago I reinstalled the Rock5B on NVMe SSD from scratch, based on your new stable Radxa Debian 12 Bookworm image.
The reason to reinstall was because the Radxa Debian 11 Bullseye had simply become too old to support recent software.
The whole Rock5B-Radxa-Debian 12 Bookworm deployment died permanently when I did apt update; apt upgrade.
This was upsetting because it lost me a lot of time. The big clarification was to never do apt upgrade again but intead have your rsetup tool do the same but safely. Fair. [3]
On this installation I had also upgraded the SPI flash to the latest version. [3]
The first thing I did yesterday was to see what happens when booting the Rock5B with the previous apt upgrade:d deployment on it.
To my surprise, it did boot. So uboot or other boot code had not been crushed by the apt upgrade.
When I logged in, I noticed that the onboard Ethernet controller did show in lspci, but did not show up ip a, ifconfig -a and otherwise in dmesg.
This shows that the apt update; apt upgrade installed a broken kernel and made the Rock5B go offline from the network permanently, this is of course a great disaster. But at least the boot code had sustained. (This is in alignment the explanation you provided here On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade .)
As we’ll see below, after the fresh reinstall of the Radxa Debian 12 Bookworm image, rsetup actually installed the same broken kernel and caused the same disaster.
Fresh install disaster yesterday
Yesterday, I devoted the whole work day 8AM to 7PM to reinstall the Rock5B on NVMe SSD again from scratch, and it ended in a disaster.
The first disappointment was that “rsetup” actually crashed the system permanently just like apt upgrade did a month ago.
The last 3-4 hours went to making NetworkManager set DHCP client on the WAN/Internet network interface (that’s the onboard Realtek Ethernet controller) and a fixed IPv4 + optionally IPv6 on the LAN network interface (this Ethernet controller is in the MiniPCI slot), and this was a total disaster.
What happened was NetworkManager worked totally unreliably and ultimately the networking was just totally dead. I’ll describe the details below.
You had clarified (in [3]) that a Radxa Debian deployment where you have ever run “apt upgrade” is to be viewed as compromised, so I proceeded with reinstalling the device from scratch.
I flashed https://github.com/radxa-build/rock-5b/releases/download/rsdk-b5/rock-5b_bookworm_kde_b5.output.img.xz to a MicroSD card using BalenaEtcher, booted the Rock5B with the MicroSD (the RK3588 gives MicroSD boot priority over SPI flash/NVMe SSD), transferred the rock-5b_bookworm_kde_b5.output.img.xz image in to the Rock5B separately via Ethernet and wrote it to the NVMe SSD at /dev/nvme0n1 using xzcat | dd.
Then expanded the EXT4 volume to the disk’s available space with growpart /dev/nvme0n1 3; partx /dev/nvme0n1; resize2fs /dev/nvme0n1p3.
Booting the Rock5B off the NVMe SSD worked fine.
- DHCP client not pre-configured, set it up manually
The first new behavior in this Debian versus previous deployments I saw was that the onboard Realtek 8125 NIC (device name enP4p65s0) was now not pre-configured with DHCP client. I.e. the Rock5B with Internet connected on the built-in Ethernet controller did not have Internet access out-of-the-box.
Having DHCP client working is of course is a prerequisite for doing a system upgrade in rsetup.
Yesterday for confirmation I made a separate thread about this here The Radxa Debian distribution does not configure the onboard Ethernet controller right? and you clarified that NetworkManager’s default behavior is actually to do DHCP client on the built-in Ethernet controller.
At this stage, the Rock5B ran the Rock5B-Radxa-Debian 12 Bookworm image freshly installed, with the bundled 6.1.43-15-rk2312 image. Either this shows us just how badly broken NetworkManager is, or perhaps there’s some bug in the bundled image/kernel?
I see that you mentioned here On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade that you had switched which driver the built-in Ethernet controller uses - could it be that the new driver is broken, or has problems with NetworkManager?
As I write this now, I remember one relevant fact: NetworkManager called the built-in Ethernet “Wired connection 2” and called the other MiniPCI I225 ethernet controller “Wired connection 1”. I would presume NM’s expected behaviour in absence of any configuration files is to do DHCP client on all ethernet controllers though, or? @RadxaYuntian
While this, then, supposedly, showed us a grave bug in NetworkManager, we will see an abundance of much worse bugs in NetworkManager below - at least the DHCP client function worked as soon as I made a configuration file for it:
So I set up NetworkManager to do DHCP client on enP4p65s0 separately, by creating a configuration file in /etc/NetworkManager/system-connections/ (content shown below) and rebooting.
(Also I noted that rsetup's menu option for Connections actually launches NetworkManager’s Text User Interface program nmtui. This is an alternative path to setting up the DHCP client that not involves authoring the configuration file yourself.)
Getting the outward network connectivity including DNS client working this way, worked right away.
- rsetup disaster
The next step now was to upgrade this Rock5B-Radxa-Debian 12 Bookworm deployment to the latest packages and Linux kernel that you prepared.
So I went through system update in rsetup and it did not produce any errors. [4]
After the system upgrade, rebooting the Rock5B did work, but the built-in Ethernet controller was now dead. It showed up in lspci, but not in ip a, ifconfig -a and otherwise in dmesg.
In other words, rsetup caused exactly the same permanent system crash as apt upgrade. This was really disappointing.
- Kernel downgrade fix
I figured that the Ethernet controller stopped working should be a matter of a broken kernel.
So I had a look in /boot to see what could be done to go back to the original kernel from the Debian image which was “6.1.43-15-rk2312”, from before rsetup's upgrade to a new kernel version named “6.1.84-6-rk2410”. And I figured that the old kernel was still in /boot, intact, and that the boot manager’s configuration file /boot/extlinux/extlinux.conf even contained the old kernel option boot configuration, but just that the new kernel had been installed there too and configured to be the default boot option.
The new kernel boot option was named l0 and the new kernel l1. So I just changed the default l0 line to default l1.
For reference, this is the whole content of /boot/extlinux/extlinux.conf post-rsetup:
## /boot/extlinux/extlinux.conf
##
## IMPORTANT WARNING
##
## The configuration of this file is generated automatically.
## Do not edit this file manually, use: u-boot-update
default l0
menu title U-Boot menu
prompt 1
timeout 10
label l0
menu label Debian GNU/Linux 12 (bookworm) 6.1.84-6-rk2410
linux /boot/vmlinuz-6.1.84-6-rk2410
initrd /boot/initrd.img-6.1.84-6-rk2410
fdtdir /usr/lib/linux-image-6.1.84-6-rk2410/
append root=UUID=2e6a37fe-c35a-4fa1-b3e6-932217ec4dcb console=ttyFIQ0,1500000n8 quiet splash loglevel=4 rw earlycon consoleblank=0 console=tty1 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1
label l0r
menu label Debian GNU/Linux 12 (bookworm) 6.1.84-6-rk2410 (rescue target)
linux /boot/vmlinuz-6.1.84-6-rk2410
initrd /boot/initrd.img-6.1.84-6-rk2410
fdtdir /usr/lib/linux-image-6.1.84-6-rk2410/
append root=UUID=2e6a37fe-c35a-4fa1-b3e6-932217ec4dcb console=ttyFIQ0,1500000n8 splash loglevel=4 rw earlycon consoleblank=0 console=tty1 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1 single
label l1
menu label Debian GNU/Linux 12 (bookworm) 6.1.43-15-rk2312
linux /boot/vmlinuz-6.1.43-15-rk2312
initrd /boot/initrd.img-6.1.43-15-rk2312
fdtdir /usr/lib/linux-image-6.1.43-15-rk2312/
append root=UUID=2e6a37fe-c35a-4fa1-b3e6-932217ec4dcb console=ttyFIQ0,1500000n8 quiet splash loglevel=4 rw earlycon consoleblank=0 console=tty1 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1
label l1r
menu label Debian GNU/Linux 12 (bookworm) 6.1.43-15-rk2312 (rescue target)
linux /boot/vmlinuz-6.1.43-15-rk2312
initrd /boot/initrd.img-6.1.43-15-rk2312
fdtdir /usr/lib/linux-image-6.1.43-15-rk2312/
append root=UUID=2e6a37fe-c35a-4fa1-b3e6-932217ec4dcb console=ttyFIQ0,1500000n8 splash loglevel=4 rw earlycon consoleblank=0 console=tty1 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1 single
Simply rebooting, the Rock5B now appeared to work fine: the boot worked fine, and ip a, ifconfig -a and dmesg all showed enP4p65s0.
(I see the note now inside /boot/extlinux/extlinux.conf that the file is autogenerated and I’m supposed to not edit it but run u-boot-update instead. @RadxaYuntian was this a problem, if I want to switch kernel in the future which file should I edit and which command should I run to do it according to best practice?)
So in summary up to this point, I had installed the Rock5B-Radxa-Debian 12 Bookworm image to the NVMe SSD from scratch, set up DHCP client to bring Internet connectivity, upgraded the system via rsetup, and downgraded the kernel to the original bundled kernel, and the Rock5B now booted fine and Internet worked.
Absolute disaster with NetworkManager
I do not have experience with NetworkManager configuration from before. However looking online and also taking learning from the congfiguration files nmtui produces (it stores its produced configuration files into /etc/NetworkManager/system-connections/), it looks clear that configuring NM through placing configuration files in /etc/NetworkManager/system-connections/ is straightforward.
Basically what NM does is, it reads all files in /etc/NetworkManager/system-connections/, and each file found there should accord with its configuration file format.
The first issue I encountered with NM was that the status CLI tool nmcli connection show showed really erratic results almost all the time, through my whole process trying to make the WAN and LAN work in NM.
nmcli connection show shows one line for each Ethernet interface, including one line for lo0.
Each line has a colour, where green should mean it’s online, and I guess grey means disabled and yellow means in activation phase.
Which color the command showed seemed totally random and probably showed that NM is in a totally crashed, buggy state.
Each row has a few columns. The first column is NM’s internal logical name for the interface. The default is “Wired connection 1”, 2, and so on.
The second column is NM’s unique identifier for the interface, in the form of an UUID, not sure how it’s sourced but at least it was unique between the two Ethernet controllers.
And on column four is shown the Ethernet controller’s interface name e.g. enP4p65s0 .
Through my attempts below to make NetworkManager work, nmcli connection show showed the most bizarre states. For example, on my I225 interface, the fourth column became empty (!?) and the line became grey (!?!?!).
Furthermore, the configuration files in /etc/NetworkManager/system-connections/ can specify a name to replace the default “Wired connection 1”/2/3/etc., through the id value under [connection]. It happened that this value was not even read correctly so nmcli connection show still showed “Wired connection 1”.
Next, to see NM’s output on Debian you do journalctl -u NetworkManager -f.
Despite that the configuration file for the I225 interface was fixed IP (for both IPv4 and IPv6), this command showed that NM tried to run DHCP client there.
I also did not see any report of error parsing any configuration file in its output, at any point.
Next, you see the individual configuration for an interface in NM using nmcli -p connection show UUID where UUID is NM’s UUID for that network connection.
This command tended to show absolute chaos, mostly seemed to be default settings as if it hadn’t even read the configuration file in the first place.
The first NM configuration file /etc/NetworkManager/system-connections/OnboardEthernetInterface.nmconnection I produced, which did work then, was approximately this:
[connection]
# NetworkManager logical name
id=OnboardEthernetInterface
# NetworkManager-generated persistent unique identifier (redacted)
uuid=22222222-2222-2222-2222-222222222222
type=ethernet
# This is the Linux network interface name
interface-name=enP4p65s0
autoconnect=true
[ipv4]
method=auto
# NetworkClient's internal DHCP client should retry forever
dhcp-timeout=infinity
may-fail=no
[ipv6]
method=auto
(UUID redacted.)
The intention with dhcp-timeout=infinity and may-fail=no were to keep trying with the DHCP client permanently every 20 seconds or so.
At least when only this configuration file was in /etc/NetworkManager/system-connections/, the machine had connection to the Internet.
The next step was to set up a fixed IP in IPv4 and IPv6 on the LAN.
For this purpose I created /etc/NetworkManager/system-connections/MyLAN.nmconnection:
[connection]
id=MyLAN
uuid=33333333-3333-3333-3333-333333333333
type=ethernet
interface-name=enNNNNNNN
autoconnect=true
[ipv4]
method=manual
host_ip_address=192.168.77.1
netmask=255.255.255.0
gateway=192.168.77.1
dns=192.168.77.1;
[ipv6]
method=manual
addresses=2001:db8::1/64
With inspriation from nmtui in both files,
-
I added a unique “timestamp” value under “[connection]” e.g. 1728568663,
-
I also added an empty “[ethernet]” and “[proxy]” section,
-
removed
autoconnect=true, -
under both “[ipv4]” and “[ipv6]” added
ignore-auto-dns=trueandignore-auto-routes=true, and -
I also tried particular ordering of values.
To OnboardEthernetInterface,
- Tried removing
dhcp-timeout=infinityandmay-fail=no.
To MyLAN,
-
I added under “[connection]”
autoconnect-priority=-999, -
under both “[ipv4]” and “[ipv6]” added
never-default=true(to show this device is never the default route i.e. the Internet connection), changed “method” to “auto”, -
under “[ipv4]” changed host_ip_address to
address1=192.168.77.1/24, and also removednetmask,gatewayanddns, -
under “[ipv6]” set
addr-gen-mode=default, changed “addresses” to “address1”, addedgateway=2001:db8::1, addeddns=2001:db8::1;ordns=2001:4860:4860::8888;.
I also tried to chmod 600 and chown root both files however this was not needed in the first place when getting DHCP client going. Also the files were originally created by root anyhow.
I switched between all the sections above and either rebooted or did systemctl restart NetworkManager and nmcli con up UUID, and my conclusion is that NetworkManager did not function at all.
This total disaster was so bad that I thought, could it be in addition to NM bugs there are also kernel bugs? E.g. NM worked so badly that it’s not even clear that it got to the phase of reading configuration files.
Could it be that NM and its subsystems expected some particular behavior from the kernel or network controller drivers, and something broke somewhere along this chain?
- NM destroys /etc/resolv.conf
One of the brutal side-effects of NetworkManager is that it added the LAN’s IPv6 IP as only DNS server for the whole computer in /etc/resolv.conf. This of course was a total disaster.
When I responded to that with switching the IPv6 configuration on the LAN to auto, NetworkManager created a new /etc/resolv.conf where the only DNS server was now ::1, which was absolutely not intended.
If I run a local DNS server it’d be on 127.0.0.1 but I had not asked NM to do that. Instead /etc/resolv.conf should contain the DNS server that the DHCP client received from the Internet/WAN interface. Nothing of that worked at all, of course, as NM was totally broken.
- dnsmasq OK?
Note, above separately I enabled dnsmasq separately as it works as DHCP server on the LAN.
dnsmasq is preinstalled on Rock5B-Radxa-Debian 12 Bookworm. To /etc/dnsmasq.conf I added interface=enNNNNNNN and dhcp-range=192.168.77.2,192.168.77.50,10h.
Since these DHCP announcements go OUT on the LAN interface, I would presume that they do not interfere with NM’s DHCP client that should work on the WAN interface only.
What about SDK version bump
Last on the topic of making the Rock5B work: I see you mentioned yesterday in On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade that you will “update the RockChip SDK”. What is that about? When will you publish that update?
Will this be a new Debian 12 Bookworm image with a change of URL from https://github.com/radxa-build/rock-5b/releases/download/rsdk-b5/rock-5b_bookworm_kde_b5.output.img.xz to some other name that reflects a new version number?
Summary
As you see above this attempt at installing the Rock5B-Radxa-Debian 12 Bookworm image has ended up in a total and absolute disaster. Perhaps switching to ifupdown might be a magic fix. Hopefully I find the time to try later.
But, the last month I have lost a lot of time - both my own time and SBC uptime - because of this and that’s of course really upsetting.
References:
[1] https://techdocs.akamai.com/cloud-computing/docs/network-configuration-using-ifupdown
[2] https://wiki.debian.org/SystemdNetworkd
https://techdocs.akamai.com/cloud-computing/docs/network-configuration-using-systemd-networkd
[3] Threads:
- Rock5B Radxa Debian 12 image question: OK to install u-boot-rk2410 and remove u-boot-rknext?
- On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade
- Learned that the “SPI image” is flashed (to the SPI flash chip) while the “SPL image” is not flashed but only used when bringing the Rock5B alive via USB in MaskROM mode, as in-RAM boot code. Thread Flash SPI and SPL on Rock5B questions. (Caveat: SPL is never flashed.)
There was also the learning that the Wiki is now obsolete and the Radxa Docs web site is now the authoritative source of information for how to install and configure the SBC.
[4] Some copies of how it looked like here On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade and in the next post after that.