A horrible failure report in latest Radxa Debian - no networking due to kernel, NetworkManager or other component broken. ifupdown a fix?

Hi @radxaYuntian ,

Perhaps the whole issue was just NetworkManager, let’s see.
Talking to experts I realize that there is a very substantial criticism against NetworkManager out there. NetworkManager is not just problematic, but an absolute and total disaster.

Basically NetworkManager still after all these years does not work. NM does all kinds of weird things. I have seen it do “worse things” than not work, details below. Users easily become blamed for that the networking on their Linux computers malfunctions gravely while the issue is actualy NetworkManager. And this is now in 2025, not 10 years ago and fixed since.

So the first thing many experts do when installing Linux is to purge NM, and instead use the ifupdown package [1] i.e. configuration files in /etc/network/interfaces.d/ (or perhaps alternatively use systemd-networkd [2] or even just their own ifconfig scripts).

Question to you
@RadxaYuntian I had a complete failure yesterday with reinstalling the Rock5B from scratch and I wonder if the problem could be either or also the rsetup / kernel disaster yesterday, or some other problem, instead of or together with NM.

In other words, NM behaved undescribably catastrophic that I wonder if NM is enough to explain the great brokenness, and for this reason I want to ask you if the rsetup/kernel disaster could have contributed.

It is really important that a device like Rock5B actually works and can be trusted to work and I am quite upset at the last 1.5 months of downtime and absolut mess with the SBC.

Please let me know any ideas or suggestions you have.

I’d try this fix
If I would allocate more time to try something more to make this Rock5B-Radxa-Debian 12 Bookworm deployment work at all, I think the first thing I would try would be not to reinstall from scratch and then stay away from rsetup system upgrade altogether (which would otherwise be logical), but instead:

On this deployment which was a fresh install from image, then rsetup-upgraded, and then with subsequent manual kernel downgrade in /boot/extlinux/extlinux.conf to make the onboard Ethernet controller work, I would permanently disable NetworkManager by systemctl disable NetworkManager.service; systemctl mask NetworkManager.service; systemctl stop NetworkManager.service.

Note, NM must be purged altogether in this fashion because if NM is ever running, it risks compromising /etc/resolv.conf and this way destroying all DNS lookups on the device, which is almost as bad as a complete meltdown of the system. Also, when NM is entrusted with DHCP client function, there will be no other software on the device that updates /etc/resolv.conf, so NM’s damage is permanent. I’ll give an example of the incredible destruction caused by NM this way below.

(Note carefully that NM’s configuration option unmanaged-devices=interface-name:eth0;.. under [keyfile] still permits NM to crush the system permanently via /etc/resolv.conf arbitrarily, anytime.)

After disabling NM, do apt install ifupdown isc-dhcp-client, place configuration files for the network interfaces in /etc/network/interfaces/, reboot, and see if the system has now magically become stable.

Postmortem of the disaster install one month ago
As you know one month ago I reinstalled the Rock5B on NVMe SSD from scratch, based on your new stable Radxa Debian 12 Bookworm image.

The reason to reinstall was because the Radxa Debian 11 Bullseye had simply become too old to support recent software.

The whole Rock5B-Radxa-Debian 12 Bookworm deployment died permanently when I did apt update; apt upgrade.

This was upsetting because it lost me a lot of time. The big clarification was to never do apt upgrade again but intead have your rsetup tool do the same but safely. Fair. [3]

On this installation I had also upgraded the SPI flash to the latest version. [3]

The first thing I did yesterday was to see what happens when booting the Rock5B with the previous apt upgrade:d deployment on it.

To my surprise, it did boot. So uboot or other boot code had not been crushed by the apt upgrade.

When I logged in, I noticed that the onboard Ethernet controller did show in lspci, but did not show up ip a, ifconfig -a and otherwise in dmesg.

This shows that the apt update; apt upgrade installed a broken kernel and made the Rock5B go offline from the network permanently, this is of course a great disaster. But at least the boot code had sustained. (This is in alignment the explanation you provided here On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade .)

As we’ll see below, after the fresh reinstall of the Radxa Debian 12 Bookworm image, rsetup actually installed the same broken kernel and caused the same disaster.

Fresh install disaster yesterday
Yesterday, I devoted the whole work day 8AM to 7PM to reinstall the Rock5B on NVMe SSD again from scratch, and it ended in a disaster.

The first disappointment was that “rsetup” actually crashed the system permanently just like apt upgrade did a month ago.

The last 3-4 hours went to making NetworkManager set DHCP client on the WAN/Internet network interface (that’s the onboard Realtek Ethernet controller) and a fixed IPv4 + optionally IPv6 on the LAN network interface (this Ethernet controller is in the MiniPCI slot), and this was a total disaster.

What happened was NetworkManager worked totally unreliably and ultimately the networking was just totally dead. I’ll describe the details below.

You had clarified (in [3]) that a Radxa Debian deployment where you have ever run “apt upgrade” is to be viewed as compromised, so I proceeded with reinstalling the device from scratch.

I flashed https://github.com/radxa-build/rock-5b/releases/download/rsdk-b5/rock-5b_bookworm_kde_b5.output.img.xz to a MicroSD card using BalenaEtcher, booted the Rock5B with the MicroSD (the RK3588 gives MicroSD boot priority over SPI flash/NVMe SSD), transferred the rock-5b_bookworm_kde_b5.output.img.xz image in to the Rock5B separately via Ethernet and wrote it to the NVMe SSD at /dev/nvme0n1 using xzcat | dd.

Then expanded the EXT4 volume to the disk’s available space with growpart /dev/nvme0n1 3; partx /dev/nvme0n1; resize2fs /dev/nvme0n1p3.

Booting the Rock5B off the NVMe SSD worked fine.

- DHCP client not pre-configured, set it up manually
The first new behavior in this Debian versus previous deployments I saw was that the onboard Realtek 8125 NIC (device name enP4p65s0) was now not pre-configured with DHCP client. I.e. the Rock5B with Internet connected on the built-in Ethernet controller did not have Internet access out-of-the-box.

Having DHCP client working is of course is a prerequisite for doing a system upgrade in rsetup.

Yesterday for confirmation I made a separate thread about this here The Radxa Debian distribution does not configure the onboard Ethernet controller right? and you clarified that NetworkManager’s default behavior is actually to do DHCP client on the built-in Ethernet controller.

At this stage, the Rock5B ran the Rock5B-Radxa-Debian 12 Bookworm image freshly installed, with the bundled 6.1.43-15-rk2312 image. Either this shows us just how badly broken NetworkManager is, or perhaps there’s some bug in the bundled image/kernel?

I see that you mentioned here On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade that you had switched which driver the built-in Ethernet controller uses - could it be that the new driver is broken, or has problems with NetworkManager?

As I write this now, I remember one relevant fact: NetworkManager called the built-in Ethernet “Wired connection 2” and called the other MiniPCI I225 ethernet controller “Wired connection 1”. I would presume NM’s expected behaviour in absence of any configuration files is to do DHCP client on all ethernet controllers though, or? @RadxaYuntian

While this, then, supposedly, showed us a grave bug in NetworkManager, we will see an abundance of much worse bugs in NetworkManager below - at least the DHCP client function worked as soon as I made a configuration file for it:

So I set up NetworkManager to do DHCP client on enP4p65s0 separately, by creating a configuration file in /etc/NetworkManager/system-connections/ (content shown below) and rebooting.

(Also I noted that rsetup's menu option for Connections actually launches NetworkManager’s Text User Interface program nmtui. This is an alternative path to setting up the DHCP client that not involves authoring the configuration file yourself.)

Getting the outward network connectivity including DNS client working this way, worked right away.

- rsetup disaster
The next step now was to upgrade this Rock5B-Radxa-Debian 12 Bookworm deployment to the latest packages and Linux kernel that you prepared.

So I went through system update in rsetup and it did not produce any errors. [4]

After the system upgrade, rebooting the Rock5B did work, but the built-in Ethernet controller was now dead. It showed up in lspci, but not in ip a, ifconfig -a and otherwise in dmesg.

In other words, rsetup caused exactly the same permanent system crash as apt upgrade. This was really disappointing.

- Kernel downgrade fix
I figured that the Ethernet controller stopped working should be a matter of a broken kernel.

So I had a look in /boot to see what could be done to go back to the original kernel from the Debian image which was “6.1.43-15-rk2312”, from before rsetup's upgrade to a new kernel version named “6.1.84-6-rk2410”. And I figured that the old kernel was still in /boot, intact, and that the boot manager’s configuration file /boot/extlinux/extlinux.conf even contained the old kernel option boot configuration, but just that the new kernel had been installed there too and configured to be the default boot option.

The new kernel boot option was named l0 and the new kernel l1. So I just changed the default l0 line to default l1.

For reference, this is the whole content of /boot/extlinux/extlinux.conf post-rsetup:

## /boot/extlinux/extlinux.conf
##
## IMPORTANT WARNING
##
## The configuration of this file is generated automatically.
## Do not edit this file manually, use: u-boot-update

default l0
menu title U-Boot menu
prompt 1
timeout 10


label l0
        menu label Debian GNU/Linux 12 (bookworm) 6.1.84-6-rk2410
        linux /boot/vmlinuz-6.1.84-6-rk2410
        initrd /boot/initrd.img-6.1.84-6-rk2410
        fdtdir /usr/lib/linux-image-6.1.84-6-rk2410/

        append root=UUID=2e6a37fe-c35a-4fa1-b3e6-932217ec4dcb console=ttyFIQ0,1500000n8 quiet splash loglevel=4 rw earlycon consoleblank=0 console=tty1 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1

label l0r
        menu label Debian GNU/Linux 12 (bookworm) 6.1.84-6-rk2410 (rescue target)
        linux /boot/vmlinuz-6.1.84-6-rk2410
        initrd /boot/initrd.img-6.1.84-6-rk2410
        fdtdir /usr/lib/linux-image-6.1.84-6-rk2410/
        append root=UUID=2e6a37fe-c35a-4fa1-b3e6-932217ec4dcb console=ttyFIQ0,1500000n8 splash loglevel=4 rw earlycon consoleblank=0 console=tty1 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1 single


label l1
        menu label Debian GNU/Linux 12 (bookworm) 6.1.43-15-rk2312
        linux /boot/vmlinuz-6.1.43-15-rk2312
        initrd /boot/initrd.img-6.1.43-15-rk2312
        fdtdir /usr/lib/linux-image-6.1.43-15-rk2312/

        append root=UUID=2e6a37fe-c35a-4fa1-b3e6-932217ec4dcb console=ttyFIQ0,1500000n8 quiet splash loglevel=4 rw earlycon consoleblank=0 console=tty1 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1

label l1r
        menu label Debian GNU/Linux 12 (bookworm) 6.1.43-15-rk2312 (rescue target)
        linux /boot/vmlinuz-6.1.43-15-rk2312
        initrd /boot/initrd.img-6.1.43-15-rk2312
        fdtdir /usr/lib/linux-image-6.1.43-15-rk2312/
        append root=UUID=2e6a37fe-c35a-4fa1-b3e6-932217ec4dcb console=ttyFIQ0,1500000n8 splash loglevel=4 rw earlycon consoleblank=0 console=tty1 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1 single

Simply rebooting, the Rock5B now appeared to work fine: the boot worked fine, and ip a, ifconfig -a and dmesg all showed enP4p65s0.

(I see the note now inside /boot/extlinux/extlinux.conf that the file is autogenerated and I’m supposed to not edit it but run u-boot-update instead. @RadxaYuntian was this a problem, if I want to switch kernel in the future which file should I edit and which command should I run to do it according to best practice?)

So in summary up to this point, I had installed the Rock5B-Radxa-Debian 12 Bookworm image to the NVMe SSD from scratch, set up DHCP client to bring Internet connectivity, upgraded the system via rsetup, and downgraded the kernel to the original bundled kernel, and the Rock5B now booted fine and Internet worked.

Absolute disaster with NetworkManager
I do not have experience with NetworkManager configuration from before. However looking online and also taking learning from the congfiguration files nmtui produces (it stores its produced configuration files into /etc/NetworkManager/system-connections/), it looks clear that configuring NM through placing configuration files in /etc/NetworkManager/system-connections/ is straightforward.

Basically what NM does is, it reads all files in /etc/NetworkManager/system-connections/, and each file found there should accord with its configuration file format.

The first issue I encountered with NM was that the status CLI tool nmcli connection show showed really erratic results almost all the time, through my whole process trying to make the WAN and LAN work in NM.

nmcli connection show shows one line for each Ethernet interface, including one line for lo0.

Each line has a colour, where green should mean it’s online, and I guess grey means disabled and yellow means in activation phase.

Which color the command showed seemed totally random and probably showed that NM is in a totally crashed, buggy state.

Each row has a few columns. The first column is NM’s internal logical name for the interface. The default is “Wired connection 1”, 2, and so on.

The second column is NM’s unique identifier for the interface, in the form of an UUID, not sure how it’s sourced but at least it was unique between the two Ethernet controllers.

And on column four is shown the Ethernet controller’s interface name e.g. enP4p65s0 .

Through my attempts below to make NetworkManager work, nmcli connection show showed the most bizarre states. For example, on my I225 interface, the fourth column became empty (!?) and the line became grey (!?!?!).

Furthermore, the configuration files in /etc/NetworkManager/system-connections/ can specify a name to replace the default “Wired connection 1”/2/3/etc., through the id value under [connection]. It happened that this value was not even read correctly so nmcli connection show still showed “Wired connection 1”.

Next, to see NM’s output on Debian you do journalctl -u NetworkManager -f.

Despite that the configuration file for the I225 interface was fixed IP (for both IPv4 and IPv6), this command showed that NM tried to run DHCP client there.

I also did not see any report of error parsing any configuration file in its output, at any point.

Next, you see the individual configuration for an interface in NM using nmcli -p connection show UUID where UUID is NM’s UUID for that network connection.

This command tended to show absolute chaos, mostly seemed to be default settings as if it hadn’t even read the configuration file in the first place.


The first NM configuration file /etc/NetworkManager/system-connections/OnboardEthernetInterface.nmconnection I produced, which did work then, was approximately this:

[connection]
# NetworkManager logical name
id=OnboardEthernetInterface
# NetworkManager-generated persistent unique identifier (redacted)
uuid=22222222-2222-2222-2222-222222222222
type=ethernet
# This is the Linux network interface name
interface-name=enP4p65s0
autoconnect=true

[ipv4]
method=auto
# NetworkClient's internal DHCP client should retry forever
dhcp-timeout=infinity
may-fail=no

[ipv6]
method=auto

(UUID redacted.)

The intention with dhcp-timeout=infinity and may-fail=no were to keep trying with the DHCP client permanently every 20 seconds or so.

At least when only this configuration file was in /etc/NetworkManager/system-connections/, the machine had connection to the Internet.

The next step was to set up a fixed IP in IPv4 and IPv6 on the LAN.

For this purpose I created /etc/NetworkManager/system-connections/MyLAN.nmconnection:

[connection]
id=MyLAN
uuid=33333333-3333-3333-3333-333333333333
type=ethernet
interface-name=enNNNNNNN
autoconnect=true

[ipv4]
method=manual
host_ip_address=192.168.77.1
netmask=255.255.255.0
gateway=192.168.77.1
dns=192.168.77.1;

[ipv6]
method=manual
addresses=2001:db8::1/64

With inspriation from nmtui in both files,

  • I added a unique “timestamp” value under “[connection]” e.g. 1728568663,

  • I also added an empty “[ethernet]” and “[proxy]” section,

  • removed autoconnect=true,

  • under both “[ipv4]” and “[ipv6]” added ignore-auto-dns=true and ignore-auto-routes=true, and

  • I also tried particular ordering of values.

To OnboardEthernetInterface,

  • Tried removing dhcp-timeout=infinity and may-fail=no.

To MyLAN,

  • I added under “[connection]” autoconnect-priority=-999,

  • under both “[ipv4]” and “[ipv6]” added never-default=true (to show this device is never the default route i.e. the Internet connection), changed “method” to “auto”,

  • under “[ipv4]” changed host_ip_address to address1=192.168.77.1/24, and also removed netmask, gateway and dns,

  • under “[ipv6]” set addr-gen-mode=default, changed “addresses” to “address1”, added gateway=2001:db8::1, added dns=2001:db8::1;or dns=2001:4860:4860::8888;.

I also tried to chmod 600 and chown root both files however this was not needed in the first place when getting DHCP client going. Also the files were originally created by root anyhow.

I switched between all the sections above and either rebooted or did systemctl restart NetworkManager and nmcli con up UUID, and my conclusion is that NetworkManager did not function at all.

This total disaster was so bad that I thought, could it be in addition to NM bugs there are also kernel bugs? E.g. NM worked so badly that it’s not even clear that it got to the phase of reading configuration files.

Could it be that NM and its subsystems expected some particular behavior from the kernel or network controller drivers, and something broke somewhere along this chain?


- NM destroys /etc/resolv.conf
One of the brutal side-effects of NetworkManager is that it added the LAN’s IPv6 IP as only DNS server for the whole computer in /etc/resolv.conf. This of course was a total disaster.

When I responded to that with switching the IPv6 configuration on the LAN to auto, NetworkManager created a new /etc/resolv.conf where the only DNS server was now ::1, which was absolutely not intended.

If I run a local DNS server it’d be on 127.0.0.1 but I had not asked NM to do that. Instead /etc/resolv.conf should contain the DNS server that the DHCP client received from the Internet/WAN interface. Nothing of that worked at all, of course, as NM was totally broken.

- dnsmasq OK?
Note, above separately I enabled dnsmasq separately as it works as DHCP server on the LAN.

dnsmasq is preinstalled on Rock5B-Radxa-Debian 12 Bookworm. To /etc/dnsmasq.conf I added interface=enNNNNNNN and dhcp-range=192.168.77.2,192.168.77.50,10h.

Since these DHCP announcements go OUT on the LAN interface, I would presume that they do not interfere with NM’s DHCP client that should work on the WAN interface only.

What about SDK version bump
Last on the topic of making the Rock5B work: I see you mentioned yesterday in On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade that you will “update the RockChip SDK”. What is that about? When will you publish that update?

Will this be a new Debian 12 Bookworm image with a change of URL from https://github.com/radxa-build/rock-5b/releases/download/rsdk-b5/rock-5b_bookworm_kde_b5.output.img.xz to some other name that reflects a new version number?

Summary
As you see above this attempt at installing the Rock5B-Radxa-Debian 12 Bookworm image has ended up in a total and absolute disaster. Perhaps switching to ifupdown might be a magic fix. Hopefully I find the time to try later.

But, the last month I have lost a lot of time - both my own time and SBC uptime - because of this and that’s of course really upsetting.

References:

[1] https://techdocs.akamai.com/cloud-computing/docs/network-configuration-using-ifupdown

[2] https://wiki.debian.org/SystemdNetworkd
https://techdocs.akamai.com/cloud-computing/docs/network-configuration-using-systemd-networkd

[3] Threads:

  1. Rock5B Radxa Debian 12 image question: OK to install u-boot-rk2410 and remove u-boot-rknext?
  2. On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade
  3. Learned that the “SPI image” is flashed (to the SPI flash chip) while the “SPL image” is not flashed but only used when bringing the Rock5B alive via USB in MaskROM mode, as in-RAM boot code. Thread Flash SPI and SPL on Rock5B questions. (Caveat: SPL is never flashed.)

There was also the learning that the Wiki is now obsolete and the Radxa Docs web site is now the authoritative source of information for how to install and configure the SBC.

[4] Some copies of how it looked like here On Rock5B with Radxa's Debian image, always use rsetup to upgrade the system, never apt upgrade and in the next post after that.

I’m sorry for your experience. You might have better luck with Armbian.

Thanks for suggesting I can try Armbian. To your awareness generally does Armbian track the Rock5B/RK3588 specific patches well, and work fine with the ROCK5B so things like the R8125 onboard ethernet, NVMe SSD, USB3, MiniPCIe slot etc. are stable?

On https://www.armbian.com/rock-5b/ they seem to say Armbian doesn’t support proper USBC PD negotiation. When I look at a USBC power meter connected to the Rock5B it always says 20V, means your Rock5B-Radxa-Debian does support it well however?

(Even if using Armbian, I guess flashing the SPI flash to your latest u-boot distribution version should be done via your image and instructions ref. https://docs.radxa.com/en/rock5/rock5b/getting-started/install-os .)

I’ll give another try with the Rock5B soon. Possibly the local setup like the cable or the DHCP server the Rock5B was connected could have trigged some bug in the Rock5B’s NetworkManager. But I should try ifupdown.

Hope your onboard NIC driver situation will become stable soon.

When do you believe you release the Rock5B Bookworm image with updated RSDK that you mentioned few days ago?

What is RSDK in the first place

Those should be fine. Their system does not have nearly as much stuff in an apt repo like what we do. On one hand you can’t do online update to fix a issue, on the other hand you don’t have issue caused by online update.

We support PD negotiation but the negotiation process is kinda out of spec due to hardware design. This caused the device to reset on certain power supplies. But those work works.

Armbian has its own SPI bootloader and installation method.

OK. Thanks for letting me know.

Their system does not have nearly as much stuff in an apt repo like what we do.

Do you mean they did not fork Debian’s packages repository so some programs like LibreOffice or screen or whatever could be missing, or do you mean lower-level SBC/SoC-specific tooling like GPIO could be missing?

Do you know why the onboard Ethernet disappeared in “6.1.84-6-rk2410”, does it have to do with that you changed driver for it?

Back in Rock5B-Radxa-Debian 11 Bullseye with kernel 5., the ethernet was never an issue.

Now in Debian 12 Bookworm with kernel 6., why did you change driver, did you see issues with the other one?

Are you planning to release another kernel in “rsetup” soon that should fix the onboard ethernet

In a way I’d prefer to use a Radxa or Rockchip made Debian distribution, with the assessment that you should track kernel patches relating to for instance device driver specifics and CPU optimization the best.

Anyhow I’ll test the Rock5B a bit later and maybe switch to ifupdown. It could be helpful if ifupdown and isc-dhcp-client is pre-installed in your Debian image for anyone who is cut off from package downloads. At the very least they serve as instrumentation for testing. Right now I can’t even test if it’s the kernel and or NM that’s the issue, because apt install isn’t working because NM is so broken it didn’t even do DHCP client.

About your own custom code for the PD support, what about you send a note to Armbian on how to get it going with the patches you made already. They should have a minimum level of interest to include working patches that make their distribution better.

We are both based on Debian, so packages provided by Debian (LibreOffice or screen) are always there, but we also ship kernel bootloader and system configs in apt repo, which has little Armbian equivalent.

Likely.

I don’t remember the details but yes we did have issue with it.

Next will be a full system image release. Can’t be updated in-place with rsetup.

PD is enabled by default. It takes extra effort to disable it and lose DP alt mode, in return you get better power supply compatibility. It is an intentional decision to be that way.

Ah so you will not release any more kernel updates in rsetup on the current Rock5B Radxa Debian 12 Bookworm image, ok.

About what date do you expect to release it e.g. next week or in August?

Oh that’s a bummer, OK. Realtek ethernet controllers should normally be very stable.

Internally now did you see any problems with the new driver? In your own testing with your “6.1.84-6-rk2410” does the Realtek NIC work? So weird it’s gone on my computer.

With “new Radxa SDK”, is this something you make inhouse, or do you mean some RK3588 microcode or device drivers or what?

I think you mean that Armbian prefer to only supply power on the USBC, while you like power + USBC + Displayport to all be available on the port. I think your approach makes more sense.

A good piece of news:

When connected to a higher quality network, the Rock5B with NetworkManager actually did connect successfully via DHCP client.

I’ll just leave the system like this and not do any more rsetup update.