Load average constantly above 1

#1

I’m using radxa-zero-ubuntu-focal-server-arm64-20220701-0329-mbr.img.xz from the radxa-build/radxa-zero repository.

With the system idle, the load average always seems to be above 1:

rock@radxa-zero:~$ uname -a
Linux radxa-zero 5.10.69-12-amlogic-g98700611d064 #amlogic SMP PREEMPT Tue Jun 14 12:47:53 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
rock@radxa-zero:~$ uptime
 09:20:00 up 1 day,  2:00,  1 user,  load average: 1.18, 1.17, 1.11

Temperature:

rock@radxa-zero:~$ sudo cat /sys/class/thermal/thermal_zone0/temp
36400

The system is connected to a WIFI network. I’m not using HDMI output or any USB peripherals – it’s powered by USB and that’s it. I have not made any changes to /boot/uEnv.txt. I have installed libmraa.

When I check with htop, I can see one core always at 20-30% utilization, but no processes with significant CPU usage (at the top is htop itself with ~2%).

Any tips on how to investigate what’s causing the above-1 load average? Thanks in advance!

#2

Look for processes in D state. Average load on Linux is not CPU utilization.

#3

There is two processes that are from time to time in “disk sleep” state:

rock@radxa-zero:~$ grep "^State" /proc/*/status | grep -v sleeping | grep -v idle
/proc/3771/status:State:	D (disk sleep)
/proc/4704/status:State:	R (running)
/proc/self/status:State:	R (running)
/proc/thread-self/status:State:	R (running)

rock@radxa-zero:~$ ps aux | grep 3771
root        3771  9.5  0.0      0     0 ?        D    02:26  45:03 [kworker/0:2+events]
rock        4747  0.0  0.1   4728   644 pts/0    S+   10:18   0:00 grep --color=auto 3771
rock@radxa-zero:~$ ps aux | grep 4097
root        4097  9.5  0.0      0     0 ?        I    05:18  28:34 [kworker/0:1-events]
rock        4749  0.0  0.1   4728   676 pts/0    S+   10:19   0:00 grep --color=auto 4097
#4

It’s the kernel doing this and that. If you’re running off a slow SD card then %iowait could be the culprit.

Check iostat 10 for example (needs package sysstat).

#5

Thanks for the tips, appreciate it! iostat output:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.76    0.00    6.43    0.03    0.00   92.78

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
mmcblk1           0.20         0.00         1.60         0.00          0         16          0


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.84    0.00    6.55    0.00    0.00   92.62

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
mmcblk1           0.00         0.00         0.00         0.00          0          0          0
#6

If it isn’t %iowait I wouldn’t be that much concerned. Other platforms suffer from this symptom also and nobody is going to fix anything…

#7

Thanks. It’s not burning CPU, and the performance is fine, so yeah, I’ll just leave it at that.

PS I experimented with removing a bunch of kernel modules (video, bluetooth, …), but no change. One extra thing I suppose I could still try is to disable WIFI and then check the load average via serial console.

#8

I think I found the cause: this commit adds support to a usb-c controller connected via i2c to the SoC.
In top I kept seeing a load of at least 1 with nothing running, and the process using the most cpu (3%) was kworker/0:3+events, and it had status D. Following advice from here I ran multiple times

[root@alarm ~]# cat /proc/293/stack
[<0>] meson_i2c_xfer_msg+0xf8/0x284
[<0>] meson_i2c_xfer+0x60/0xa0
[<0>] __i2c_transfer+0x90/0x1b0
[<0>] i2c_smbus_xfer_emulated+0x10c/0x740
[<0>] __i2c_smbus_xfer+0x8c/0x1f0
[<0>] i2c_smbus_xfer+0x80/0x120
[<0>] i2c_smbus_read_byte_data+0x3c/0x70
[<0>] fusb302_irq_work+0x6c/0x4b0
[<0>] process_one_work+0x1a4/0x300
[<0>] worker_thread+0x6c/0x40c
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork+0x10/0x20

and most of the time that was the output I got.
I cloned the kernel repo, edited the dts and removed lines (referring to the diff of the commit) 71-78 and changed line 347 to status = "disabled"; (found by sheer luck here), compiled only the dtbs (with make radxa_zero_defconfig and then make dtbs) and rebooted.
Load is now 0:

[root@alarm ~]# uptime
 00:49:54 up  1:27,  1 user,  load average: 0.00, 0.00, 0.00

Now, clearly this is not a solution, this disables …something… that is on the board and is needed for usb-c features.
These results are on a Radxa Zero v1.5 board with Arch Linux ARM and kernel

Linux alarm 5.18.1-1-aarch64-ARCH #1 SMP PREEMPT Wed Jun 1 19:34:45 MDT 2022 aarch64 GNU/Linux

I do not know what can be done at this point, mine was just an experiment. I noticed in linux upstream docs for fusb302 bindings that it is required to have a connector sub-node, and it is missing currently in raxda device tree.

1 Like
#9

@RadxaYuntian something for your TODO list :wink:

#10

As a workaround, changing line 347 to status = "disabled"; of the current dts file is enough. I’m sure this can be done a lot quicker by decompiling, searching for the block fusb302@22, changing the status value and recompiling the right dtb file in /boot.

#11

This is added to our investigation list.

#12

I think @RadxaNaoki has found the root cause of this issue. I’ll do a test build and see if FUSB302 still works.

Update: tested OK with PD USB hub. Fix is now merged in our kernel repo: https://github.com/radxa/kernel/commit/104342c599524fb16a28049ecf38cbdb41f3df55

1 Like
#13

I just updated the kernel (apt update && apt upgrade). The old kernel was:

Linux radxa-zero 5.10.69-12-amlogic-g98700611d064 #amlogic SMP PREEMPT Tue Jun 14 12:47:53 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

The new kernel is:

Linux radxa-zero 5.10.69-13-amlogic-g104342c59952 #amlogic SMP PREEMPT Thu Sep 15 13:06:12 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Happy to report the load average is now below 1, at ~0.11 currently. Thanks to everyone involved!

#14

Nice! Thank you, it works:

Linux alarm 5.18.1-1-aarch64-ARCH #1 SMP PREEMPT Wed Jun 1 19:34:45 MDT 2022 aarch64 GNU/Linux

uptime shows:

 04:44:45 up 57 min,  1 user,  load average: 0.00, 0.00, 0.00

@RadxaYuntian I have to ask: when will this be sent to linux upstream? The dts available in linux at this point is missing quite a few things.
I’m asking because I’ve spent (a bit too much) time thinkering with u-boot (upstream), I’ve managed to add led support to it (via dts for radxa zero rev 1.5). I don’t think u-boot will accept a patch that just changes the radxa zero dts file, since they pull dts from linux upstream (where currently there is no green-led defined, see meson-g12a-radxa-zero.dts). Newer u-boot supports fdtoverlays keyword in extlinux, it can now boot this board with console on uart without a boot.scr file!

#15

sending patches to upstream linux/u-boot will be done eventually. but, as you may know, when it will be merged is up to the maintainers.

by the way, you (and anyone) may be able to send patches to upstream at anytime. thank you very much for your contribution.

1 Like
#16

This is what we are moving to right now. 5B image made with debos-radxa uses extlinux, and rbuild uses extlinux exclusively.
I also checked the efi boot support again recently. Some kernel hooks are now shipped in systemd-boot package, but installing the backport package breaks the current image generation code. Hopefully we can add this with bookworm.