Buildroot Image crashes at high RAM usage (Possible U-Boot / DTB issue)

#1

I builded https://github.com/flatmax/buildroot.rockchip

I noticed the resulting image always crashes on high IO operations. Like real time NPU, untar.

I narrowed the issue to the RAM, everytime I run “stress --cpu 4 --vm 10” the board instantly crashes and reboots

This is not happening if I first build into a standard Debian image on a SD card. I can boot into Debian, interrupt at U-Boot command line, remove the SD card and restart to load the Buildroot image from eMMC

If I do this the Buildroot image then stops failing RAM Stress Tests and doesn’t crash running NPU programs

So this has to be an issue on U-Boot somewhere. My theory is, U-Boot on Buildroot is not configuring something. When I boot into Debian, it is configuring everything properly, and the thing not being configured by Buildroot, persists configured after reboot

If I remove power and cold boot into Buildroot it keeps crashing again

Any clue? I’ll leave my log

U-Boot SPL board init
U-Boot SPL 2017.09 (Oct 23 2023 - 17:07:39)
unknown raw ID phN
unrecognized JEDEC id bytes: 00, 00, 00
Trying to boot from MMC2
MMC error: The cmd index is 1, ret is -110
Card did not respond to voltage select!
mmc_init: -95, time 10
spl: mmc init failed with error: -95
Trying to boot from MMC1

Verified-boot: 0

Checking atf-1 0x00040000 … sha256(569842593f…) + OK

Checking uboot 0x00a00000 … sha256(92306e40ee…) + OK

Checking fdt 0x00b2def8 … sha256(5eac431258…) + OK

Checking atf-2 0xfdcc9000 … sha256(71fe14fa9d…) + OK

Checking atf-3 0xfdcd0000 … sha256(a5730ca9cc…) + OK

Checking optee 0x08400000 … sha256(b8cddafab0…) + OK

Jumping to U-Boot(0x00a00000) via ARM Trusted Firmware(0x00040000)
Total: 221.47 ms

INFO: Preloader serial: 2
NOTICE: BL31: v2.3():v2.3-135-gcda1658bc-dirty:xsf
NOTICE: BL31: Built : 14:19:26, Jun 1 2021
INFO: GICv3 without legacy support detected.
INFO: ARM GICv3 driver initialized in EL3
INFO: pmu v1 is valid
INFO: dfs DDR fsp_param[0].freq_mhz= 1056MHz
INFO: dfs DDR fsp_param[1].freq_mhz= 324MHz
INFO: dfs DDR fsp_param[2].freq_mhz= 528MHz
INFO: dfs DDR fsp_param[3].freq_mhz= 780MHz
INFO: Using opteed sec cpu_context!
INFO: boot cpu mask: 0
INFO: BL31: Initializing runtime services
INFO: BL31: Initializing BL32
I/TC:
I/TC: Start rockchip platform init
I/TC: Rockchip release version: 1.0
I/TC: OP-TEE version: 3.6.0-307-g0b06ae94 #1 Fri May 7 01:52:27 UTC 2021 aarch6 4
I/TC: Initialized
INFO: BL31: Preparing for EL3 exit to normal world
INFO: Entry point address = 0xa00000
INFO: SPSR = 0x3c9

U-Boot 2017.09 (Oct 23 2023 - 17:07:39 -0400)

Model: Radxa CM3 IO
PreSerial: 2, raw, 0xfe660000
DRAM: 2 GiB
Sysmem: init
Relocation Offset: 7d34a000
Relocation fdt: 7b9f7bc0 - 7b9fecc8
CR: M/C/I
Using default environment

failed to find reset-gpios property
dwmmc@fe2b0000: 1, dwmmc@fe2c0000: 2, sdhci@fe310000: 0
Bootdev(atags): mmc 0
MMC0: HS200, 200Mhz
PartType: EFI
No misc partition
boot mode: None
FIT: No fdt blob
No resource partition
No resource partition
Failed to load DTB, ret=-19
Failed to get kernel dtb, ret=-19
I2c0 speed: 100000Hz
vsel-gpios- not found! Error: -2
vdd_cpu init 900000 uV
PMIC: RK8170 (on=0x10, off=0x00)
vdd_logic init 900000 uV
vdd_gpu init 900000 uV
io-domain: OK
Model: Radxa CM3 IO
rockchip_set_serialno: could not find efuse/otp device
CLK: (sync kernel. arm: enter 816000 KHz, init 816000 KHz, kernel 0N/A)
apll 816000 KHz
dpll 528000 KHz
gpll 1188000 KHz
cpll 1000000 KHz
npll 24000 KHz
vpll 24000 KHz
hpll 24000 KHz
ppll 200000 KHz
armclk 816000 KHz
aclk_bus 150000 KHz
pclk_bus 50000 KHz
aclk_top_high 300000 KHz
aclk_top_low 200000 KHz
hclk_top 150000 KHz
pclk_top 50000 KHz
aclk_perimid 300000 KHz
hclk_perimid 150000 KHz
pclk_pmu 100000 KHz
No misc partition
Net: No ethernet found.
Hit key to stop autoboot(‘CTRL+C’): 0
failed to find reset-gpios property

Device 0: unknown device
MMC error: The cmd index is 1, ret is -110
Card did not respond to voltage select!
mmc_init: -95, time 9
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1…
Found U-Boot script /boot.scr
reading /boot.scr
1113 bytes read in 11 ms (98.6 KiB/s)

Executing script at 00c00000

loading boot vars
reading vars.txt
78 bytes read in 8 ms (8.8 KiB/s)
setting boot args
reading rockchip/rk3566-radxa-cm3-io.dtb
139875 bytes read in 19 ms (7 MiB/s)
reading Image
20879368 bytes read in 149 ms (133.6 MiB/s)
loading overlay rk3568-fiq-debugger-uart2m0
reading rockchip/overlays/rk3568-fiq-debugger-uart2m0.dtbo
513 bytes read in 29 ms (16.6 KiB/s)
Applying kernel provided DT overlay rk3568-fiq-debugger-uart2m0.dtbo
booting linux …
Fdt Ramdisk skip relocation
No misc partition

Flattened Device Tree blob at 0x0a100000

Booting using the fdt blob at 0x0a100000
reserving fdt memory region: addr=a100000 size=88000
‘reserved-memory’ ramoops@110000: addr=110000 size=f0000
Using Device Tree in place at 000000000a100000, end 000000000a18afff
Adding bank: 0x00200000 - 0x08400000 (size: 0x08200000)
Adding bank: 0x09400000 - 0x80000000 (size: 0x76c00000)
Total: 1447.4 ms

Buildroot for the rock 3 a
#2

I am now 95% confident this is all because U-Boot is not finding it’s DTB file

MMC0: HS200, 200Mhz
PartType: EFI
No misc partition
boot mode: None
FIT: No fdt blob
No resource partition
No resource partition
Failed to load DTB, ret=-19
Failed to get kernel dtb, ret=-19

Any clue on how to fix this?

#3

After lots of experimenting I have found the following

I can make U-Boot load a .dtb file just like Debian enabling EARLY_DISTRO_DTB on config. However:

If I keep RK817 on the DTS I get this error at startup and crash

DTB(Distro): /rk-kernel.dtb
Error binding driver ‘fan53555_regulator’: -22
Some drivers failed to bind
Error binding driver ‘i2c_rockchip’: -22
Error binding driver ‘fixed regulator’: -22
Some drivers failed to bind
of_get_regulator: Get (pmuio1-supply) regulator: /i2c@fdd40000/pmic@20/regulators/LDO_REG6 failed, ret=-22
of_get_regulator: Get (pmuio2-supply) regulator: /i2c@fdd40000/pmic@20/regulators/LDO_REG6 failed, ret=-22
of_get_regulator: Get (vccio1-supply) regulator: /i2c@fdd40000/pmic@20/regulators/LDO_REG4 failed, ret=-22
of_get_regulator: Get (vccio2-supply) regulator: /i2c@fdd40000/pmic@20/regulators/LDO_REG7 failed, ret=-22
of_get_regulator: Get (vccio3-supply) regulator: /i2c@fdd40000/pmic@20/regulators/LDO_REG5 failed, ret=-22
of_get_regulator: Get (vccio4-supply) regulator: /i2c@fdd40000/pmic@20/regulators/LDO_REG1 failed, ret=-22
of_get_regulator: Get (vccio5-supply) regulator: /i2c@fdd40000/pmic@20/regulators/DCDC_REG4 failed, ret=-22
of_get_regulator: Get (vccio6-supply) regulator: /i2c@fdd40000/pmic@20/regulators/DCDC_REG4 failed, ret=-22
of_get_regulator: Get (vccio7-supply) regulator: /i2c@fdd40000/pmic@20/regulators/DCDC_REG4 failed, ret=-22
io-domain: OK
Model: Pine64 RK3566 Quartz64-A Board
“Synchronous Abort” handler, esr 0x8a000000

  • Reason: Exception from an Illegal execution state, or a PC or SP alignment fault
  • PC = ffffffff82e36102
  • LR = 0000000000a6366c
  • SP = 000000007b9f7860
  • ESR_EL2 = 000000008a000000
  • Reloc Off = 000000007d34a000

This same DTB works perfectly fine on Debian

If I remove RK817 from the DTB used on U-Boot EARLY_DISTRO_DTB. Then it boots normally. But it keeps crashing at high IO operations. In both Buildroot and Debian

I think I am at a dead end, the only way out seems to be trying another U-Boot version

What could this be possibly about? I dumped the entire I2C map from both Buildroot (crashing at high IO) and Debian (working fine) and in both dumps RK817 seems to be configured exactly the same. I am starting to feel it’s misleading me and the error is something else, but I can’t know

#4

Buildroot’s DTB is missing this entry related to the Dynamic Memory Controller

Could this be the issue? I have not been able to add it in

dmc_fsp: dmc-fsp {
compatible = “rockchip,rk3568-dmc-fsp”;

debug_print_level = <0>;
ddr3_params = <&ddr3_params>;
ddr4_params = <&ddr4_params>;
lpddr3_params = <&lpddr3_params>;
lpddr4_params = <&lpddr4_params>;
lpddr4x_params = <&lpddr4x_params>;

status = “okay”;
};

Using dmesg | grep dmc also complains about this entry being disabled

dmesg | grep dmc

[ 0.322421] rockchip-drm display-subsystem: dmc is disabled

#5

I managed to add the dmc_fsp entry to the dts but it didn’t fix anything

I feel like I am running in circles. I am considering trying another U-Boot version or Kernel version. Can anyone give me any suggestion as to what to do?

#6

For anyone who may be reading I fixed the issue on a hacky and kind of messy way:

I used U-Boot from rockchip repo : https://github.com/rockchip-linux/u-boot ( VERSION : next-dev)
I copied over /config/radxa-cm3-io-rk3566_defconfig from the old U-Boot folder, and copied it renamed to: /config/rk3566_defconfig on the new U-Boot folder
I enabled
CONFIG_ROCKCHIP_EARLY_DISTRO_DTB=y CONFIG_ROCKCHIP_EARLY_DISTRO_DTB_PATH=“rk-kernel.dtb”

( DO THIS ON rk3566_defconfig )

On the command line from Buildroot’s root folder I did: make uboot-reconfigure and then make

I flashed the new image to my SD card

I placed Debian’s DTB file on the root of the Boot partition as “rk-kernel.dtb” (I know this is hacky but whatever, it works)

Buildroot distro still uses its own DTB located on /rockchip/rk3566-radxa-cm3-io.dtb

I deleted boot.scr as it didn’t work, and did the following:

I created extlinux directory on the Boot partition, and I inside created extlinux.conf with the following content:

label rockchip
kernel /Image
fdt /rockchip/rk3566-radxa-cm3-io.dtb
append root=/dev/mmcblk1p2 earlyprintk console=ttyS2,1500000n8 rw rootwait

And I now have fully functional Buildroot image without crash issues
I kind of feel worried because I didn’t really fix the real root issue. I still wonder what’s the real reason behind this
But anyway, since it works I’ll leave it at that

`