Linux Kernel giving panic

Hi there,

I’m booting the kernel from uboot after loading the rk3568-rock-3-a.dtb like so :
booti ${kernel_addr_r} - ${fdt_addr_r}

Buildroot us using the following kernel repo and version :
BR2_LINUX_KERNEL_CUSTOM_REPO_URL=“https://github.com/radxa/kernel.git
BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION=“a4e78ea509f7c32206d84301e674b9718583ace1”

I get the following kernel panic :
=> booti ${kernel_addr_r} - ${fdt_addr_r}
Fdt Ramdisk skip relocation
## Flattened Device Tree blob at 0x0a100000
Booting using the fdt blob at 0x0a100000
reserving fdt memory region: addr=a100000 size=8a000
‘reserved-memory’ ramoops@110000: addr=110000 size=f0000
Using Device Tree in place at 000000000a100000, end 000000000a18cfff
Adding bank: 0x00200000 - 0xf0000000 (size: 0xefe00000)
Adding bank: 0x1f0000000 - 0x200000000 (size: 0x10000000)
Total: 29473.539 ms

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[    0.000000] Linux version 4.19.193 (flatmax@monstilationax) (gcc version 10.3.0 (Buildroot 2021.11-rc2-22-ge49fa6631b), GNU ld (GNU Binutils) 2.36.1) #1 SMP Mon Dec 6 13:27:07 AEDT 2021
[    0.000000] Machine model: Radxa ROCK 3 Model A
[    0.000000] OF: fdt: Reserved memory: failed to reserve memory for node 'drm-logo@00000000': base 0x000000000000000] percpu: Embedded 23 pages/cpu s55272 r8192 d30744 u94208
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: Virtualization Host Extensions
[    0.000000] CPU features: detected: Speculative Store Bypassing Safe (SSBS)
[    0.000000] Built 1 zonelists, mobility grouping on0.  Total pages: 1031688
[    0.000000] Kernel command line: root=/dev/mmcblk0p2 earlyprintk console=ttyS2,115200n8 rw rootwait
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0xeafff000-0xeefff000] (64MB)
[    00.000000] Memory: 3866052K/4192256K available (12414K kernel code, 1812K rwdata, 4460K rodata, 1536K init, 1791K bss, 178748K reserved, 147456K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 45575 entries in 179 pages
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: no VLPI support, no direct LPI support
[    0.000000] ITS [mem 0xfd440000-0xfd45ffff]
[    0.000000] ITS@0x00000000fd440000: allocated 8192 Devices @210000 (indirect, esz 8, psz 64K, shr 0)
[    0.000000] ITS@0x00000000fd440000: allocated 32768 Interrupt Collections @220000 (flat, esz 2, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GIC: using LPI property table @0x0000000000230000
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000fd460000
[    0.000000] CPU0: using LPI pending table @0x0000000000240000
0
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] random: random: get_random_bytes called from start_kernel+0x358/0x4e8 with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 24.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x588fe9dc0, max_idle_ns: 440795202592 ns
[    0.000006] sched_clock: 56 bits at 24MHz, resolution 41ns, wraps every 43980460511097ns
[    0.001703] Console: colour dummy device 80x25
[    0.001762] Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=80000)
[    0.001778] pid_max: default: 32768 minimum: 301
[    0.001987] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
[    0.002013] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
[    0.004115] ASID allocator initialised with 32768 entries
[    0.004300] rcu: Hierarchical SRCU implementation.
[    0.006598] Platform MSI: interrupt-controller@fd440000 domain created
[    0.007184] PCI/MSI: /interrupt-controller@fd400000/interrupt-controller@fd440000 domain created
[    0.008127] smp: Bringing up secondary CPUs ...
[    0.008975] Detected VIPT I-cache on CPU1
[    0.009016] GICv3: CPU1: found redistributor 100 region 0:0x00000000fd480000
[    0.009077] CPU1: using LPI pending table @0x0000000000250000
[    0.009137] CPU1: Booted secondary processor 0x0000000100 [0x412fd050]
[    0.010180] Detected VIPT I-cache on CPU2
[    0.010213] GICv3: CPU2: found redistributor 200 region 0:0x00000000fd4a0000
[    0.010264] CPU2: using LPI pending table @0x0000000000260000
[    0.010309] CPU2: Booted secondary processor 0x0000000200 [0x412fd050]
[    0.011211] Detected VIPT I-cache on CPU3
[    0.011243] GICv3:0 CPU3: found redistributor 300 region 0:0x00000000fd4c0000
[    0.011294] CPU3: using LPI pending table @0x0000000000270000
[    0.011337] CPU3: Booted secondary processor 0x0000000300 [0x412fd050]
[    0.011472] smp: Brought up 1 node, 4 CPUs
[    0.011526] SMP: Total of 4 processors activated.
[    0.011537] CPU features: detected: GIC system register CPU interface
[    0.011545] CPU features: detected: Privileged Access Never
[    000.011552] CPU features: detected: LSE atomic instructions
[    0.011560] CPU features: detected: User Access Override
[    0.011570] CPU features: detected: 32-bit EL0 Support
[    0.011577] CPU features: detected: RAS Extension Support
[    0.011754] CPU: All CPU(s) started at EL2
[    0.011789] alternatives: patching kernel code
[    0.019437] devtmpfs: initialized
[    0.052481] Registered cp15_barrier emulation handler
[    0.052507] Registered setend emulation handler
[    0.052893] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6370867519511994 ns
[    0.052918] futex hash table entries: 1024 (order: 4, 65536 bytes)
[    0.053810] pinctrl core: initialized pinctrl subsystem
[    0.055125] NET: Registered protocol family 16
[    0.059976] cpuidle: using governor menu
[    0.060027] Registered FIQ tty driver
[    0.060456] hw-breakp,oint: found 6 breakpoint and 4 watchpoint registers.
[    0.061875] DMA: preallocated 256 KiB pool for atomic allocations
[    0.066174] console [pstore-1] enabled
[    0.066186] pstore: Registered ramoops as persistent store backend
[    0.066198] ramoops: attached 0xf0000@0x110000, ecc: 0/0
[    0.074021] platform fde40000.npu: assigned reserved memory node rknpu
[    0.113120] rockchip-gpio fdd60000.gpio: probed gpio0 (fdd60000.gpio )
[    0.113905] rockchip-gpio fe740000.gpio: probed gpio1 (fe740000.gpio)
[    0.114648] rockchip-gpio fe750000.gpio: probed gpio2 (fe750000.gpio)
[    0.115447] rockchip-gpio fe760000.gpio: probed gpio3 (fe760000.gpio)
[    0.116209] rockchip-gpio fe770000.gpio: probed gpio4 (fe770000.gpio)
[    0.116412] rockchip-pinctrl pinctrl: probed pinctrl
[    0.131478] cryptd: max_cpu_qlen set to 1000
[    0.134341] console [ttyFIQ0] enabled
s[    0.134673] Registered fiq debugger ttyFIQ0
[    0.135984] vcc3v3_sys: supplied by dc_12v
[    0.136508] vcc5v0_sys: supplied by dc_12v
[    0.138793] pcie30_avdd0v9: supplied by vcc3v3_sys
[    0.139304] pcie30_avdd1v8: supplied by vcc3v3_sys
[    0.139832] vcc3v3_bu: supplied by vcc5v0_sys
[    0.152751] rockchip-pm-domain fdd90000.power-management:power-controller: failed to get ack on domain 'pd_npu', target_idle = 0, target_ack = 0, val=0x6
[    0.152782] Kernel panic - not syncing: panic_on_set_idle set ...
[    0.152782] 
[    0.202834] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.19.193 #1
[    0.203377] Hardware name: Radxa ROCK 3 Model A (DT)
[    0.203828] Call trace:
[    0.204071]  dump_backtrace+0x0/0x188
[    0.204411]  show_stack+0x28/0x34
[    0.204716]  dump_stack+0x90/0xb8
ize 0 MiB
[    0.000000] OF: fdt: Reserved memory: failed t[    0.205471]  panic+0x138/0x29c
[    0.205754]  rockchip_pmu_set_idle_request+0x10c/0x1b8
[    0.206217]  rockchip_pd_power+0x218/0x250
[    0.206587]  rockchip_pd_power_on+0x30/0x44
[    0.206974]  genpd_power_on+0x110/0x1fc
[    0.207322]  __genpd_dev_pm_attach+0x1a0/0x1ec
[    0.207728]  genpd_dev_pm_attach+0x64/0x68
[    0.208103]  dev_pm_domain_attach+0x2c/0x48
o reserve memory for node 'drm-cubic-lut@00[    0.208804]  platform_drv_probe+0x44/0xa8
[    0.209164]  really_probe+0x2a4/0x398
[    0.209500]  driver_probe_device+0x128/0x138
[    0.209883]  device_driver_attach+0x4c/0x74
[    0.210265]  __driver_attach+0x14c/0x150
[    0.210623]  bus_for_each_dev+0x80/0xc8
[    0.210970]  driver_attach+0x34/0x40
[    0.211295]  bus_add_driver+0x1b4/0x1f8
000000': base 0x0000000000000000, size 0 MiB
[    0.000000] Reserved [    0.212168]  driver_register+0xb4/0xec
[    0.212505]  __platform_driver_register+0x5c/0x68
[    0.212937]  rk_iommu_init+0x24/0x2c
[    0.213264]  do_one_initcall+0xb0/0x1b0
[    0.213615]  kernel_init_freeable+0x2f8/0x2fc
[    0.214008]  kernel_init+0x1c/0x108
[    0.214324]  ret_from_fork+0x10/0x18
[    0.214654] SMP: stopping secondary CPUs

If you use Radxa U-boot branch, make sure the latest commit is https://github.com/radxa/u-boot/commit/fd3d0a5310494dcbba510406e1e2d66806c1b901.

Are you saying that the radxa uboot branch effects the kernel booting and generates this kernel panic ?

No. I think this kernel panic is caused by the NPU not given power earlier stage. The Radxa U-Boot fixs the issue by giving power for NPU earlier.

ok great. Is this version of uboot available on any prebuilt images ?

No. u-boot of prebuilt images is not built from the latest U-Boot.

OK - now that I got the latest uboot working the problem has gone.

Good news, I can boot the buildroot OS now and get a command prompt over console, plus it looks like the other devices such as network and audio and video are working. Very nice !

It will take more work to understand what is going on to make u-boot.itb and idbloader.img in your uboot build scripts … something to do with python2 there.
Once the uboot integration issues are solved, I should have the buildroot external dir up and running for this board : https://github.com/flatmax/buildroot.rockchip

3 Likes