ROCK 5B Debug Party Invitation

#442

The only thing i did not change was the dmc/governor .
No visible improvement with (maybe 0.5 fps spikes):

Previous:

root@rock5b:/home/rock# cat /sys/kernel/debug/clk/clk_summary | grep scmi_clk_
 scmi_clk_npu                         0        3        0   200000000          0     0  50000
 scmi_clk_gpu                         0        2        0   200000000          0     0  50000
 scmi_clk_ddr                         0        0        0   528000000          0     0  50000
 scmi_clk_cpub23                      0        0        0   408000000          0     0  50000
 scmi_clk_cpub01                      0        0        0   408000000          0     0  50000
 scmi_clk_dsu                         0        0        0           0          0     0  50000
 scmi_clk_cpul                        0        0        0  1416000000          0     0  50000

performance:

echo performance >/sys/devices/platform/dmc/devfreq/dmc/governor 
echo performance >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo performance >/sys/devices/system/cpu/cpu4/cpufreq/scaling_governor
echo performance >/sys/devices/system/cpu/cpu6/cpufreq/scaling_governor

checking again:

root@rock5b:/home/rock# cat /sys/kernel/debug/clk/clk_summary | grep scmi_clk_
 scmi_clk_npu                         0        3        0   200000000          0     0  50000
 scmi_clk_gpu                         0        2        0   200000000          0     0  50000
 scmi_clk_ddr                         0        0        0  2112000000          0     0  50000
 scmi_clk_cpub23                      0        0        0  2256000000          0     0  50000
 scmi_clk_cpub01                      0        0        0  2256000000          0     0  50000
 scmi_clk_dsu                         0        0        0           0          0     0  50000
 scmi_clk_cpul                        0        0        0  1800000000          0     0  50000

and monitoring while running the demo:

root@rock5b:/home/rock# ./mon.sh 
 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1800    2256    2256    2112       0     200     200
   1800    2256    2256    2112       0     200     200
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000

 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000

 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000

 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     300    1000
   1800    2256    2256    2112       0     200    1000

Update: forgot gpu and npu (1 fps improvement)

root@rock5b:/home/rock# ./mon.sh 
 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
   1800    2256    2256    2112       0    1000    1000
#443

So adjusting the cpufreq settings brought the biggest gain. Still interested in SMP/IRQ affinity… do both interrupts and tasks end up on big cores? Can you please post /proc/interrupts contents after such a test run and check with htop/atop where the tasks run and whether there are CPU bottlenecks present?

#444

This may be a naive question, but what kernel are you using to have access to the dmc governor settings ? I pulled the latest one from yesterday on the radxa github repo and am still not seeing the entry under /sys. Did you change any config ? I would have liked to test here to take measurements as well. The debug/clk entries say I’m at 2112 MHz for the DDR, and idling around 1.6W (mesured at the USB plug).

#445

check:
https://github.com/radxa/kernel/commit/4ce9a743b253c0c344686085213de1c4059b9d59
CONFIG_ARM_ROCKCHIP_DMC_DEVFREQ=y

#446

Thanks! So I’m having the right option:

$ zgrep DMC_DEVFREQ /proc/config.gz 
CONFIG_ARM_ROCKCHIP_DMC_DEVFREQ=y

I checked the DTB and found the matching dmc node:

$ dtc -I dtb -O dts < /boot/dtbs/5.10.66-21-rockchip-gbbe5dbfb385d/rockchip/rk3588-rock-5b.dtb
        dmc {
                compatible = "rockchip,rk3588-dmc";
                interrupts = <0x00 0x49 0x04>;
                interrupt-names = "complete";
                devfreq-events = <0x37>;
                clocks = <0x0e 0x04>;
                clock-names = "dmc_clk";
                operating-points-v2 = <0x38>;
                upthreshold = <0x28>;
                downdifferential = <0x14>;
                system-status-level = <0x01 0x04 0x08 0x08 0x02 0x01 0x10 0x04 0x10000 0x04 0x1000 0x08 0x4000 0x08 0x2000 0x08 0xc00 0x08>;
                auto-freq-en = <0x01>;
                status = "okay";
                center-supply = <0x39>;
                mem-supply = <0x3a>;
                phandle = <0x1d8>;
        };

One possibility could be that I’m running on rock5b-v1.1.dtb instead, as it doesn’t have this entry, but I’m seeing regulator-name “vcc3v3_pcie2x1l2” which isn’t in that one, so I’m still digging.

Edit: here’s what I’m seeing there:

$ ll /sys/devices/platform/dmc
ls: cannot access '/sys/devices/platform/dmc': No such file or directory
$ sudo find /sys -name '*dmc*
'
/sys/bus/platform/drivers/rockchip-dmc
/sys/firmware/devicetree/base/dmc
/sys/firmware/devicetree/base/__symbols__/dmc
/sys/firmware/devicetree/base/__symbols__/dmc_opp_table
/sys/firmware/devicetree/base/dmc-opp-table

$ ll /sys/bus/platform/drivers/rockchip-dmc
total 0
drwxr-xr-x   2 root root    0 Sep  4 17:58 .
drwxr-xr-x 180 root root    0 Sep  4 17:58 ..
--w-------   1 root root 4096 Sep  4 18:10 bind
--w-------   1 root root 4096 Sep  4 18:10 uevent
--w-------   1 root root 4096 Sep  4 18:10 unbind

But I do see the opp:

$ ll /sys/firmware/devicetree/base/dmc-opp-table
total 0
drwxr-xr-x   6 root root  0 Sep  4 17:58 .
drwxr-xr-x 373 root root  0 Sep  4 17:58 ..
-r--r--r--   1 root root 20 Sep  4 18:12 compatible
-r--r--r--   1 root root 14 Sep  4 18:12 name
-r--r--r--   1 root root  8 Sep  4 18:12 nvmem-cell-names
-r--r--r--   1 root root  4 Sep  4 18:12 nvmem-cells
drwxr-xr-x   2 root root  0 Sep  4 17:58 opp-1068000000
drwxr-xr-x   2 root root  0 Sep  4 17:58 opp-1560000000
drwxr-xr-x   2 root root  0 Sep  4 17:58 opp-2750000000
drwxr-xr-x   2 root root  0 Sep  4 17:58 opp-528000000
-r--r--r--   1 root root  4 Sep  4 18:12 phandle
-r--r--r--   1 root root 48 Sep  4 18:12 rockchip,leakage-voltage-sel
-r--r--r--   1 root root  4 Sep  4 18:12 rockchip,low-temp
-r--r--r--   1 root root  4 Sep  4 18:12 rockchip,low-temp-min-volt
-r--r--r--   1 root root  4 Sep  4 18:12 rockchip,temp-hysteresis
#447

/proc/interrupts and top while running:

top
top.png768×754 69.9 KB 

CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
 13:      30521      13066      10028       9245      11691       8999      13868       2998     GICv3  26 Level     arch_timer
 14:     260790      69239      10867      19868     140050       2317       1490        613     GICv3 321 Level     rk_timer
 15:          0          0          0          0          0          0          0          0     GICv3  23 Level     arm-pmu
 16:         23          0          0          0          0          0          0          0     GICv3 105 Level     dmc
 17:      11253          0          0          0          0          0          0          0     GICv3 126 Level     fb000000.gpu
 18:          1          0          0          0          0          0          0          0     GICv3 125 Level     fb000000.gpu
 19:      25112          0          0          0          0          0          0          0     GICv3 124 Level     fb000000.gpu
 20:        232          0          0          0          0          0          0          0     GICv3 247 Level     ehci_hcd:usb1
 21:          0          0          0          0          0          0          0          0     GICv3 248 Level     ohci_hcd:usb3
 22:       5319          0          0          0          0          0          0          0     GICv3 250 Level     ehci_hcd:usb2
 23:          0          0          0          0          0          0          0          0     GICv3 251 Level     ohci_hcd:usb4
 24:          0          0          0          0          0          0          0          0     GICv3 425 Level     rockchip_usb2phy
 25:          2          0          0          0          0          0          0          0     GICv3 423 Level     rockchip_usb2phy
 26:          2          0          0          0          0          0          0          0     GICv3 424 Level     rockchip_usb2phy
 27:      13858          0          0          0          0          0          0          0     GICv3 349 Level     fd880000.i2c
 28:       8172          0          0          0          0          0          0          0     GICv3 142 Level     fdab9000.iommu, fdab0000.npu
 29:          0          0          0          0          0          0          0          0     GICv3 143 Level     fdab9000.iommu, fdab0000.npu
 30:          0          0          0          0          0          0          0          0     GICv3 144 Level     fdab9000.iommu, fdab0000.npu
 31:          0          0          0          0          0          0          0          0     GICv3 151 Level     fdb50400.vdpu
 32:          0          0          0          0          0          0          0          0     GICv3 150 Level     fdb50800.iommu
 33:       5449          0          0          0          0          0          0          0     GICv3 146 Level     fdb60f00.iommu, rga3_core0
 34:          0          0          0          0          0          0          0          0     GICv3 147 Level     fdb70f00.iommu, rga3_core1
 35:          0          0          0          0          0          0          0          0     GICv3 148 Level     rga2
 36:          0          0          0          0          0          0          0          0     GICv3 161 Level     fdb90000.jpegd
 37:          0          0          0          0          0          0          0          0     GICv3 162 Level     fdb90480.iommu
 38:          0          0          0          0          0          0          0          0     GICv3 154 Level     fdba0000.jpege-core
 39:          0          0          0          0          0          0          0          0     GICv3 153 Level     fdba0800.iommu
 40:          0          0          0          0          0          0          0          0     GICv3 156 Level     fdba4000.jpege-core
 41:          0          0          0          0          0          0          0          0     GICv3 155 Level     fdba4800.iommu
 42:          0          0          0          0          0          0          0          0     GICv3 158 Level     fdba8000.jpege-core
 43:          0          0          0          0          0          0          0          0     GICv3 157 Level     fdba8800.iommu
 44:          0          0          0          0          0          0          0          0     GICv3 160 Level     fdbac000.jpege-core
 45:          0          0          0          0          0          0          0          0     GICv3 159 Level     fdbac800.iommu
 46:          0          0          0          0          0          0          0          0     GICv3 149 Level     fdbb0800.iommu, fdbb0000.iep
 47:          0          0          0          0          0          0          0          0     GICv3 133 Level     fdbd0000.rkvenc-core
 48:          0          0          0          0          0          0          0          0     GICv3 131 Level     fdbdf000.iommu
 49:          0          0          0          0          0          0          0          0     GICv3 132 Level     fdbdf000.iommu
 50:          0          0          0          0          0          0          0          0     GICv3 136 Level     fdbe0000.rkvenc-core
 51:          0          0          0          0          0          0          0          0     GICv3 134 Level     fdbef000.iommu
 52:          0          0          0          0          0          0          0          0     GICv3 135 Level     fdbef000.iommu
 53:          0          0          0          0          0          0          0          0     GICv3 127 Level     fdc38100.rkvdec-core
 54:          0          0          0          0          0          0          0          0     GICv3 128 Level     fdc38700.iommu
 55:          0          0          0          0          0          0          0          0     GICv3 129 Level     fdc48100.rkvdec-core
 56:          0          0          0          0          0          0          0          0     GICv3 130 Level     fdc48700.iommu
 57:       8199          0          0          0          0          0          0          0     GICv3 163 Level     rkisp_hw
 58:       4102          0          0          0          0          0          0          0     GICv3 165 Level     rkisp_hw
 59:          0          0          0          0          0          0          0          0     GICv3 166 Level     rkisp_hw
 60:          0          0          0          0          0          0          0          0     GICv3 164 Level     fdcb7f00.iommu
 61:          2          0          0          0          0          0          0          0     GICv3 187 Level     rkcifhw
 62:          0          0          0          0          0          0          0          0     GICv3 145 Level     fdce0800.iommu
 63:          0          0          0          0          0          0          0          0     GICv3 179 Level     rockchip-mipi-csi2
 64:          0          0          0          0          0          0          0          0     GICv3 180 Level     rockchip-mipi-csi2
 65:       9667          0          0          0          0          0          0          0     GICv3 188 Level     fdd97e00.iommu, fdd90000.vop
 67:          0          0          0          0          0          0          0          0     GICv3 201 Level     fde80000.hdmi
 68:          0          0          0          0          0          0          0          0     GICv3 202 Level     dw-hdmi-qp-cec
 69:          0          0          0          0          0          0          0          0     GICv3 203 Level     fde80000.hdmi
 70:        771          0          0          0          0          0          0          0     GICv3 204 Level     fde80000.hdmi
 71:          1          0          0          0          0          0          0          0     GICv3 392 Level     dw-hdmi-qp-hpd
 72:          0          0          0          0          0          0          0          0     GICv3 285 Level     pcie-sys
 77:      14515          0          0          0          0          0          0          0     GICv3 235 Level     dw-mci
 78:          8          0          0          0          0          0          0          0     GICv3 237 Level     mmc1
 81:         10          0          0          0          0          0          0          0     GICv3 118 Level     fea10000.dma-controller
 82:          0          0          0          0          0          0          0          0     GICv3 119 Level     fea10000.dma-controller
 83:          0          0          0          0          0          0          0          0     GICv3 120 Level     fea30000.dma-controller
 84:          0          0          0          0          0          0          0          0     GICv3 121 Level     fea30000.dma-controller
 85:         73          0          0          0          0          0          0          0     GICv3 350 Level     fea90000.i2c
 86:        759          0          0          0          0          0          0          0     GICv3 352 Level     feab0000.i2c
 87:        195          0          0          0          0          0          0          0     GICv3 353 Level     feac0000.i2c
 88:      19920          0          0          0          0          0          0          0     GICv3 360 Level     feb20000.spi
 89:          0          0          0          0          0          0          0          0     GICv3 429 Level     rockchip_thermal
 90:          0          0          0          0          0          0          0          0     GICv3 430 Level     fec10000.saradc
 91:        128          0          0          0          0          0          0          0     GICv3 355 Level     fec80000.i2c
 92:        442          0          0          0          0          0          0          0     GICv3 356 Level     fec90000.i2c
 93:         10          0          0          0          0          0          0          0     GICv3 122 Level     fed10000.dma-controller
 94:          0          0          0          0          0          0          0          0     GICv3 123 Level     fed10000.dma-controller
100:          0          0          0          0          0          0          0          0     GICv3 426 Level     rockchip_usb2phy
103:          0          0          0          0          0          0          0          0     GICv3 205 Level     fdea0000.hdmi
104:          0          0          0          0          0          0          0          0     GICv3 206 Level     dw-hdmi-qp-cec
105:          0          0          0          0          0          0          0          0     GICv3 207 Level     fdea0000.hdmi
106:       1799          0          0          0          0          0          0          0     GICv3 208 Level     fdea0000.hdmi
107:          1          0          0          0          0          0          0          0     GICv3 393 Level     dw-hdmi-qp-hpd
108:          0          0          0          0          0          0          0          0     GICv3 209 Level     rk_hdmirx_cec
109:          0          0          0          0          2          0          0          0     GICv3 468 Level     rk_hdmirx-hdmi
110:          0          0          0          0          0          0          0          0     GICv3 211 Level     rk_hdmirx-dma
121:          0          0          0          0          0          0          0          0     GICv3 455 Edge      debug-signal
122:          0          0          0          0          0          0          0          0     GICv3 365 Level     debug
123:          0          0          0          0          0          0          0          0     GICv3 140 Level     av1d-master
124:          0          0          0          0          0          0          0          0     GICv3 139 Level     irq_cache
126:          0          0          0          0          0          0          0          0  rockchip_gpio_irq   7 Level     rk806
127:          0          0          0          0          0          0          0          0     rk806   0 Edge      rk805_pwrkey_fall
128:          0          0          0          0          0          0          0          0     rk806   1 Edge      rk805_pwrkey_rise
129:          0          0          0          0          0          0          0          0     rk806   7 Level     rk806_vb_low
131:          0          0          0          0          0          0          0          0     GICv3 254 Level     xhci-hcd:usb5
132:          0          0          0          0          0          0          0          0     GICv3 253 Level     xhci-hcd:usb7
133:          8          0          0          0          0          0          0          0  rockchip_gpio_irq  12 Level     fsc_interrupt_int_n
134:          0          0          0          0          0          0          0          0  rockchip_gpio_irq   8 Level     hym8563
135:          0          0          0          0          0          0          0          0  rockchip_gpio_irq  22 Edge      rk_hdmirx-5v
136:          0          0          0          0          0          0          0          0  rockchip_gpio_irq  29 Level     headset_input
146:          0          0          0          0          0          0          0          0   ITS-MSI 570425352 Edge      PCIe PME
147:          0       8489          0          0          0          0          0          0   ITS-MSI 570949632 Edge      enP4p65s0-0
148:          0          0          0          0          0          0          0          0   ITS-MSI 570949633 Edge      enP4p65s0-1
149:          0          0          0          0          0          0          0          0   ITS-MSI 570949634 Edge      enP4p65s0-2
150:          0          0          0          0          0          0          0          0   ITS-MSI 570949635 Edge      enP4p65s0-3
151:          0          0          0          0          0          0          0          0   ITS-MSI 570949636 Edge      enP4p65s0-4
152:          0          0          0          0          0          0          0          0   ITS-MSI 570949637 Edge      enP4p65s0-5
153:          0          0          0          0          0          0          0          0   ITS-MSI 570949638 Edge      enP4p65s0-6
154:          0          0          0          0          0          0          0          0   ITS-MSI 570949639 Edge      enP4p65s0-7
155:          0          0          0          0          0          0          0          0   ITS-MSI 570949640 Edge      enP4p65s0-8
156:          0          0          0          0          0          0          0          0   ITS-MSI 570949641 Edge      enP4p65s0-9
157:          0          0          0          0          0          0          0          0   ITS-MSI 570949642 Edge      enP4p65s0-10
158:          0          0          0          0          0          0          0          0   ITS-MSI 570949643 Edge      enP4p65s0-11
159:          0          0          0          0          0          0          0          0   ITS-MSI 570949644 Edge      enP4p65s0-12
160:          0          0          0          0          0          0          0          0   ITS-MSI 570949645 Edge      enP4p65s0-13
161:          0          0          0          0          0          0          0          0   ITS-MSI 570949646 Edge      enP4p65s0-14
162:          0          0          0          0          0          0          0          0   ITS-MSI 570949647 Edge      enP4p65s0-15
163:          0       5204          0          0          0          0          0          0   ITS-MSI 570949648 Edge      enP4p65s0-16
164:          0          0          0          0          0          0          0          0   ITS-MSI 570949649 Edge      enP4p65s0-17
165:          0          0          0       2816          0          0          0          0   ITS-MSI 570949650 Edge      enP4p65s0-18
166:          0          0          0          0          0          0          0          0   ITS-MSI 570949651 Edge      enP4p65s0-19
167:          0          0          0          0          0          0          0          0   ITS-MSI 570949652 Edge      enP4p65s0-20
168:          0          0          0          0          0          0          1          0   ITS-MSI 570949653 Edge      enP4p65s0-21
169:          0          0          0          0          0          0          0          0   ITS-MSI 570949654 Edge      enP4p65s0-22
170:          0          0          0          0          0          0          0          0   ITS-MSI 570949655 Edge      enP4p65s0-23
171:          0          0          0          0          0          0          0          0   ITS-MSI 570949656 Edge      enP4p65s0-24
172:          0          0          0          0          0          0          0          0   ITS-MSI 570949657 Edge      enP4p65s0-25
173:          0          0          0          0          0          0          0          0   ITS-MSI 570949658 Edge      enP4p65s0-26
174:          0          0          0          0          0          0          0          0   ITS-MSI 570949659 Edge      enP4p65s0-27
175:          0          0          0          0          0          0          0          0   ITS-MSI 570949660 Edge      enP4p65s0-28
176:          0          0          0          0          0          0          0          0   ITS-MSI 570949661 Edge      enP4p65s0-29
177:          0          0          0          0          0          0          0          0   ITS-MSI 570949662 Edge      enP4p65s0-30
178:          0          0          0          0          0          0          0          0   ITS-MSI 570949663 Edge      enP4p65s0-31
IPI0:     23874      32924      35537      30001       3000       3622        848       1528       Rescheduling interrupts
IPI1:     67590       9725       6223       7860      27526      17384      24073      22052       Function call interrupts
IPI2:         0          0          0          0          0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0          0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:     14154      22057      11678      10418       8117       2917       4299       1336       Timer broadcast interrupts
IPI5:    237063      18620       6360       2514     183085       1552       2639        737       IRQ work interrupts
IPI6:         0          0          0          0          0          0          0          0       CPU wake-up interrupts
Err:          0
#448

after:

CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
 13:      30735      13200      10154       9328      12100       9302      13892       3034     GICv3  26 Level     arch_timer
 14:     260990      69552      10913      19904     140050       2317       1490        613     GICv3 321 Level     rk_timer
 15:          0          0          0          0          0          0          0          0     GICv3  23 Level     arm-pmu
 16:         23          0          0          0          0          0          0          0     GICv3 105 Level     dmc
 17:      11398          0          0          0          0          0          0          0     GICv3 126 Level     fb000000.gpu
 18:          1          0          0          0          0          0          0          0     GICv3 125 Level     fb000000.gpu
 19:      25453          0          0          0          0          0          0          0     GICv3 124 Level     fb000000.gpu
 20:        234          0          0          0          0          0          0          0     GICv3 247 Level     ehci_hcd:usb1
 21:          0          0          0          0          0          0          0          0     GICv3 248 Level     ohci_hcd:usb3
 22:       5319          0          0          0          0          0          0          0     GICv3 250 Level     ehci_hcd:usb2
 23:          0          0          0          0          0          0          0          0     GICv3 251 Level     ohci_hcd:usb4
 24:          0          0          0          0          0          0          0          0     GICv3 425 Level     rockchip_usb2phy
 25:          2          0          0          0          0          0          0          0     GICv3 423 Level     rockchip_usb2phy
 26:          2          0          0          0          0          0          0          0     GICv3 424 Level     rockchip_usb2phy
 27:      13858          0          0          0          0          0          0          0     GICv3 349 Level     fd880000.i2c
 28:       8337          0          0          0          0          0          0          0     GICv3 142 Level     fdab9000.iommu, fdab0000.npu
 29:          0          0          0          0          0          0          0          0     GICv3 143 Level     fdab9000.iommu, fdab0000.npu
 30:          0          0          0          0          0          0          0          0     GICv3 144 Level     fdab9000.iommu, fdab0000.npu
 31:          0          0          0          0          0          0          0          0     GICv3 151 Level     fdb50400.vdpu
 32:          0          0          0          0          0          0          0          0     GICv3 150 Level     fdb50800.iommu
 33:       5558          0          0          0          0          0          0          0     GICv3 146 Level     fdb60f00.iommu, rga3_core0
 34:          0          0          0          0          0          0          0          0     GICv3 147 Level     fdb70f00.iommu, rga3_core1
 35:          0          0          0          0          0          0          0          0     GICv3 148 Level     rga2
 36:          0          0          0          0          0          0          0          0     GICv3 161 Level     fdb90000.jpegd
 37:          0          0          0          0          0          0          0          0     GICv3 162 Level     fdb90480.iommu
 38:          0          0          0          0          0          0          0          0     GICv3 154 Level     fdba0000.jpege-core
 39:          0          0          0          0          0          0          0          0     GICv3 153 Level     fdba0800.iommu
 40:          0          0          0          0          0          0          0          0     GICv3 156 Level     fdba4000.jpege-core
 41:          0          0          0          0          0          0          0          0     GICv3 155 Level     fdba4800.iommu
 42:          0          0          0          0          0          0          0          0     GICv3 158 Level     fdba8000.jpege-core
 43:          0          0          0          0          0          0          0          0     GICv3 157 Level     fdba8800.iommu
 44:          0          0          0          0          0          0          0          0     GICv3 160 Level     fdbac000.jpege-core
 45:          0          0          0          0          0          0          0          0     GICv3 159 Level     fdbac800.iommu
 46:          0          0          0          0          0          0          0          0     GICv3 149 Level     fdbb0800.iommu, fdbb0000.iep
 47:          0          0          0          0          0          0          0          0     GICv3 133 Level     fdbd0000.rkvenc-core
 48:          0          0          0          0          0          0          0          0     GICv3 131 Level     fdbdf000.iommu
 49:          0          0          0          0          0          0          0          0     GICv3 132 Level     fdbdf000.iommu
 50:          0          0          0          0          0          0          0          0     GICv3 136 Level     fdbe0000.rkvenc-core
 51:          0          0          0          0          0          0          0          0     GICv3 134 Level     fdbef000.iommu
 52:          0          0          0          0          0          0          0          0     GICv3 135 Level     fdbef000.iommu
 53:          0          0          0          0          0          0          0          0     GICv3 127 Level     fdc38100.rkvdec-core
 54:          0          0          0          0          0          0          0          0     GICv3 128 Level     fdc38700.iommu
 55:          0          0          0          0          0          0          0          0     GICv3 129 Level     fdc48100.rkvdec-core
 56:          0          0          0          0          0          0          0          0     GICv3 130 Level     fdc48700.iommu
 57:       8332          0          0          0          0          0          0          0     GICv3 163 Level     rkisp_hw
 58:       4169          0          0          0          0          0          0          0     GICv3 165 Level     rkisp_hw
 59:          0          0          0          0          0          0          0          0     GICv3 166 Level     rkisp_hw
 60:          0          0          0          0          0          0          0          0     GICv3 164 Level     fdcb7f00.iommu
 61:          3          0          0          0          0          0          0          0     GICv3 187 Level     rkcifhw
 62:          0          0          0          0          0          0          0          0     GICv3 145 Level     fdce0800.iommu
 63:          0          0          0          0          0          0          0          0     GICv3 179 Level     rockchip-mipi-csi2
 64:          0          0          0          0          0          0          0          0     GICv3 180 Level     rockchip-mipi-csi2
 65:      10088          0          0          0          0          0          0          0     GICv3 188 Level     fdd97e00.iommu, fdd90000.vop
 67:          0          0          0          0          0          0          0          0     GICv3 201 Level     fde80000.hdmi
 68:          0          0          0          0          0          0          0          0     GICv3 202 Level     dw-hdmi-qp-cec
 69:          0          0          0          0          0          0          0          0     GICv3 203 Level     fde80000.hdmi
 70:        771          0          0          0          0          0          0          0     GICv3 204 Level     fde80000.hdmi
 71:          1          0          0          0          0          0          0          0     GICv3 392 Level     dw-hdmi-qp-hpd
 72:          0          0          0          0          0          0          0          0     GICv3 285 Level     pcie-sys
 77:      14521          0          0          0          0          0          0          0     GICv3 235 Level     dw-mci
 78:          8          0          0          0          0          0          0          0     GICv3 237 Level     mmc1
 81:         10          0          0          0          0          0          0          0     GICv3 118 Level     fea10000.dma-controller
 82:          0          0          0          0          0          0          0          0     GICv3 119 Level     fea10000.dma-controller
 83:          0          0          0          0          0          0          0          0     GICv3 120 Level     fea30000.dma-controller
 84:          0          0          0          0          0          0          0          0     GICv3 121 Level     fea30000.dma-controller
 85:         73          0          0          0          0          0          0          0     GICv3 350 Level     fea90000.i2c
 86:        760          0          0          0          0          0          0          0     GICv3 352 Level     feab0000.i2c
 87:        195          0          0          0          0          0          0          0     GICv3 353 Level     feac0000.i2c
 88:      19923          0          0          0          0          0          0          0     GICv3 360 Level     feb20000.spi
 89:          0          0          0          0          0          0          0          0     GICv3 429 Level     rockchip_thermal
 90:          0          0          0          0          0          0          0          0     GICv3 430 Level     fec10000.saradc
 91:        128          0          0          0          0          0          0          0     GICv3 355 Level     fec80000.i2c
 92:        442          0          0          0          0          0          0          0     GICv3 356 Level     fec90000.i2c
 93:         10          0          0          0          0          0          0          0     GICv3 122 Level     fed10000.dma-controller
 94:          0          0          0          0          0          0          0          0     GICv3 123 Level     fed10000.dma-controller
100:          0          0          0          0          0          0          0          0     GICv3 426 Level     rockchip_usb2phy
103:          0          0          0          0          0          0          0          0     GICv3 205 Level     fdea0000.hdmi
104:          0          0          0          0          0          0          0          0     GICv3 206 Level     dw-hdmi-qp-cec
105:          0          0          0          0          0          0          0          0     GICv3 207 Level     fdea0000.hdmi
106:       1799          0          0          0          0          0          0          0     GICv3 208 Level     fdea0000.hdmi
107:          1          0          0          0          0          0          0          0     GICv3 393 Level     dw-hdmi-qp-hpd
108:          0          0          0          0          0          0          0          0     GICv3 209 Level     rk_hdmirx_cec
109:          0          0          0          0          2          0          0          0     GICv3 468 Level     rk_hdmirx-hdmi
110:          0          0          0          0          0          0          0          0     GICv3 211 Level     rk_hdmirx-dma
121:          0          0          0          0          0          0          0          0     GICv3 455 Edge      debug-signal
122:          0          0          0          0          0          0          0          0     GICv3 365 Level     debug
123:          0          0          0          0          0          0          0          0     GICv3 140 Level     av1d-master
124:          0          0          0          0          0          0          0          0     GICv3 139 Level     irq_cache
126:          0          0          0          0          0          0          0          0  rockchip_gpio_irq   7 Level     rk806
127:          0          0          0          0          0          0          0          0     rk806   0 Edge      rk805_pwrkey_fall
128:          0          0          0          0          0          0          0          0     rk806   1 Edge      rk805_pwrkey_rise
129:          0          0          0          0          0          0          0          0     rk806   7 Level     rk806_vb_low
131:          0          0          0          0          0          0          0          0     GICv3 254 Level     xhci-hcd:usb5
132:          0          0          0          0          0          0          0          0     GICv3 253 Level     xhci-hcd:usb7
133:          8          0          0          0          0          0          0          0  rockchip_gpio_irq  12 Level     fsc_interrupt_int_n
134:          0          0          0          0          0          0          0          0  rockchip_gpio_irq   8 Level     hym8563
135:          0          0          0          0          0          0          0          0  rockchip_gpio_irq  22 Edge      rk_hdmirx-5v
136:          0          0          0          0          0          0          0          0  rockchip_gpio_irq  29 Level     headset_input
146:          0          0          0          0          0          0          0          0   ITS-MSI 570425352 Edge      PCIe PME
147:          0       8518          0          0          0          0          0          0   ITS-MSI 570949632 Edge      enP4p65s0-0
148:          0          0          0          0          0          0          0          0   ITS-MSI 570949633 Edge      enP4p65s0-1
149:          0          0          0          0          0          0          0          0   ITS-MSI 570949634 Edge      enP4p65s0-2
150:          0          0          0          0          0          0          0          0   ITS-MSI 570949635 Edge      enP4p65s0-3
151:          0          0          0          0          0          0          0          0   ITS-MSI 570949636 Edge      enP4p65s0-4
152:          0          0          0          0          0          0          0          0   ITS-MSI 570949637 Edge      enP4p65s0-5
153:          0          0          0          0          0          0          0          0   ITS-MSI 570949638 Edge      enP4p65s0-6
154:          0          0          0          0          0          0          0          0   ITS-MSI 570949639 Edge      enP4p65s0-7
155:          0          0          0          0          0          0          0          0   ITS-MSI 570949640 Edge      enP4p65s0-8
156:          0          0          0          0          0          0          0          0   ITS-MSI 570949641 Edge      enP4p65s0-9
157:          0          0          0          0          0          0          0          0   ITS-MSI 570949642 Edge      enP4p65s0-10
158:          0          0          0          0          0          0          0          0   ITS-MSI 570949643 Edge      enP4p65s0-11
159:          0          0          0          0          0          0          0          0   ITS-MSI 570949644 Edge      enP4p65s0-12
160:          0          0          0          0          0          0          0          0   ITS-MSI 570949645 Edge      enP4p65s0-13
161:          0          0          0          0          0          0          0          0   ITS-MSI 570949646 Edge      enP4p65s0-14
162:          0          0          0          0          0          0          0          0   ITS-MSI 570949647 Edge      enP4p65s0-15
163:          0       5220          0          0          0          0          0          0   ITS-MSI 570949648 Edge      enP4p65s0-16
164:          0          0          0          0          0          0          0          0   ITS-MSI 570949649 Edge      enP4p65s0-17
165:          0          0          0       2821          0          0          0          0   ITS-MSI 570949650 Edge      enP4p65s0-18
166:          0          0          0          0          0          0          0          0   ITS-MSI 570949651 Edge      enP4p65s0-19
167:          0          0          0          0          0          0          0          0   ITS-MSI 570949652 Edge      enP4p65s0-20
168:          0          0          0          0          0          0          1          0   ITS-MSI 570949653 Edge      enP4p65s0-21
169:          0          0          0          0          0          0          0          0   ITS-MSI 570949654 Edge      enP4p65s0-22
170:          0          0          0          0          0          0          0          0   ITS-MSI 570949655 Edge      enP4p65s0-23
171:          0          0          0          0          0          0          0          0   ITS-MSI 570949656 Edge      enP4p65s0-24
172:          0          0          0          0          0          0          0          0   ITS-MSI 570949657 Edge      enP4p65s0-25
173:          0          0          0          0          0          0          0          0   ITS-MSI 570949658 Edge      enP4p65s0-26
174:          0          0          0          0          0          0          0          0   ITS-MSI 570949659 Edge      enP4p65s0-27
175:          0          0          0          0          0          0          0          0   ITS-MSI 570949660 Edge      enP4p65s0-28
176:          0          0          0          0          0          0          0          0   ITS-MSI 570949661 Edge      enP4p65s0-29
177:          0          0          0          0          0          0          0          0   ITS-MSI 570949662 Edge      enP4p65s0-30
178:          0          0          0          0          0          0          0          0   ITS-MSI 570949663 Edge      enP4p65s0-31
IPI0:     23906      33229      35910      30217       3075       3687        848       1528       Rescheduling interrupts
IPI1:     67788       9815       6365       7950      27906      17672      24079      22236       Function call interrupts
IPI2:         0          0          0          0          0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0          0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:     14184      22103      11706      10451       8117       2917       4299       1336       Timer broadcast interrupts
IPI5:    237063      18620       6360       2514     183085       1552       2639        737       IRQ work interrupts
IPI6:         0          0          0          0          0          0          0          0       CPU wake-up interrupts
Err:          0
#449

Update:

I stopped the 3A engine and started again, now i have:

top2
top2.png768×754 69.3 KB

Running without 3A

top3
top3.png768×754 68.6 KB

ouch, and missed this:

#450

In this situation it seems one CPU core is maxing out with ksoftirqd while otherwise /proc/interrupts shows no IRQ bottleneck. Would still be interesting to verify with htop and atop whether there are bottlenecks (tasks/interrupts maxing out a little core) when running your tests. Since if that’s the case it’s time for IRQ affinity and SMP affinity moving that stuff over to the big cores.

#451

atop
atop.png744×567 58.6 KB

htop:

htop
htop.png688×890 133 KB

#452

Finally, i now understand RkRgaBlit (2D engine) only works passing dma bufs.
I was able to capture frames and render them with drm-fb. 1920x1080, 1.3% CPU.

drm-fb
drm-fb.png744×567 90.6 KB

#453

Anyone wants to try Python on a GPU?

Patched kernel no longer required:

$ ./repl
Use CTRL-D to quit

MicroPython v1.19.1-298-gc616721b1 on 2022-09-08; Mali CSF with Arm Cortex-M7
Type "help()" for more information.
>>> def fib(i):
...     if i <= 2:
...         return 1
...     else:
...         return fib(i - 1) + fib(i - 2)
...
>>> fib(20)
6765

Not much works yet, but soon I should have the environment usable enough to write all of the firmware code in Python. Imagine the fun you’ll be able to have once you can mess about with GPU jobs and implement all the wallhacks you want, invisible to software!

(More seriously, it would be nice to have help with writing the firmware code.)

3 Likes
#454

Nice
btw what is the status of panfrost on RK3588 ?
Is it working already?

#455

It’s been working for a while… just not well enough that you’d actually want to use it.

I’m still trying to fully work out how the command stream interacts with the firmware, so that new GPU jobs can start as soon as the old one finishes, rather than the current implementation which just waits for an amount of time and hopes that’s enough.

#456

As mali kernel driver is open source.
How mali drivers handle this?

Or it something you want to tune from (panfrost)user space?

#457

Userspace has to interact directly with the firmware; the kernel only sets up the MMU and command queues so that CSF knows where to find instructions, as well as handling power management and the like.

(“CSF” is “Command Stream Frontend”, a term which covers multiple parts of the GPU, such as the command stream processor, the firmware, and “iterators” which submit jobs to the hardware.)

#458

Thanks to @tkaiser’s image I could figure what was wrong with my dmc node not appearing. The system upgrade was performed with /boot unmounted, so the new kernel was installed in mmcblk1p2, leaving the origin one intact, which explains why I could see the node there but it wasn’t used at boot. However now apt/dpkg are angry and refuse to remove old kernels whatever the options I throw at them.

Anyway I could perform precise idle power measurements with cpu freq and dmc freq, with a watt meter connected on USB (however it interacts with PD so it’s running on 5V). I started with everything set to ondemand (thus at the lowest frequencies), then turned cluster0, then 4, then 6 to performance, then played with dmc freq and governor.

  • base: 1.023 W
  • cluster0 performance (1800 MHz): 1.075 W
  • +cluster4 performance (2304 MHz): 1.128 W
  • +cluster6 performance (2304 MHz): 1.168 W
  • dmc performance (1068 MHz): 1.385 W
  • dmc performance (1560 MHz): 1.546 W
  • dmc performance (2112 MHz): 1.758 W

This gives me 590 mW when turning the dmc from 528 to 2112 MHz. The difference with Thomas’ measurements above can be explained by several factors:

  • measurement on AC vs DC (always less accurate on DC)
  • measurement including/excluding the PSU losses
  • my board has 4 GB RAM so the measurement may also be affected by RAM consumption and not just the DMC’s. If that’s the case, it further makes sense to keep it in dynamic mode on boards with lots of memory.
#459

…isn’t that big anyway. I measured (at the wall) first with an USB PD charger a 680mW difference between powersave and performance (dmc governor) and then rechecking with the ‘dumb’ RPi 15W USB-C power brick 630mW difference.

As such it’s save to assume that we’re talking about a difference of ~600mW at the device which might translate to more at the wall based on wattage/crappyness of the PSU used.

#460

Yep and in this case the RAM size probably doesn’t count much.

#461

I already thought about measuring how RAM size matters with full CPU load. The problem: many benchmarks do different things along the way and as such actual consumption will vary every second.

While my Netio powermeters could cope with this (letting them measure not actual consumption but cumulative consumption, then doing some math and use an averaged value over an hour or so) I guess it’s impossible with a wattmeter only showing actual (fluctuating) consumption. But maybe tasks like cpuminer benchmark or stress-ng generate a constant load pattern resulting in consistent consumption numbers?

If you’re in the mood and find something we could compare (trying to take the slight differences wrt PVTM/clockspeeds of our both RK3588 into account)