ROCK 5B Debug Party Invitation

[tkaiser]

create a table (trying to filter out @amazingfate’s overvolting attempts) with pvtm values ranging from 1629 to 1805 for the A76. Here’s the table.

Looking at the different devices is interesting: RK3588S in Khadas Edge 2 shows up only with low pvtm values but also with very high idle temperatures (combined with rather low temperatures after executing the benchmark which hints at using their fansink). Also almost always the A76 end up being clocked at just 395 MHz during ‘full CPU utilization’ (running the 7-zip benchmark):

device	pvtm	pvtm-volt-sel	OPP	real mhz before/after	idle temp
edge2 (30 Oct 2022)	1638	2	2304	2186/2177	40.7°C
edge2 (02 Oct 2022)	1654	3	2352	2209/394	50.8°C
edge2 (30 Oct 2022)	1656	3	2352	2209/394	40.7°C
edge2 (02 Oct 2022)	1660	3	2352	2215/394	50.8°C
edge2 (25 Oct 2022)	1664	3	2352	2223/394	45.3°C
edge2 (08 Oct 2022)	1665	3	2352	2235/395	39.8°C
edge2 (08 Oct 2022)	1669	3	2352	2237/395	39.8°C
edge2 (25 Oct 2022)	1669	3	2352	2226/394	45.3°C
edge2 (26 Oct 2022)	1690	4	2256	2226/394	54.5°C
edge2 (26 Oct 2022)	1692	4	2256	2231/394	54.5°C
edge2 (29 Sep 2022)	1700	4	2256	2235/394	49.0°C
edge2 (29 Sep 2022)	1700	4	2256	2238/394	49.0°C
edge2 (20 Oct 2022)	1701	4	2256	2253/394	43.5°C
edge2 (20 Oct 2022)	1704	4	2256	2253/394	43.5°C
edge2 (10 Sep 2022)	1711	5	2304	2257/2257	48.1°C
edge2 (29 Sep 2022)	1711	5	2304	2249/2252	49.0°C
edge2 (29 Sep 2022)	1711	5	2304	2251/2257	49.0°C
edge2 (10 Sep 2022)	1716	5	2304	2259/2260	48.1°C

Again RK3588S in Firefly Station M3. pvtm values slightly higher compared to Khadas and also idle temps way lower:

device	pvtm	pvtm-volt-sel	OPP	real mhz before/after	idle temp
roc-rk3588s-pc (05 Sep 2022)	1685	4	2256	2245/2236	33.3°C
roc-rk3588s-pc (05 Sep 2022)	1702	4	2256	2258/2250	33.3°C
roc-rk3588s-pc (02 Sep 2022)	1709	4	2256	2262/2248	36.1°C
roc-rk3588s-pc (02 Sep 2022)	1717	5	2304	2268/2254	36.1°C
roc-rk3588s-pc (21 Sep 2022)	1739	5	2304	2312/2304	30.5°C
roc-rk3588s-pc (21 Sep 2022)	1748	6	2352	2315/2308	30.5°C
roc-rk3588s-pc (28 Aug 2022)	1753	6	2352	2311/2302	43.5°C
roc-rk3588s-pc (02 Sep 2022)	1753	6	2352	2318/2309	37.0°C
roc-rk3588s-pc (08 Sep 2022)	1754	6	2352	2317/2309	30.9°C
roc-rk3588s-pc (26 Aug 2022)	1758	6	2352	2322/2320	39.8°C
roc-rk3588s-pc (28 Aug 2022)	1762	6	2352	2318/2308	43.5°C
roc-rk3588s-pc (26 Aug 2022)	1763	6	2352	2324/2321	39.8°C
roc-rk3588s-pc (02 Sep 2022)	1763	6	2352	2326/2316	37.0°C
roc-rk3588s-pc (08 Sep 2022)	1763	6	2352	2323/2316	30.9°C

And this thing here is again @amazingfate most probably already having tweaked the OPP tables:

device	pvtm	pvtm-volt-sel	OPP	real mhz before/after	idle temp
rk3588-owl-h88k (19 Oct 2022)	1696	4	2400	2406/2393*	25.9°C
rk3588-owl-h88k (19 Oct 2022)	1696	4	2400	2443/2428*	25.0°C
rk3588-owl-h88k-v1 (19 Oct 2022)	1697	4	2400	2443/2429*	25.9°C
rk3588-owl-h88k (19 Oct 2022)	1699	4	2400	2410/2399	25.9°C
rk3588-owl-h88k (19 Oct 2022)	1700	4	2400	2446/2433	25.0°C
rk3588-owl-h88k-v1 (19 Oct 2022)	1700	4	2400	2446/2434	25.9°C
rk3588-owl-h88k-v1 (19 Oct 2022)	1702	4	2400	2285/2277	25.0°C
rk3588-owl-h88k-v1 (19 Oct 2022)	1785	7	2400	2290/2282	25.0°C

[willy]

Very useful, thanks for sharing! It clarifies a few points indeed, about how the opp should be interpreted.

[willy]

The 395 MHz is quite annoying, I’ve met it a few times already during my experimentations. I noticed that my changes were not consistently accepted across reboots, and that sometimes a power cycle was needed, and even sometimes some power cycles didn’t help. I suspect that the existence of other OPPs that are conditioned by PVTM values which may change between boots have an impact. There might be some bugs in the regulator or opp code that makes some opps conflict with others. It would be nice if we could make sure nobody meets the 395 MHz by accident anymore.

[tkaiser]

It would be nice if we could make sure nobody meets the 395 MHz by accident anymore.

So far with default settings (using what’s present in RK’s BSP) only other devices seem to be affected (all of them showing a high idle temperature):

device	pvtm	pvtm-volt-sel	OPP	real mhz before/after	idle temp
rk3588-evb7-lp4-v10 (15 Aug 2022)	1649	3	2352	394/394	49.9°C
edge2 (02 Oct 2022)	1654	3	2352	2209/394	50.8°C
edge2 (30 Oct 2022)	1656	3	2352	2209/394	40.7°C
edge2 (02 Oct 2022)	1660	3	2352	2215/394	50.8°C
rk3588-evb7-lp4-v10 (15 Aug 2022)	1662	3	2352	394/394	49.9°C
edge2 (25 Oct 2022)	1664	3	2352	2223/394	45.3°C
edge2 (08 Oct 2022)	1665	3	2352	2235/395	39.8°C
edge2 (08 Oct 2022)	1669	3	2352	2237/395	39.8°C
edge2 (25 Oct 2022)	1669	3	2352	2226/394	45.3°C
edge2 (26 Oct 2022)	1690	4	2256	2226/394	54.5°C
edge2 (26 Oct 2022)	1692	4	2256	2231/394	54.5°C
edge2 (29 Sep 2022)	1700	4	2256	2235/394	49.0°C
edge2 (29 Sep 2022)	1700	4	2256	2238/394	49.0°C
edge2 (20 Oct 2022)	1701	4	2256	2253/394	43.5°C
edge2 (20 Oct 2022)	1704	4	2256	2253/394	43.5°C

[stuartiannaylor]

I’m particularly surprised to see that the frequency seems to be completely ignored and only the voltage is used. It’s as if they were just using a VCO (after all that’s what a PLL is made of, but…).

I have no idea but is all this some state of flux to employ https://www.kernel.org/doc/html/latest/scheduler/sched-energy.html#dependencies-and-requirements-for-eas
https://docs.kernel.org/scheduler/schedutil.html

Anyway I thought I would barge in with a tangential that as with big.Little layouts the schedulers don’t make sense especially the default ordering of the cores would always seem the wrong way round.
Also with affinity near everything seems to be running on core 0 and being lazy (in fact wanting dynamic) I installed irqbalance which has seen some improvements.

The default scheduling core and irq affinity is pretty bad and wondered if you guys had any solutions?

[tkaiser]

The default scheduling core and irq affinity is pretty bad and wondered if you guys had any solutions?

The device DT contains capacity-dmips-mhz = <530>; for the A55 and capacity-dmips-mhz = <1024>; for the A76 as such the scheduler should prefer the latter for jobs. Works just fine for me (using ondemand for reasons):

tk@rock-5b:~$ 7zr b -mmt=1

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=eo_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs LE)

LE
CPU Freq: - - - 64000000 - - - - -

RAM size:   15720 MB,  # CPU hardware threads:   8
RAM usage:    435 MB,  # Benchmark threads:      1

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       2196   100   2140   2137  |      37441   100   3198   3197
23:       2001   100   2044   2039  |      36239   100   3139   3137
24:       1918   100   2066   2063  |      34390   100   3020   3019
25:       1823   100   2084   2082  |      32139   100   2862   2861
----------------------------------  | ------------------------------
Avr:             100   2083   2080  |              100   3055   3053
Tot:             100   2569   2567

tk@rock-5b:~$ taskset -c 7 7zr b -mmt=1

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=eo_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs LE)

LE
CPU Freq: 64000000 - - - - 256000000 - - -

RAM size:   15720 MB,  # CPU hardware threads:   8
RAM usage:    435 MB,  # Benchmark threads:      1

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       2164   100   2108   2106  |      37132   100   3172   3170
23:       1992   100   2033   2030  |      36189   100   3134   3133
24:       1908   100   2054   2052  |      34312   100   3013   3012
25:       1822   100   2083   2081  |      31840   100   2835   2834
----------------------------------  | ------------------------------
Avr:             100   2070   2067  |              100   3038   3037
Tot:             100   2554   2552

Wrt IRQ affinity I would believe the best is static mappings suitable for different use cases (moving e.g. PCIe and USB3 interrupts away from cpu0 to another little core).

[stuartiannaylor]

taskset -c 7 7zr b -mmt=1

Irqbalance seems to be doing a good enough job, but reason I installed was near everything seemed to start on cpu0

cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
  9:          0          0          0          0          0          0          0          0     GICv3  25 Level     vgic
 11:          0          0          0          0          0          0          0          0     GICv3  30 Level     kvm guest ptimer
 12:          0          0          0          0          0          0          0          0     GICv3  27 Level     kvm guest vtimer
 13:     118390     151517     101039     106317      51717      22788      55563      43893     GICv3  26 Level     arch_timer
 14:      44070      13991       8033      19607      14779      12781      28246      11313     GICv3 321 Level     rk_timer
 15:          0          0          0          0          0          0          0          0     GICv3  23 Level     arm-pmu
 16:          2          0          0          0         93          0          0          0     GICv3 105 Level     dmc
 17:        108          0          0          0          0          0       1007          0     GICv3 126 Level     fb000000.gpu
 18:          2          0          0          0          0          0          0          0     GICv3 125 Level     fb000000.gpu
 19:        438          0          0          0     551155          0          0          0     GICv3 124 Level     fb000000.gpu
 20:        125          0          0          0          0          0        820          0     GICv3 247 Level     ehci_hcd:usb1
 21:          0          0          0          0          0          0          0          0     GICv3 248 Level     ohci_hcd:usb3
 22:          0          0          0          0          0          0          0          0     GICv3 250 Level     ehci_hcd:usb2
 23:          0          0          0          0          0          0          0          0     GICv3 251 Level     ohci_hcd:usb4
 24:          0          0          0          0          0          0          0          0     GICv3 425 Level     rockchip_usb2phy
 25:          2          0          0          0          0          0          0          0     GICv3 423 Level     rockchip_usb2phy
 26:          1          0          0          0          0          0          1          0     GICv3 424 Level     rockchip_usb2phy
 27:      12136          0          0          0          0          0          0          0     GICv3 349 Level     fd880000.i2c
 28:          0          0          0          0          0          0          0          0     GICv3 142 Level     fdab9000.iommu, fdab0000.npu
 29:          0          0          0          0          0          0          0          0     GICv3 143 Level     fdab9000.iommu, fdab0000.npu
 30:          0          0          0          0          0          0          0          0     GICv3 144 Level     fdab9000.iommu, fdab0000.npu
 31:          0          0          0          0          0          0          0          0     GICv3 151 Level     fdb50400.vdpu
 32:          0          0          0          0          0          0          0          0     GICv3 150 Level     fdb50800.iommu
 33:          0          0          0          0          0          0          0          0     GICv3 146 Level     fdb60f00.iommu, rga3_core0
 34:          0          0          0          0          0          0          0          0     GICv3 147 Level     fdb70f00.iommu, rga3_core1
 35:          0          0          0          0          0          0          0          0     GICv3 148 Level     rga2
 36:          0          0          0          0          0          0          0          0     GICv3 161 Level     fdb90000.jpegd
 37:          0          0          0          0          0          0          0          0     GICv3 162 Level     fdb90480.iommu
 38:          0          0          0          0          0          0          0          0     GICv3 154 Level     fdba0000.jpege-core
 39:          0          0          0          0          0          0          0          0     GICv3 153 Level     fdba0800.iommu
 40:          0          0          0          0          0          0          0          0     GICv3 156 Level     fdba4000.jpege-core
 41:          0          0          0          0          0          0          0          0     GICv3 155 Level     fdba4800.iommu
 42:          0          0          0          0          0          0          0          0     GICv3 158 Level     fdba8000.jpege-core
 43:          0          0          0          0          0          0          0          0     GICv3 157 Level     fdba8800.iommu
 44:          0          0          0          0          0          0          0          0     GICv3 160 Level     fdbac000.jpege-core
 45:          0          0          0          0          0          0          0          0     GICv3 159 Level     fdbac800.iommu
 46:          0          0          0          0          0          0          0          0     GICv3 149 Level     fdbb0800.iommu, fdbb0000.iep
 47:          0          0          0          0          0          0          0          0     GICv3 133 Level     fdbd0000.rkvenc-core
 48:          0          0          0          0          0          0          0          0     GICv3 131 Level     fdbdf000.iommu
 49:          0          0          0          0          0          0          0          0     GICv3 132 Level     fdbdf000.iommu
 50:          0          0          0          0          0          0          0          0     GICv3 136 Level     fdbe0000.rkvenc-core
 51:          0          0          0          0          0          0          0          0     GICv3 134 Level     fdbef000.iommu
 52:          0          0          0          0          0          0          0          0     GICv3 135 Level     fdbef000.iommu
 53:          0          0          0          0          0          0          0          0     GICv3 127 Level     fdc38100.rkvdec-core
 54:          0          0          0          0          0          0          0          0     GICv3 128 Level     fdc38700.iommu
 55:          0          0          0          0          0          0          0          0     GICv3 129 Level     fdc48100.rkvdec-core
 56:          0          0          0          0          0          0          0          0     GICv3 130 Level     fdc48700.iommu
 57:      28446          0          0          0          0          0          0          0     GICv3 188 Level     fdd97e00.iommu, fdd90000.vop
 59:          0          0          0          0          0          0          0          0     GICv3 193 Level     fde50000.dp
 60:          0          0          0          0          0          0          0          0     GICv3 201 Level     fde80000.hdmi
 61:          0          0          0          0          0          0          0          0     GICv3 202 Level     dw-hdmi-qp-cec
 62:          0          0          0          0          0          0          0          0     GICv3 203 Level     fde80000.hdmi
 63:        645          0          0          0          0          0        129          0     GICv3 204 Level     fde80000.hdmi
 64:          0          0          0          0          0          0          0          0     GICv3 392 Level     dw-hdmi-qp-hpd
 65:          0          0          0          0          0          0          0          0     GICv3 285 Level     pcie-sys
 70:          1          0          0          0          0          0          0          0     GICv3 238 Level     fe2b0000.spi
 71:      32762          0          0          0          0          0          0          0     GICv3 235 Level     dw-mci
 72:         86          0          0          0          0          0          0          0     GICv3 236 Level     dw-mci
 73:         11          0          0          0          0          0          0          0     GICv3 237 Level     mmc1
 76:         20          0          0          0          0          0          0          0     GICv3 118 Level     fea10000.dma-controller
 77:          0          0          0          0          0          0          0          0     GICv3 119 Level     fea10000.dma-controller
 78:          0          0          0          0          0          0          0          0     GICv3 120 Level     fea30000.dma-controller
 79:          0          0          0          0          0          0          0          0     GICv3 121 Level     fea30000.dma-controller
 80:         63          0          0          0          0          0          0          0     GICv3 350 Level     fea90000.i2c
 81:        167          0          0          0          0          0          0          0     GICv3 353 Level     feac0000.i2c
 82:     109067          0          0          0          0          0          0          0     GICv3 360 Level     feb20000.spi
 85:          0          0          0          0          0          0          0          0     GICv3 429 Level     rockchip_thermal
 86:          3          0          0          0          0          0          0          0     GICv3 430 Level     fec10000.saradc
 87:         33          0          0          0          0          0          0          0     GICv3 355 Level     fec80000.i2c
 88:        412          0          0          0        398          0          0          0     GICv3 356 Level     fec90000.i2c
 89:          2          0          0          0          0          0         18          0     GICv3 122 Level     fed10000.dma-controller
 90:          0          0          0          0          0          0          0          0     GICv3 123 Level     fed10000.dma-controller
 96:          0          0          0          0          0          0          0          0     GICv3 426 Level     rockchip_usb2phy
 99:          0          0          0          0          0          0          0          0     GICv3 205 Level     fdea0000.hdmi
100:          0          0          0          0          0          0          0          0     GICv3 206 Level     dw-hdmi-qp-cec
101:          0          0          0          0          0          0          0          0     GICv3 207 Level     fdea0000.hdmi
102:          0          0          0          0          0          0          0          0     GICv3 208 Level     fdea0000.hdmi
103:          0          0          0          0          0          0          0          0     GICv3 393 Level     dw-hdmi-qp-hpd
104:          0          0          0          0          0          0          0          0     GICv3 209 Level     rk_hdmirx_cec
105:          0          0          0          0          2          0          0          0     GICv3 468 Level     rk_hdmirx-hdmi
106:          0          0          0          0          0          0          0          0     GICv3 211 Level     rk_hdmirx-dma
107:          1          0          0          0          0          0          0          0     GICv3 295 Level     pcie-sys
117:          0          0          0          0          0          0          0          0     GICv3 455 Edge      debug-signal
118:          0          0          0          0          0          0          0          0     GICv3 365 Level     debug
119:          0          0          0          0          0          0          0          0     GICv3 140 Level     av1d-master
120:          0          0          0          0          0          0          0          0     GICv3 139 Level     irq_cache
122:          0          0          0          0          0          0          0          0  rockchip_gpio_irq   7 Level     rk806
123:          0          0          0          0          0          0          0          0     rk806   0 Edge      rk805_pwrkey_fall
124:          0          0          0          0          0          0          0          0     rk806   1 Edge      rk805_pwrkey_rise
134:          0          0          0          0          0          0          0          0   ITS-MSI   8 Edge      PCIe PME
135:          0         16          0          0          0          0          0          0   ITS-MSI 524288 Edge      nvme0q0
136:          0          0          0          0          0          0          0          0     rk806   7 Level     rk806_vb_low
138:          0          0          0          0          0          0          0          0     GICv3 254 Level     xhci-hcd:usb5
139:          0          0          0          0          0          0          0          0     GICv3 253 Level     xhci-hcd:usb7
140:        105          0          0          0          0          0          0          0   ITS-MSI 524289 Edge      nvme0q1
141:          0          0          4          0          0          0          0          0   ITS-MSI 524290 Edge      nvme0q2
142:          0          0          0        111          0          0          0          0   ITS-MSI 524291 Edge      nvme0q3
143:          0          0          0          0         16          0          0          0   ITS-MSI 524292 Edge      nvme0q4
144:          0          0          0          0          0          0          0          0   ITS-MSI 524293 Edge      nvme0q5
145:          0          0          0          0          0          0          4          0   ITS-MSI 524294 Edge      nvme0q6
146:          0          0          0          0          0          0          0         49   ITS-MSI 524295 Edge      nvme0q7
147:          8          0          0          0          0          0          0          0  rockchip_gpio_irq  12 Level     fsc_interrupt_int_n
148:          0          0          0          0          0          0          0          0  rockchip_gpio_irq   8 Level     hym8563
149:          0          0          0          0          0          0          0          0  rockchip_gpio_irq  22 Edge      rk_hdmirx-5v
150:          0          0          0          0          0          0          0          0  rockchip_gpio_irq  21 Edge      bt_default_wake_host_irq
151:          0          0          0          0          0          0          0          0  rockchip_gpio_irq  29 Level     headset_input
161:          0          0          0          0          0          0          0          0   ITS-MSI 570425352 Edge      PCIe PME
162:          0          0          0       4531          0          0          0          0   ITS-MSI 570949632 Edge      enP4p65s0-0
163:          0          0          0          0          0          0          0          0   ITS-MSI 570949633 Edge      enP4p65s0-1
164:          0          0          0          0          0          0          0          0   ITS-MSI 570949634 Edge      enP4p65s0-2
165:          0          0          0          0          0          0          0          0   ITS-MSI 570949635 Edge      enP4p65s0-3
166:          0          0          0          0          0          0          0          0   ITS-MSI 570949636 Edge      enP4p65s0-4
167:          0          0          0          0          0          0          0          0   ITS-MSI 570949637 Edge      enP4p65s0-5
168:          0          0          0          0          0          0          0          0   ITS-MSI 570949638 Edge      enP4p65s0-6
169:          0          0          0          0          0          0          0          0   ITS-MSI 570949639 Edge      enP4p65s0-7
170:          0          0          0          0          0          0          0          0   ITS-MSI 570949640 Edge      enP4p65s0-8
171:          0          0          0          0          0          0          0          0   ITS-MSI 570949641 Edge      enP4p65s0-9
172:          0          0          0          0          0          0          0          0   ITS-MSI 570949642 Edge      enP4p65s0-10
173:          0          0          0          0          0          0          0          0   ITS-MSI 570949643 Edge      enP4p65s0-11
174:          0          0          0          0          0          0          0          0   ITS-MSI 570949644 Edge      enP4p65s0-12
175:          0          0          0          0          0          0          0          0   ITS-MSI 570949645 Edge      enP4p65s0-13
176:          0          0          0          0          0          0          0          0   ITS-MSI 570949646 Edge      enP4p65s0-14
177:          0          0          0          0          0          0          0          0   ITS-MSI 570949647 Edge      enP4p65s0-15
178:          0       1082          0          8          0          0          0          0   ITS-MSI 570949648 Edge      enP4p65s0-16
179:          0          0          0          0          0          0          0          0   ITS-MSI 570949649 Edge      enP4p65s0-17
180:          0          0          0          0          0         57          0       3357   ITS-MSI 570949650 Edge      enP4p65s0-18
181:          0          0          0          0          0          0          0          0   ITS-MSI 570949651 Edge      enP4p65s0-19
182:          0          0          0          0          0          0          0          0   ITS-MSI 570949652 Edge      enP4p65s0-20
183:          1          0          0          0          0          0          0          0   ITS-MSI 570949653 Edge      enP4p65s0-21
184:          0          0          0          0          0          0          0          0   ITS-MSI 570949654 Edge      enP4p65s0-22
185:          0          0          0          0          0          0          0          0   ITS-MSI 570949655 Edge      enP4p65s0-23
186:          0          0          0          0          0          0          0          0   ITS-MSI 570949656 Edge      enP4p65s0-24
187:          0          0          0          0          0          0          0          0   ITS-MSI 570949657 Edge      enP4p65s0-25
188:          0          0          0          0          0          0          0          0   ITS-MSI 570949658 Edge      enP4p65s0-26
189:          0          0          0          0          0          0          0          0   ITS-MSI 570949659 Edge      enP4p65s0-27
190:          0          0          0          0          0          0          0          0   ITS-MSI 570949660 Edge      enP4p65s0-28
191:          0          0          0          0          0          0          0          0   ITS-MSI 570949661 Edge      enP4p65s0-29
192:          0          0          0          0          0          0          0          0   ITS-MSI 570949662 Edge      enP4p65s0-30
193:          0          0          0          0          0          0          0          0   ITS-MSI 570949663 Edge      enP4p65s0-31
IPI0:    302340     353172     371588     329987       1986      16243       7952       8500       Rescheduling interrupts
IPI1:    207652     133181     190279     172301     146941     141856     288433     276009       Function call interrupts
IPI2:         0          0          0          0          0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0          0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:      3550       2638       1772       3404       1430       2970       4601       3301       Timer broadcast interrupts
IPI5:     35307      15268      11870      11755      25266      14706      20207      18404       IRQ work interrupts
IPI6:         0          0          0          0          0          0          0          0       CPU wake-up interrupts
Err:          0

Seems fairly balanced for the irqs used now and don’t have to think of static requirements.
The default scheduling is just me musing how badly the ondemand scheduler does in regards to the running Glmark2 than running the performance scheduler.
Its ok if it hits a solid wall of load where the demand threshold is hit on a smaller it swaps out to big.
If you get varying load across cores then that sampling of starting on small at low frequency and ramping up then jumping to big and doing the same.
I was just sat here thinking why can it not start big and ramp down or if there was an implementation and how do you swap from the default CFS scheduler (we are using that?) to either
https://www.kernel.org/doc/html/latest/scheduler/sched-capacity.html
or
https://www.kernel.org/doc/html/latest/scheduler/sched-energy.html

Purely out of interest and wondered if you guys knew?

[edit]
The more I look and think about it, the more I don’t get big.Liitle especially when they are linked into profiles where all 4 cpu’s ramp up and down together, with 2 groups of big that ramp up together.
You can do something really weird and leave profile 0 as is but lower the /ondemand/up_threshold of the x2 big to some silly low value such as 3%.
Then when you run something like glmark2 you get a score near what a performance governor would give but also the cpu’s ramp down when not in use.
Also even though there is practically no load on profile 0 all 4 cpu’s are often ramping up to the top end opp frequencies, which prob makes sense on a phone but as a mains powered SBC it just confuses me.

[solaris3308]

Thanks! It’s very useful!!!
Can you provid your info of dmesg | grep cpu.cpu
Something really confuse me,for example:
3.5.1 Modify Voltage According to Leakage
Background: we find that the Vmin of small leakage chips is larger than big leakage chips from test, so we can
reduce the voltage for big leakage chips to reduce power consumption and improve performance.

[solaris3308]

Thanks very much!Slightly diff from rk3588s.But I will interpret it

[solaris3308]

I have some info of your board’s pvtm but what about pvtm?
Can you provid dmesg | grep cpu.cpu

[willy]

It just depends on chip generations. Older ones were measuring leakage in the fab and using that to adjust voltage. Newer ones use PVTM to measure it at boot time.

[willy]

I have some info of your board’s pvtm but what about pvtm?
Can you provid dmesg | grep cpu.cpu

Sure!

~$ dmesg|grep cpu.cpu
[    4.706906] cpu cpu0: leakage=14
[    4.706927] cpu cpu0: Looking up cpu-supply from device tree
[    4.708386] cpu cpu0: pvtm=1481
[    4.708476] cpu cpu0: pvtm-volt-sel=3
[    4.708503] cpu cpu0: Looking up cpu-supply from device tree
[    4.708619] cpu cpu0: Looking up mem-supply from device tree
[    4.708955] cpu cpu4: leakage=10
[    4.708973] cpu cpu4: Looking up cpu-supply from device tree
[    4.715458] cpu cpu4: pvtm=1719
[    4.719393] cpu cpu4: pvtm-volt-sel=5
[    4.719418] cpu cpu4: Looking up cpu-supply from device tree
[    4.719914] cpu cpu4: Looking up mem-supply from device tree
[    4.720625] cpu cpu6: leakage=10
[    4.720641] cpu cpu6: Looking up cpu-supply from device tree
[    4.727141] cpu cpu6: pvtm=1740
[    4.731097] cpu cpu6: pvtm-volt-sel=5
[    4.731120] cpu cpu6: Looking up cpu-supply from device tree
[    4.731611] cpu cpu6: Looking up mem-supply from device tree
[    4.732680] cpu cpu0: avs=0
[    4.733371] cpu cpu4: avs=0
[    4.734058] cpu cpu6: avs=0

One of the two big clusters is close to the boundary and randomly switches between two volt-sel values (I believe it’s the 2nd one which oscillates between 5 and 6 but it might also be the 1st one between 4 and 5, I don’t remember).

At least with the doc above I could improve the measurement time in the DT to make it more stable.

[tkaiser]

One of the two big clusters is close to the boundary and randomly switches between two volt-sel values (I believe it’s the 2nd one which oscillates between 5 and 6

Yes, the 2nd one according to start of the debug party. If the pvtm value is between 1744 and 1776 --> pvtm-volt-sel=6, with 1743 --> pvtm-volt-sel=5

[tkaiser]

The more I look and think about it, the more I don’t get big.Liitle especially when they are linked into profiles where all 4 cpu’s ramp up and down together, with 2 groups of big that ramp up together.

Quick test with Rock 5B (with a ‘good’ RK3588 on it: pvtm-volt-sel 5/7/7), RPi USB-C power brick, only GbE networking and Netio 4KF Powerbox measuring idle consumption:

• DMC governor and all cpufreq policies set to powersave: 1280mW
• just switching policy0 to performance (the four A55 idling now at ~1820 MHz instead of ~400 MHz): 1350mW
• all cpufreq policies set to performance as such all CPU cores idling at highest cpufreq OPP: 1500mW
• now also DMC governor set to performance: 2100mW

That’s a whopping 70mW or 17.5mW per A55 depending on whether it is doing nothing at ~400 MHz or ~1800 MHz. With the A76 the difference it makes whether they’re idling at ~400 MHz or ~2350 MHz is 150mW or 37.5mW per core. So as long as they aren’t doing anything the consumption difference with them staying on the lowest vs. highest cpufreq OPP is negligible. Now we would need another Rock 5B Rev 1.3 with a ‘weak’ RK3588 on it ( pvtm-volt-sel=2) to compare with same equipment and methodology how much of a difference this makes.

Memory is a different thing: the difference between 528 MHz and 2112 MHz is a massive 600mW compared to CPU clockspeeds. As such the DMC governor and the up_treshold setting really matter.

The small consumption difference with ARM cores doing nothing regardless of cpufreq (sitting on an ‘Wait for Interrupt’ instruction which was already there with ARMv7) might have driven the decision on ARM to have one cpufreq policy per cluster (which you call profile?) unlike on other platforms like x86_64 where each core seems to behave independently (no idea which influence the cpufreq driver on modern x86_64 implementations really has or whether it’s just reporting what something else inside an Intel/AMD CPU decides).

And I guess the decision to split the four A76 into two clusters was mostly motivated by PVTM as to be able to drive them with different supply voltages and not so much keep idle A76 on lower cpufreq OPP.

[stuartiannaylor]

Yeah I don’t know enough but policy (apols). Arm wise I presume its setup in a similar way to the newer triple-cluster architecture and maybe one policy can push further in Mhz than the other in a similar way the core count in Intel speedstep may do? I guess really its a verybig.biG.Little config really but just uses the same for verybig & biG.

This time I am totally asking you as expecting you have measured hard data of various governors.
I have only got one of those really crappy usb-c meters and haven’t bothered to get a readout.

Using my crappy usb-c meter and a 5v pi supply as it doesn’t handle PD its telling me 3.3watt approx on idle.
Running Glmark2 as an example with no governor or taskset changes ups wattage to approx avg 4.6 watts sometimes above 6 watt but often below and prob a decent guesstimate avg and ends up with a lacklustre glmark2 Score of 582.
Doing a weird and wonderful of setting only echo 2 > sys/devices/system/cpu/cpufreq/policy4/6/ondemand/up_threshold with another horrid guesstimate wattage of approx avg 5.3 watt end seems to have the same occasional max outs of just over 6 watt with a not brilliant but ok glmark2 Score of 920.
After running it drops to that 3.3watt approx idle again.

echo performance | sudo tee /sys/bus/cpu/devices/cpu[046]/cpufreq/scaling_govenor /sys/class/devfreq/dmc/governor
Only seems to up wattage to a guesstimate of 5.8 watts whilst idle rises to 3.9 watt whilst returning a glmark2 score of 1014

Lols as yeah it makes no sense to me, but really need a logging meter to get a true avg than guesstimating it Actually just noticed it does have a resettable mWh log doh.

[solaris3308]

I encount exactly problem as you.when I set voltage bigger than 1000mv,Frequency is locked to 400mhz.
[ 476.994512] vdd_cpu_big0_s0: Restricting voltage, 1025000-1000000uV
[ 476.994567] vdd_cpu_big0_s0: Restricting voltage, 1025000-1000000uV
[ 476.994591] cpu cpu4: rockchip_cpufreq_set_volt: failed to set voltage (1025000 1025000 1025000 uV): -22
[ 476.994613] cpufreq: __target_index: Failed to change cpu frequency: -22

opp table:
opp-2400000000 {
opp-supported-hw = <0xfd 0xffff>;
opp-hz = <0x00 0x8f0d1800>;
opp-microvolt = <0xFA3E8 0xFA3E8 0xFA3E8 0xFA3E8 0xFA3E8 0xFA3E8>;
clock-latency-ns = <0x9c40>;
};
Do you have find a way to solve it?

[solaris3308]

How and why!
I try to set higher voltage,but got this:
[ 1070.744651] vdd_cpu_big0_s0: Restricting voltage, 1025000-1000000uV
[ 1070.744701] vdd_cpu_big0_s0: Restricting voltage, 1025000-1000000uV
[ 1070.744726] cpu cpu4: rockchip_cpufreq_set_volt: failed to set voltage (1025000 1025000 1025000 uV): -22
[ 1070.744746] cpufreq: __target_index: Failed to change cpu frequency: -22

[tkaiser]

opp-microvolt = <0xFA3E8 0xFA3E8 0xFA3E8 0xFA3E8 0xFA3E8 0xFA3E8>;

Six times same value. Now compare to what @amazingfate and me were posting as opp-microvolt values.

[tkaiser]

Not entirely sure what this Glmark2 testing is about (as @icecream95 already explained it’s not testing what people expect).

In general I would always test with every governor set to powersave and then with performance again to get worst and best case. Afterwards with a somewhat ‘smart’ powermeter an optimization process could happen to get a good compromise between performance (when needed) and consumption. Only then the other governors and further settings like ondemand/up_threshold (DMC) or ondemand/io_is_busy (CPU) will be explored to find a good balance (as we’ve seen ondemand/up_threshold=40 isn’t a good choice but nobody cares, same with io_is_busy).

But without power measurements that generate graphs this is a process most probably not worth the time and efforts.

BTW there’s also /sys/devices/platform/fb000000.gpu/devfreq/fb000000.gpu/ defaulting to simple_ondemand and as such dynamically clocking the GPU between 200 and 1000 MHz. Is it worth measuring ‘GL performance’ without taking this into account?

[solaris3308]

You means I only need to set 3 and 6 postion values?
Like this? :
opp-2400000000 { //org 2400000000
opp-supported-hw = <0xff 0xffff>;
opp-hz = <0x00 0x8f0d1800>; //org <0x00 0x8f0d1800>
opp-microvolt = <0xf4240 0xf4240 0xFA3E8 0xf4240 0xf4240 0xFA3E8>;
clock-latency-ns = <0x9c40>;
};
When I done.No warning appear again,but volts has no change,still 1000mv
root@firefly2:~# cat /sys/kernel/debug/regulator/regulator_summary | grep big
vdd_cpu_big0_s0 1 3 0 normal 1000mV 0mA 550mV 1050mV
vdd_cpu_big0_s0 0 0mA 0mV 0mV
vdd_cpu_big1_s0 1 3 0 normal 1000mV 0mA 550mV 1050mV
vdd_cpu_big1_s0 0 0mA 0mV 0mV