Well, without adding coherent_pool=2M to extlinux.conf the board does not even boot when the USB3 attached SSDs are connected. This is with an mdraid0 (so zero practical relevance for future Rock 5B users):
root@rock-5b:/mnt/md0# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid0 sdb[0] sda[1]
234307584 blocks super 1.2 512k chunks
unused devices: <none>Its only recent last year sometime just can not remember which revision it went in.
I can not even remember why I was reading about it, but prob due to being on 5.10
Purely mentioned that probably they should not be included in the Radxa image but definitely documented as tweaks and fixes.
Some consumption data…
I’m measuring with a NetIO PowerBox 4KF in a rather time consuming process that means ‘at the wall’ with charger included with averaged idle values over 4 minutes. I tested with 2 chargers so far:
When checking with sensors tcpm_source_psy_4_0022-i2c-4-22 this looks like this with the RPi part
in0: 5.00 V (min = +5.00 V, max = +5.00 V)
curr1: 3.00 A (max = +3.00 A)
and like this with the Apple part:
in0: 9.00 V (min = +9.00 V, max = +9.00 V)
curr1: 3.00 A (max = +3.00 A)
27W could become a problem with lots of peripherals but @hipboi said the whole USB PD negotiation thing should be configurable via DT or maybe even at runtime.
Measured with RPi power brick and 2.5GbE network connection:
That means the little fan on my dev sample consumes 700mW that can be substracted from numbers (given Radxa will provide a quality metal case that dissipates heat out of the enclosure so no fan needed)
Now with the Apple Charger:
The Apple charger obviously needs a little more juice for itself (+ ~150mW compared to the RPi power brick) and powering the 2.5GbE PHY vs. only Gigabit Ethernet speeds results in 300mW consumption difference.
echo default >/sys/module/pcie_aspm/parameters/policy --> idle consumption: 3380mWSwitching from powersave to default with PCIe powermanagement (needed to get advertised network speeds) adds ~100mW to idle consumption.
As such I propose the following changes in all of Radxa’s OS images (preferably ASAP but at least before review samples are sent out to those YT clowns like ExplainingComputers, ETA Prime and so on):
/sys/module/pcie_aspm/parameters/policy to default (w/o network RX performance is ruined)coherent_pool=2M to extlinux.conf (w/o most probably ‘UAS hassles’)ondemand cpufreq governor with io_is_busy and friends (w/o storage performance sucks)Would be great if you put it in the wiki.
In my tests on the USB cable, I measured 1.6W in idle with no fan, and 2.6W with it. At higher loads under 5V, the difference was a bit less but the fan wouldn’t spin as fast. When powered under 12V however, the fan speed remains stable but I can’t monitor the current. This seems consistent with your measurements, and you can count on 500-800mW for your PSU 
Yeah, too bad I don’t have the Apple 30W charger around which was the most efficient charger I ever had in my hands.
BTW: I removed the fansink in the meantime and tested without any cooling at an ambient temp of ~26°C (explanation of results/numbers):
| Cooling | Clockspeed | 7-zip | AES-256 (16 KB) | memcpy | memset | kH/s |
|---|---|---|---|---|---|---|
| with fansink | 2350/1830 | 16450 | 1337540 | 10830 | 29220 | 25.31 |
| w/o any cooling | 2010/1070 | 15290 | 1316480 | 10890 | 28430 | 22.14 |
This is just impressive since only slight performance decreases occured even with no cooling at all. So there’s definitive hope that Radxa will come up with a metal enclosure passively dissipating the heat away and still providing high sustained performance.
When running this demanding stuff for a longer time it looks like this:
Time big.LITTLE load %cpu %sys %usr %nice %io %irq Temp
11:17:29: 2208/ 408MHz 7.63 89% 0% 89% 0% 0% 0% 84.1°C
11:17:34: 2016/1416MHz 7.82 76% 1% 74% 0% 0% 0% 85.0°C
11:17:39: 2016/1416MHz 7.84 100% 0% 99% 0% 0% 0% 85.0°C
11:17:44: 2208/1416MHz 7.29 62% 0% 61% 0% 0% 0% 84.1°C
11:17:49: 2016/1416MHz 7.35 94% 1% 92% 0% 0% 0% 84.1°C
11:17:54: 2016/1416MHz 7.40 100% 0% 99% 0% 0% 0% 85.0°C
11:17:59: 2016/1200MHz 6.89 54% 1% 53% 0% 0% 0% 84.1°C
11:18:04: 1800/1416MHz 7.14 99% 2% 97% 0% 0% 0% 85.0°C
11:18:09: 2016/1416MHz 7.29 97% 2% 95% 0% 0% 0% 85.0°C
11:18:15: 2208/ 408MHz 7.34 96% 0% 96% 0% 0% 0% 84.1°C
11:18:20: 2016/1416MHz 7.56 75% 1% 74% 0% 0% 0% 85.0°C
11:18:25: 2016/1416MHz 7.59 95% 0% 94% 0% 0% 0% 85.0°C
11:18:30: 2016/1416MHz 7.30 67% 0% 67% 0% 0% 0% 83.2°C
11:18:35: 2016/1416MHz 7.36 96% 1% 95% 0% 0% 0% 85.0°C
11:18:40: 2208/ 408MHz 7.41 86% 0% 86% 0% 0% 0% 84.1°C
11:18:45: 2016/1416MHz 7.62 76% 2% 74% 0% 0% 0% 85.0°C
11:18:50: 2016/1416MHz 7.65 96% 1% 95% 0% 0% 0% 85.0°C
11:18:55: 2208/ 408MHz 7.68 78% 0% 78% 0% 0% 0% 84.1°C
11:19:00: 2016/1200MHz 7.78 76% 2% 74% 0% 0% 0% 84.1°C
11:19:05: 2016/1200MHz 7.88 99% 1% 97% 0% 0% 0% 84.1°C
11:19:10: 2016/1416MHz 7.89 95% 1% 94% 0% 0% 0% 85.0°C
11:19:15: 2208/ 408MHz 7.90 76% 0% 76% 0% 0% 0% 84.1°C
11:19:20: 2016/1416MHz 8.15 90% 1% 89% 0% 0% 0% 85.0°C
11:19:25: 2016/1416MHz 8.14 100% 0% 99% 0% 0% 0% 85.0°C
11:19:31: 2016/1200MHz 7.81 67% 1% 66% 0% 0% 0% 84.1°C
11:19:36: 1608/1416MHz 7.82 93% 0% 93% 0% 0% 0% 84.1°C
11:19:41: 2208/1608MHz 7.67 70% 0% 69% 0% 0% 0% 83.2°C
11:19:46: 2016/1200MHz 7.86 87% 1% 85% 0% 0% 0% 84.1°C
11:19:51: 2016/1416MHz 7.87 96% 0% 95% 0% 0% 0% 85.0°C
11:19:56: 2208/1608MHz 7.56 59% 0% 59% 0% 0% 0% 82.2°C
11:20:01: 2016/1200MHz 7.68 94% 2% 91% 0% 0% 0% 84.1°C
11:20:06: 2208/1200MHz 7.86 99% 1% 97% 0% 0% 0% 84.1°C
11:20:11: 2016/1416MHz 7.87 96% 1% 94% 0% 0% 0% 85.0°C
11:20:16: 2016/1416MHz 7.56 72% 0% 71% 0% 0% 0% 85.0°C
11:20:21: 2016/1416MHz 7.84 97% 0% 96% 0% 0% 0% 85.0°C
11:20:26: 2208/ 408MHz 7.85 84% 0% 84% 0% 0% 0% 84.1°C
11:20:31: 2016/1416MHz 7.94 81% 1% 80% 0% 0% 0% 85.0°C
11:20:36: 2016/1416MHz 7.95 100% 0% 99% 0% 0% 0% 85.0°C
11:20:41: 2016/1416MHz 7.63 62% 1% 61% 0% 0% 0% 85.0°C
11:20:46: 2016/1416MHz 7.66 93% 1% 91% 0% 0% 0% 84.1°C
11:20:51: 2208/ 408MHz 7.69 93% 0% 93% 0% 0% 0% 84.1°C
11:20:57: 2016/1200MHz 7.39 61% 1% 59% 0% 0% 0% 84.1°C
11:21:02: 2016/1008MHz 7.68 99% 1% 97% 0% 0% 0% 85.0°C
11:21:07: 2016/1416MHz 7.79 92% 2% 89% 0% 0% 0% 85.0°C
11:21:12: 2208/ 408MHz 7.81 93% 0% 92% 0% 0% 0% 84.1°C
11:21:17: 2016/1200MHz 7.50 77% 1% 75% 0% 0% 0% 84.1°C
11:21:22: 2016/1416MHz 7.54 98% 0% 97% 0% 0% 0% 85.0°C
11:21:27: 2016/1416MHz 7.58 67% 0% 67% 0% 0% 0% 85.0°C
11:21:32: 2016/1416MHz 7.61 96% 0% 95% 0% 0% 0% 85.0°C
11:21:37: 2208/ 408MHz 7.64 83% 0% 83% 0% 0% 0% 84.1°C
11:21:42: 2016/1200MHz 7.35 79% 1% 77% 0% 0% 0% 84.1°C
11:21:47: 2016/1416MHz 7.40 96% 0% 95% 0% 0% 0% 85.0°C
11:21:52: 2208/ 408MHz 7.45 74% 0% 74% 0% 0% 0% 84.1°C
11:21:58: 1800/1200MHz 6.93 79% 2% 77% 0% 0% 0% 85.0°C
11:22:03: 1800/1200MHz 7.10 99% 1% 97% 0% 0% 0% 85.0°C
11:22:08: 2016/1416MHz 7.41 94% 2% 92% 0% 0% 0% 85.0°C
11:22:13: 2016/1416MHz 7.46 72% 0% 72% 0% 0% 0% 84.1°C
11:22:18: 2016/1416MHz 7.66 97% 0% 96% 0% 0% 0% 85.0°C
11:22:23: 2208/ 408MHz 7.69 95% 0% 95% 0% 0% 0% 84.1°C
11:22:28: 1800/1416MHz 7.39 70% 1% 69% 0% 0% 0% 85.0°C
11:22:33: 1800/1416MHz 7.44 100% 0% 99% 0% 0% 0% 84.1°C
11:22:38: 2016/1416MHz 7.99 65% 0% 65% 0% 0% 0% 84.1°C
11:22:43: 2016/1416MHz 7.99 91% 1% 89% 0% 0% 0% 84.1°C
11:22:48: 2016/1416MHz 7.99 100% 0% 99% 0% 0% 0% 85.0°C
11:22:53: 2016/1200MHz 8.23 53% 0% 52% 0% 0% 0% 84.1°C
11:22:58: 2016/1200MHz 8.38 99% 1% 97% 0% 0% 0% 84.1°C
11:23:03: 2208/1608MHz 7.87 93% 1% 91% 0% 0% 0% 83.2°C
11:23:08: 2016/1416MHz 7.88 95% 0% 94% 0% 0% 0% 85.0°C
11:23:13: 2016/1416MHz 8.05 72% 0% 71% 0% 0% 0% 85.0°C
11:23:18: 2016/1416MHz 8.04 96% 0% 95% 0% 0% 0% 85.0°C
11:23:24: 2208/ 408MHz 7.72 80% 0% 80% 0% 0% 0% 84.1°C
11:23:29: 1800/1608MHz 7.98 84% 1% 82% 0% 0% 0% 85.0°C
11:23:34: 1800/1416MHz 7.98 100% 0% 99% 0% 0% 0% 84.1°C
11:23:39: 2016/1200MHz 7.98 65% 1% 63% 0% 0% 0% 84.1°C
11:23:44: 1800/1416MHz 8.15 97% 1% 95% 0% 0% 0% 84.1°C
Or in other words: thermal treshold is 85°C and throttling works. But we need to keep in mind that RK3588 is a lot more than just 8 ARM CPU cores. If GPU, I/O, NPU and so on are fully loaded the SoC needs a lot more thermal headroom.
Ok, we will fix this for the MP version.
Along with the heatsink skewness I hope?)
Yes I noticed as well that it’s extremely efficient. Last evening I replaced the fan+heatsink with a larger heatsink attached using thermal tape, a solution that I used with success on the much hotter RK3288. I’ve ran stress-ng for 20 minutes, the SoC temperature reached 76°C stable after 10-12 minutes, for about 53 on the heat sink, which is very reasonable:
At least that shows some good stability on the power path and the temperature.
Also Ifound a bit of info on this PVTM thing that gives me a varying top frequency at boot, it’s a concept I never heard of which tries to estimate the “quality” of the silicon based on a free-running clock, the temperature and the voltage, to estimate the max frequency that will be supported. Mine has a random frequency at boot because the measure oscillates between 1742 and 1744, the latter being the next offset
I find the concept interesting, except that it should then not remove the OPP from the list but just condition them by voltage/temp as well. I.e. if you manage to cool your board or to slightly overvolt it (or both), no reason to deal with the constraint. Anyway I think that just fiddling with the DTB to adjust the threshold will do the job, so for now I don’t really care. And possibly that we’ll be able to enable them when boost will be enabled, that’s something to check as well.
Last point (I’m grouping all my comments due to the limit of 3/day), I tried to run a network test at 2.5GbE but it ended up being at 1G because the other NIC was an intel X540AT2, and that one only supports 1 or 10G but not 2.5… I ordered a 1/2.5/5/10G one to run more tests. However during this time frame I could notice that queues were totally uneven on Rx (a single CPU was used), so there’s possibly room for improvement using ethtool. Something else to keep on the check list!
Yes, yesterday I read chapter 17 in RK3588 TRM-Part2 (other readers beware: ~3700 pages PDF 56 MB in size).
The reason to operate the board today w/o fansink was also to check whether ‘cold boot’ with ‘hot board’ results in different values but I’m still at
cpu cpu0: pvtm=1528
cpu cpu0: pvtm-volt-sel=5
cpu cpu4: pvtm=1784
cpu cpu4: pvtm-volt-sel=7
cpu cpu6: pvtm=1782
cpu cpu6: pvtm-volt-sel=7
which is almost exactly the same (except cpu4) than yesterday with fansink an an idle temp of 30.5°C when booting:
[ 3.117399] cpu cpu0: pvtm=1528
[ 3.117491] cpu cpu0: pvtm-volt-sel=5
[ 3.124529] cpu cpu4: pvtm=1785
[ 3.128495] cpu cpu4: pvtm-volt-sel=7
[ 3.136234] cpu cpu6: pvtm=1782
[ 3.140173] cpu cpu6: pvtm-volt-sel=7
As for the network tunables I’m waiting for your insights
Currently searching a fast enough NVMe SSD…
Nope:
root@rock-5b:/home/rock# dpkg -l | grep -i mali
ii gpg 2.2.19-3ubuntu2 arm64 GNU Privacy Guard -- minimalist public key operations
ii libmnl0:arm64 1.0.4-2 arm64 minimalistic Netlink communication library
In case you want me to look up other things please feel free to ask!
Not much to ask knowing there is no mali installed, but let’s try:
sudo find /usr -name mali_csffw.bin
sudo find /usr -name libmali.so
sudo find /usr -name librga.soroot@rock-5b:/home/rock# find /usr -name mali_csffw.bin
root@rock-5b:/home/rock# find /usr -name libmali.so
root@rock-5b:/home/rock# find /usr -name librga.so
/usr/lib/aarch64-linux-gnu/librga.so
root@rock-5b:/home/rock# ldd /usr/lib/aarch64-linux-gnu/librga.so
linux-vdso.so.1 (0x0000007f971bc000)
libdrm.so.2 => /lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f9713e000)
libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f96f59000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f96eae000)
libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f96e8a000)
libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f96e59000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f96ce6000)
/lib/ld-linux-aarch64.so.1 (0x0000007f9718c000)I have not read anything about a Mali blob for the G610 and Mesa slowed down with Valhall as focussed on the Apple M1 GPU.
I think some G57 updates went into the last Kernel release (5.18) but not sure if that will work at all with the G610 even though the same architecture family and don’t know what the state of play the G57 is as haven’t got a MediaTek MT8192 that Alisia of collabora is doing testing on.
Mesa was romping ahead with Mali then Apple Arm silicon occurred, its not stopped but definitely progressing at a slower pace than it was before.
The blobs seem to be:
libmali-valhall-g610-g6p0-gbm.so (GBM)
libmali-valhall-g610-g6p0-wayland.so (WAYLAND)
libmali-valhall-g610-g6p0-x11.so (X11)Haven’t seen anything out in the wild yet or a Linux bench all seems to be Android but maybe.
Let’s be honest, I’m sure it’s certainly not easy. Yes it would be nice, but if not possible, be it. Maybe the arrangement at 45 degrees like on the R6S above can help get the SoC closer to the center, if possible at all. I’m noting that they have less distance between the SoC and the DRAM chips, I don’t know if it’s a matter of number of copper layers, different pinouts or routing constraints. And the fact that the SoC heats very little makes the arrangement even less critical here.
BTW @hipboi I tried a number of different heat sinks, including various from north bridges and south bridges for intel and AMD, but couldn’t find one that matches this exact step (some were 5mm too long). I found some where the plots were reversed, i.e. top-left and bottom-right. I thought that maybe such variations could be enough to solve the problem, if the spacing doesn’t correspond that much to an easily findable heat sink model and is not that important in the end.