ROCK 5B Debug Party Invitation

icecream95 · October 13, 2022, 2:08am

Can some people using Radxa Debian go to https://webglsamples.org/persistence/persistence.html in Chromium and drag the persistence slider at the top all the way to the right?

What happens?

Hangs or reboots
Works fine

0 voters

(I’m interested in the default configuration of XFCE with GPU-accelerated Chromium using libmali. “Hang” is talking about a system hang, the webpage is supposed to stop visually updating when persistence reaches 1.0. If it is still working fine at that point, try dragging the slider back down to see if the system hangs then.)

icecream95 · October 14, 2022, 7:47am

Currently I think that the issue reproduced by WebGL “persistence” is a hardware issue which only affects a few boards—I think there have been at least four people seeing such instability, myself included.

When checking kernel logs over a serial console, commonly the kernel breaks to such a degree that there is silence, but in many other cases an “Access fault” shows up, which my modified kernel prints like this:

<3>[ 1710.308816][ T1673] mali fb000000.gpu: Unhandled Page fault in AS2 at VA 0x0000005FFBD78000
<3>[ 1710.308816][ T1673] Reason: Access fault
<3>[ 1710.308816][ T1673] raw fault status: 0x4D003DB
<3>[ 1710.308816][ T1673] exception type 0xDB: UNKNOWN
<3>[ 1710.308816][ T1673] access type 0x3: WRITE
<3>[ 1710.308816][ T1673] source id 0x4D0
<3>[ 1710.308816][ T1673] pid: 2216
<4>[ 1710.317359][ T1673] mali fb000000.gpu: Access flag unexpectedly set

The actual address for the fault varies, but often it is inside a render target, depth buffer, or tiler heap. This makes me think that it happens at the tail end of a fragment job. Both READ and WRITE access types have been obvserved.

So what is an access fault? For AArch64 MMU Page descriptors, if the “Access Flag” (bit 10) is not set then the MMU will automatically set it, so that the kernel can check if a page has been accessed by userspace or not. However, if it is unset, and hardware is not configured to update it, then it will raise an Access fault. I assume that the GPU MMU always raises the fault when the bit is unset.

I have modified my kernel to perform a dump of GPU MMU tables after receiving an Access fault, which I can then extract using memory dump commands in kdb. From this, I’ve noticed that the ATE (kbase's term for a Page descriptor) for the faulty address looks completely normal—the corruption does not exist in RAM! The addresses where it happens are not recently mapped memory region either.

This means that the hardware issue happens while reading a cacheline of ATEs from RAM into the GPU, or possibly inside the GPU. I think that the corruption has the potential to change a number of bits in the ATE—I have noticed a lot of kernel .text corruption, so some of the bits of the PFN (i.e. phsyical address) must get changed as well.

I think the reason for the Access faults is just because of how common it will be—for random corruption, 50% of ATEs will have the Access Flag unset and so raise a fault. Unfortunately, the other 50% can end up corrupting RAM.

icecream95 · October 14, 2022, 11:39pm

I suspect that the corruption can affect any memory transfer by the GPU, because I often see artefacts looking like this when the hang happens, sometimes in multiple regions of the display:

Note that the region of corruption is about 80x50 pixels, so it covers multiple 32x32 GPU tiles.

avaf · October 15, 2022, 12:05am

FYI, the WebGL “persistence” runs fine with my Debian 10.Note that is not radxa img.
While on my Debian 11 (same board) had some glitches when my eMMC was still alive.

avaf · October 15, 2022, 12:27am

The drawback of using Debian 10 is that glmark2-es2 is extremely low, i have 551 (score) while in Debian 11 it was something like 750 if i recall correctly. In a post above i have glmark2-es2 Score with hdmi-rx which was higher than hdmi-1 or hdmi-2.

You have to trade speed for stability.
I would rule out a hw problem. IMHO it is a sw issue.

icecream95 · October 15, 2022, 2:40am

Unless the kernel is misconfiguring hardware, I’m really unsure how corrupted memory transactions can be a software problem, especially when it happens with two completely different userspace driver stacks.

(In terms of glmark2-es2 scores, I have gotten as high as 2958 with Wayland compositors, it is less a benchmark of performance than of the per-frame compositor overhead.)

Here is a “screenshot” of a corrupted FBO rescued using the kernel debugger, with a 32x32 grid overlaid:

corruption-crop

So although the MMU table corruption does not appear in RAM, this does, suggesting that the problem can happen with both read and write transfers. However, it could just be that the corruption happens when reading the source texture for the blit into GPU memory, so I am not absolutely sure about this yet.

I wonder if the problem only happens with one of the two DRAM chips, that could explain why (in this case) only the green and alpha bytes are affected, and red and blue are correct.

I hypothesise that the problem can only result in bits being set to zero when they shouldn’t be, this would explain why the kernel (which lives at the bottom of phsyical memory) is so often corrupted.

icecream95 · October 15, 2022, 9:52am

I think the problem is just that the RAM isn’t stable at 2112MHz. If I restrict the maximum frequency in /sys/class/devfreq/dmc/max_freq then everything seems to be much more reliable.

Is it possible to get DDR frequencies between 1560MHz and 2112MHz for boards that can’t reliably handle the latter, or can RAM stability be improved otherwise?

tinymembench shows a drop from 10 GB/s to 8 GB/s copy and 28 GB/s fill to 23 GB/s fill when reducing the frequency. I don’t know how much that affects real applications, but it would be nice if I didn’t have to lose all of that performance.

avaf · October 15, 2022, 3:40pm

Could be. the kernel i am using for Debian 10 has no optimizations. RAM never runs at 2112 MHz, GPU is at 300MHz. It is almost an Android kernel.

What is your glmark2-es2 score with X11?

New kernel (debug) and Monitor with such a low score:

rock@rock5b:~$ sudo ./mon.sh 
 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1200    2256     408     528       0     200     200
   1200     408     408     528       0     200     200
   1200     408     408    1068       0    1000     200
   1200    1608     408    1068       0    1000     200
   1200     408     408    1068       0    1000     200
   1416    1416     408    1068       0    1000     200
   1200    1416     816     528       0    1000     200
   1608     600    1800    1068       0    1000     200
   1200     408    1200     528       0    1000     200
   1416     408     408     528       0    1000     200
   1200     408    1416    1068       0    1000     200
   1200    1416     408    1068       0    1000     200
   1416     408    1608     528       0    1000     200
   1200     408    1416    1068       0    1000     200
   1200     408    1200     528       0    1000     200
   1416    1608    1416     528       0    1000     200
   1200    1608    1416     528       0    1000     200
    816    1416     408    1068       0    1000     200
   1200    1416     408     528       0    1000     200
   1200    1608     408    1068       0    1000     200
   1200     600    1416    1068       0    1000     200
   1416    1416    1200    1068       0    1000     200
   1200    1416     600     528       0     300     200
   1200    1200    1608     528       0    1000     200

 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1200    1200    1200     528       0    1000     200
   1200    1416     816    1068       0    1000     200
   1200    1416     408     528       0    1000     200
   1416    1416     816     528       0    1000     200
   1200    1416     408     528       0    1000     200
   1200    1416     408     528       0    1000     200
   1200    1416     408     528       0    1000     200
   1416     408    1416     528       0    1000     200
   1200     408     408    2112       0    1000     200
   1008     816    1608    1068       0    1000     200
   1416     408     408    1068       0    1000     200
   1200     408    1416    1068       0    1000     200
   1008     408    1200     528       0     800     200
   1200    1416     408     528       0    1000     200
   1200    1608     408     528       0     800     200
    816    1608    1608     528       0     800     200
   1200     408     408     528       0    1000     200
   1200    1416    1608     528       0    1000     200
    816     816    1416     528       0    1000     200
   1416     600    2208    1068       0    1000     200
   1800    1416    1416     528       0     300     200
   1416    1416     408     528       0    1000     200
   1416     408     408    1068       0    1000     200

Update:

Here are my complete experiments that rule out the HW problem. (at least on my Developer Board).

1 - Running glmark2-es2 with performance governor, DRAM 2112 MHz, GPU 1000 MHz.

rock@rock5b:~$ glmark2-es2 
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
=======================================================
    glmark2 2021.02
=======================================================
    OpenGL Information
    GL_VENDOR:     ARM
    GL_RENDERER:   Mali-LODX
    GL_VERSION:    OpenGL ES 3.2 v1.g6p0-01eac0.efb75e2978d783a80fe78be1bfb0efc1
=======================================================
[build] use-vbo=false: FPS: 1214 FrameTime: 0.824 ms
[build] use-vbo=true: FPS: 924 FrameTime: 1.082 ms
[texture] texture-filter=nearest: FPS: 956 FrameTime: 1.046 ms
[texture] texture-filter=linear: FPS: 935 FrameTime: 1.070 ms
[texture] texture-filter=mipmap: FPS: 895 FrameTime: 1.117 ms
[shading] shading=gouraud: FPS: 843 FrameTime: 1.186 ms
[shading] shading=blinn-phong-inf: FPS: 870 FrameTime: 1.149 ms
[shading] shading=phong: FPS: 854 FrameTime: 1.171 ms
[shading] shading=cel: FPS: 869 FrameTime: 1.151 ms
[bump] bump-render=high-poly: FPS: 671 FrameTime: 1.490 ms
[bump] bump-render=normals: FPS: 1041 FrameTime: 0.961 ms
[bump] bump-render=height: FPS: 958 FrameTime: 1.044 ms
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 961 FrameTime: 1.041 ms
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 840 FrameTime: 1.190 ms
[pulsar] light=false:quads=5:texture=false: FPS: 908 FrameTime: 1.101 ms
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 771 FrameTime: 1.297 ms
[desktop] effect=shadow:windows=4: FPS: 1201 FrameTime: 0.833 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 532 FrameTime: 1.880 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 450 FrameTime: 2.222 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 608 FrameTime: 1.645 ms
[ideas] speed=duration: FPS: 1012 FrameTime: 0.988 ms
[jellyfish] <default>: FPS: 819 FrameTime: 1.221 ms
[terrain] <default>: FPS: 237 FrameTime: 4.219 ms
[shadow] <default>: FPS: 814 FrameTime: 1.229 ms
[refract] <default>: FPS: 348 FrameTime: 2.874 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 844 FrameTime: 1.185 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 836 FrameTime: 1.196 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 854 FrameTime: 1.171 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 848 FrameTime: 1.179 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 837 FrameTime: 1.195 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 844 FrameTime: 1.185 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 850 FrameTime: 1.176 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 846 FrameTime: 1.182 ms
=======================================================
                                  glmark2 Score: 826 
=======================================================
rock@rock5b:~$

top

Tasks: 227 total,   2 running, 225 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.3 us,  4.5 sy,  0.0 ni, 85.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15721.4 total,  14975.4 free,    432.4 used,    313.6 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  15121.7 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                        
 2951 rock      20   0 1750924 104120  68132 S  67.9   0.6   0:52.69 glmark2-es2                    
  619 root      20   0 2492392 177604 134204 R  49.0   1.1   2:04.31 Xorg                           
 2068 rock      20   0   35948  18908  15172 S   5.3   0.1   0:10.17 xfwm4                          
 1992 root       0 -20       0      0      0 I   3.3   0.0   0:01.83 kworker/u17:2-csf_scheduler_g+ 
 2962 root       0 -20       0      0      0 I   2.6   0.0   0:01.44 kworker/u17:3-mali_kbase_csf_+ 
    1 root      20   0  165132   9244   6948 S   0.3   0.1   0:05.81 systemd                        
   11 root      20   0       0      0      0 I   0.3   0.0   0:00.50 rcu_sched                      
   55 root      20   0       0      0      0 I   0.3   0.0   0:00.13 kworker/1:1-mm_percpu_wq       
   96 root      20   0       0      0      0 I   0.3   0.0   0:00.21 kworker/u16:1-events_unbound   
  280 root      20   0   49736  16696  15524 S   0.3   0.1   0:01.77 systemd-journal                
  486 message+  20   0    6992   3912   3112 S   0.3   0.0   0:01.90 dbus-daemon                    
 2088 rock      20   0   41460  24316  19336 S   0.3   0.2   0:04.57 panel-1-cpufreq                
 2915 root      20   0    9528   3520   2764 R   0.3   0.0   0:01.03 top                            
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd                       
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp                         
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp                     
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq                   
    9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_                
   10 root      20   0       0      0      0 S   0.0   0.0   0:00.20 ksoftirqd/0                    
   12 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/0                    
   13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0                        
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1                        
   15 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/1                    
   16 root      20   0       0      0      0 S   0.0   0.0   0:00.10 ksoftirqd/1                    
   19 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2                        
   20 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/2                    
   21 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftirqd/2                    
   24 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/3                        
   25 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/3                    
   26 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftirqd/3                    
   29 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/4                        
   30 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/4                    
   31 root      20   0       0      0      0 S   0.0   0.0   0:00.01 ksoftirqd/4                    
   34 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/5                        
   35 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/5                    
   36 root      20   0       0      0      0 S   0.0   0.0   0:00.01 ksoftirqd/5                    
   39 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/6                        
   40 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/6                    
   41 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/6                    
   44 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/7                        
   45 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/7                    
   46 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/7                    
   49 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kdevtmpfs                      
   50 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns                          
   51 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/4:1-cgroup_destroy     
   52 root      20   0       0      0      0 I   0.0   0.0   0:00.02 kworker/5:1-mm_percpu_wq       
   53 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker/0:1-pm

monitoring the whole benchmark

rock@rock5b:~$ sudo ./mon.sh 
[sudo] password for rock: 
 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200

 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200

 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0     200     200
   1800    2256    2256    2112       0     200     200
   1800    2256    2256    2112       0     200     200
   1800    2256    2256    2112       0     200     200
   1800    2256    2256    2112       0     200     200
^C

2 - Running https://webglsamples.org/persistence/persistence.html

top

Tasks: 237 total,   2 running, 235 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.7 us,  1.0 sy,  0.0 ni, 94.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15721.4 total,  14495.0 free,    700.2 used,    526.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  14818.6 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                        
 4628 rock      20   0 6509856 301916 219608 S  28.5   1.9   0:05.76 chromium-bin                   
 4803 rock      20   0   37.6g 108452  80676 R  23.8   0.7   0:04.28 chromium-bin                   
  619 root      20   0 2838848 185500 141920 S  10.3   1.2   3:34.04 Xorg                           
   99 root       0 -20       0      0      0 I   1.7   0.0   0:01.72 kworker/u17:0-mali_kbase_csf_+ 
 2068 rock      20   0   35948  18908  15172 S   1.7   0.1   0:18.00 xfwm4                          
    1 root      20   0  165132   9244   6948 S   0.3   0.1   0:07.02 systemd                        
  175 root      20   0       0      0      0 I   0.3   0.0   0:00.74 kworker/2:2-mm_percpu_wq       
  202 root      20   0       0      0      0 I   0.3   0.0   0:00.35 kworker/u16:3-flush-179:0      
  210 root       0 -20       0      0      0 I   0.3   0.0   0:04.17 kworker/u17:1-mali_kbase_csf_+ 
  486 message+  20   0    7108   3968   3112 S   0.3   0.0   0:02.28 dbus-daemon                    
 2088 rock      20   0   41460  24320  19336 S   0.3   0.2   0:05.59 panel-1-cpufreq                
 4568 root      20   0    9528   3512   2760 R   0.3   0.0   0:00.32 top                            
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd                       
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp                         
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp                     
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq                   
    9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_                
   10 root      20   0       0      0      0 S   0.0   0.0   0:00.24 ksoftirqd/0                    
   11 root      20   0       0      0      0 I   0.0   0.0   0:00.63 rcu_sched                      
   12 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/0                    
   13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0                        
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1                        
   15 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/1                    
   16 root      20   0       0      0      0 S   0.0   0.0   0:00.12 ksoftirqd/1                    
   19 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2                        
   20 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/2                    
   21 root      20   0       0      0      0 S   0.0   0.0   0:00.05 ksoftirqd/2                    
   24 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/3                        
   25 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/3                    
   26 root      20   0       0      0      0 S   0.0   0.0   0:00.05 ksoftirqd/3                    
   29 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/4                        
   30 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/4                    
   31 root      20   0       0      0      0 S   0.0   0.0   0:00.01 ksoftirqd/4                    
   34 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/5                        
   35 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/5                    
   36 root      20   0       0      0      0 S   0.0   0.0   0:00.01 ksoftirqd/5                    
   39 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/6                        
   40 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/6                    
   41 root      20   0       0      0      0 S   0.0   0.0   0:00.01 ksoftirqd/6                    
   44 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/7                        
   45 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/7                    
   46 root      20   0       0      0      0 S   0.0   0.0   0:00.00 ksoftirqd/7                    
   49 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kdevtmpfs                      
   50 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns                          
   52 root      20   0       0      0      0 I   0.0   0.0   0:00.02 kworker/5:1-mm_percpu_wq       
   55 root      20   0       0      0      0 I   0.0   0.0   0:00.15 kworker/1:1-mm_percpu_wq       
   56 root      20   0       0      0      0 I   0.0   0.0   0:00.02 kworker/6:1-events

Monitoring the whole WebGL:

rock@rock5b:~$ sudo ./mon.sh 
 CPU0-3  CPU4-5  CPU6-7     DDR     DSU     GPU     NPU
   1800    2256    2256    2112       0     200     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0     200     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0    1000     200
   1800    2256    2256    2112       0     200     200
   1800    2256    2256    2112       0     200     200

No crashing, no freezing!

Stat_headcrabed · October 16, 2022, 4:08am

Maybe lift up voltage of DDR chip and controller could help?

avaf · October 16, 2022, 2:23pm

Today i tested nanovg demo and we can compare it to my ancient Intel core 2 with NVIDIA Corporation G96 [GeForce 9500 GT] (rev a1) that draws 150W.

Rock 5B (ondemand governor)

Rock 5B (performance governor)

Intel® Core™2 Duo CPU E8600 @ 3.33GHz with GeForce 9500 GT blob

I can also see some glitches on the top left that are visible on the first screenshot, not sure if this is part of the demo, but i could see some when dragging the window on the intel hw.

gnattu · October 22, 2022, 4:59pm

I tried this under gnome, x11, chromium comes with debian(but using Panfrost, confirmed in chrome://gpu), the system never hangs, but the performance suffers a lot.

I noticed that Weston performs way better than gnome under x11, but I have not found a way to get into gnome/kde wayland session, gdm3 and sddm seems like does not work with panfrost. Have you managed to run kde/gnome under wayland using panfrost? Trivial compositors like sway/weston works, but I have not managed to get into gnome yet.

icecream95 · October 23, 2022, 6:41am

Yes, performance is sometimes quite bad at the moment, mostly because the driver waits for GPU jobs to complete after submitting them rather than doing things in parallel.

Switching to the performance governor for the CPU can help a lot, and doing the same for the RAM helps performance a little more:

$ echo performance | sudo tee /sys/bus/cpu/devices/cpu[046]/cpufreq/scaling_governor /sys/class/devfreq/dmc/governor

Gnome (and gdm) work well enough for me with Wayland. I’m using an Armbian image of Debian sid, what release are you using?

(Note that sid currently has a broken version of libffi, so my patch is needed for anything using Wayland to work correctly.)

icecream95 · October 25, 2022, 8:59am

I’ve just pushed a bunch of changes to my Mesa fork, it would be nice to have some testing to see if I’ve broken anything. But all of the major display servers work now (Gnome, KWin, Sway, Weston, Xorg) and seem to be quite stable—generally things work almost as well as any other GPU supported by Panfrost.

Performance could still be much better, but RK3588 is so powerful that it should already be faster than most other boards.

Perhaps someone could help with packaging for the various distributions people are using? I think it works well enough now that most users would be happy enough with it, especially for distributions that haven’t already gotten the blob set up.

gnattu · October 25, 2022, 9:27am

I’m going to take a look at this weekend and hopefully make an Arch Linux ARM PKGBUILD.

icecream95 · October 26, 2022, 2:19am

The DRAM instability that I had seems to not be GPU-specific, because stressapptest (which currently must be compiled with CXX=clang++) also causes a system hang if I have the 2112 MHz frequency enabled. I wondered if it could’ve been because of a memory transfer command that only the GPU uses, but that appears to not be the case.

gnattu · October 26, 2022, 2:27am

I have not encounter this issue myself on my 16GB board, perhaps this is related to a specific model of the LPDDR module?

icecream95 · October 26, 2022, 2:40am

It could be per-chip variation… just like some RK3588 chips can run at 2.4 GHz while there are others that can’t even hit 2.3 GHz, except that here there is no mechanism to automatically restrict frequencies.

I have a fairly low-clocking CPU (2.32 GHz and 2.24 GHz on the two big clusters) so I wouldn’t be surprised if the problem is SoC-side in the memory controller, rather than the DRAM chips.

Stat_headcrabed · October 26, 2022, 1:16pm

Have you tried lifting voltage yet?
This document could help:

mxcone17 · October 26, 2022, 4:07pm

I have had interference with my wireless keyboard plugged into usb 2.0 slot when a thumb-drive is plugged into a usb 3.0 spot. Anyone else experience this?

Stat_headcrabed · October 26, 2022, 4:46pm

USB3’s work frequency is near to 2.4Ghz, which is used by WiFi and Bluetooth. So it’s obvious they would interference without protection