Mesa PanVK has not yet been optimized for compute performance, but is still working on adding extensions and improving compatibility. Therefore, running vkpeak on it will have lower performance.
1 Like
cix@cix-localhost:~$ taskset -c 0,5,6,7,8,9,10,11 llama-bench-vulkan -m qwen2.5-3b-instruct-q4_0.gguf -pg 128,128 -t 8 -ngl 1000
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Mali-G720-Immortalis (Mali-G720-Immortalis) | uma: 1 | fp16: 1 | warp size: 16 | shared memory: 32768 | int dot: 0 | matrix cores: none
| model | size | params | backend | ngl | threads | test | t/s |
|---|---|---|---|---|---|---|---|
| qwen2 3B Q4_0 | 1.69 GiB | 3.09 B | Vulkan | 1000 | 8 | pp512 | 240.67 ± 0.15 |
| qwen2 3B Q4_0 | 1.69 GiB | 3.09 B | Vulkan | 1000 | 8 | tg128 | 15.75 ± 0.20 |
| qwen2 3B Q4_0 | 1.69 GiB | 3.09 B | Vulkan | 1000 | 8 | pp128+tg128 | 29.28 ± 0.05 |
PanVK is not optimized yet, with Mali proprietary vulkan driver, I can get better prefill performance than CPU on O6
2 Likes
Nice, this is interesting for when you want to consume large documents. However the lower speed in TG can be annoying in chat mode or in thinking mode.
I wonder, technically, because it’s shared RAM (?), whether it’d be possible to PP on GPU and TG on CPU? Does it incur overhead if the CPU tries to access memory regions allocated to GPU? (Not sure if Llama would support any of this in practice)
Anyone know where I might be able to find rough instructions on how to install the official driver to get working on O6? I’m a Kernel pleb, but would like to do some benchmarking of QwenA3B30B.
I’ve been wondering as well but I doubt it will often work, because some models are likely to be transcoded on the fly when being loaded depending on the backend and optimal operations that can be performed on them. Let’s imagine a GPU which is already optimized for 8-bit operations but not 4-bit ones, a 4-bit model could be optimally converted to 8-bit while being loaded for example. Thus if for some reason the memory layout slightly differs between GPU and CPU, it might not be workable to use the same memory storage for both backends. At least I remember that some formats like Q4_0 are optimal for armv8 and Q4_1 for apple, so presumably these ones are not modified while loaded.
1 Like
To get latest vulkan driver support, you can visit https://developer.cixtech.com/ download latest “GO Graphics Engine“; install on your platform.
Note that our profiling show that we need to disable subgroup for dequant_mul_mat_vec shader on Mali GPU, as this causes decode performance drop.
Apply patche below,
diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
index 80ad98a5..ad5ae0f0 100644
--- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp
+++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
@@ -3149,7 +3149,7 @@ static void ggml_vk_load_shaders(vk_device& device) {
rm_stdq = 2;
uint32_t rm_iq = 2 * rm_kq;
- const bool use_subgroups = device->subgroup_arithmetic && device->architecture != vk_device_architecture::AMD_GCN;
+ const bool use_subgroups = device->subgroup_arithmetic && device->architecture != vk_device_architecture::AMD_GCN && device->architecture != vk_device_architecture::OTHER;
// Ensure a subgroup size >= 16 is available
const bool use_subgroups16 = use_subgroups && subgroup_min_size_16;
1 Like
Good point! When it comes to Vulkan, I think many quant types have their own kernel that processes them in their native format (no transform), so it might be possible for some of them (e.g. Q4KM). (I also could be mistaken here).
Thanks! I did try to give this a quick spin last night, but I’m on kernel 6.17 (Ubuntu Server 25.10), so the DKMS couldn’t build (perusing source, looks like highest it might support is around 6.10).
I did check to see whether it might be possible to write simple compat. layer over the code for later Kernel versions, but it looks like some of the APIs are just fundamentally different and I’m not well-versed enough in kernel development to work my way through that.
I might revert back to Radxa’s Debian Image when I get some time next week and give it a whirl.
Depending on how this pans out, I might be out of the game sorry: