Has anyone tried running Google’s new MoE model, “Gemma 4 26B A4B,” on the Radxa ROCK 5C yet? If so, how did it fare? And if it went well, how did you go about doing it?
I would assume you would want to use the rk-llama.cpp ecosystem with the --cpu-moe flag to make the MoE run better, but I haven’t tried it.
I also understand that the Gemma 4 MoE model may be too big and/or too beefy for the Radxa ROCK 5C and could probably only run on the 32 GB variant of the SBC. But I figured I’d ask anyway.
Somebody on the forum actually did test out several LLM’s on the Radxa ROCK 5C using the rk-llama.cpp library, including “Gemma 4 26B A4B”. This is what they got when using the int-8 quantized version of it:
On following (radxa-5c) with board with 32gigs of ram i got this, i will test more when i got some time:
Gemma4 26B-A4B Q8_0
- File: gemma-4-26B-A4B-it-Q8_0.gguf (25.00 GiB)
- https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF
- PP512: 28.54 tok/s | TG128: 5.00 tok/s
- NPU: 100% | CPU: 48% | RAM: 26 GB
They tried out several other larger models as well. You can go check those out yourself.
The results aren’t half bad; the TTFT definitely wouldn’t be instant, and generating the full response would definitely take a bit of time, but its speed is reasonable, that is, depending on what your definition of reasonable is. I bet if they used the int-4 quantized version, it would run ~2 times faster with minimal intelligence loss, especially if using the Unsloth version.
Memory is the biggest constraint though. When the Reddit User tested the int-8-quantized version, it required about 26 GB of RAM. I’m pretty sure even the int-4-quantized version would cost like 15 GB of RAM, meaning you would only be able to reasonably run either on a 32 GB board.
I will be testing it out myself sooner or later, and I will post my results.