Has anyone tried running Google’s new MoE model, “Gemma 4 26B A4B,” on the Radxa ROCK 5C yet? If so, how did it fare? And if it went well, how did you go about doing it?
I would assume you would want to use the rk-llama.cpp ecosystem with the --cpu-moe flag to make the MoE run better, but I haven’t tried it.
I have done some more digging on the internet and found this Reddit post: https://www.reddit.com/r/RockchipNPU/comments/1sc8fir/deploy_the_newest_qwen35_and_gemma4_models_of_any/
Somebody on the forum actually did test out several LLM’s on the Radxa ROCK 5C using the rk-llama.cpp library, including “Gemma 4 26B A4B”. This is what they got when using the int-8 quantized version of it:
On following (radxa-5c) with board with 32gigs of ram i got this, i will test more when i got some time:
Gemma4 26B-A4B Q8_0
- File: gemma-4-26B-A4B-it-Q8_0.gguf (25.00 GiB)
- https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF
- PP512: 28.54 tok/s | TG128: 5.00 tok/s
- NPU: 100% | CPU: 48% | RAM: 26 GB
They tried out several other larger models as well. You can go check those out yourself.
The results aren’t half bad; the TTFT definitely wouldn’t be instant, and generating the full response would definitely take a bit of time, but its speed is reasonable, that is, depending on what your definition of reasonable is. I bet if they used the int-4 quantized version, it would run ~2 times faster with minimal intelligence loss, especially if using the Unsloth version.
Memory is the biggest constraint though. When the Reddit User tested the int-8-quantized version, it required about 26 GB of RAM. I’m pretty sure even the int-4-quantized version would cost like 15 GB of RAM, meaning you would only be able to reasonably run either on a 32 GB board.
I will be testing it out myself sooner or later, and I will post my results.
Alright, I have tested everything out myself and have gotten some pretty good results.
The answer is yes, Gemma 4 MoE can run on a Radxa ROCK 5C at decent speeds! It spits out words faster than I can read, and the TTFT is usually seconds. (The ROCK 5C utilizes the Rockchip RK3588S2, so my results should also apply to any other board using the chipset.)
However, while the LLM starts spitting out tokens within only a few seconds, its reasoning time (thinking mode) is usually horrendous (bad). The time it takes for it to reason can range from a few seconds to several long minutes because it’s constantly double-checking, nitpicking, and “improving” its response. You can include things such as “Think concisely. Verify your answer once, not repeatedly.” in your prompt, but its effectiveness can vary. You can also just disable thinking mode entirely (details listed below) but accuracy during complex questions may drop.
Also, another good thing to note: it’s almost impossible to run this LLM on a board with less than 32 GB of RAM. While the actual model could technically fit on a 16 GB board (its size is 14.2 GB), you also have to account for Context Size and KV Cache, which can add several more GB if you don’t limit their sizes. Even if you do limit their sizes, you’re still really really REALLY pushing it on 16GB.
You can find more in-depth details regarding speed and setup below.
Board: Radxa ROCK 5C (32 GB)
OS: Armbian 26.5.1 Minimal - Vendor 6.1.115
(I’m pretty sure rk-llama will run on any Armbian OS as long as it’s a vendor version, not a current version; you’ll understand what I mean when booting up the Armbian installer.)
Model: Unsloth Int-4 Quantized Gemma 4 MoE (gemma-4-26B-A4B-it-qat-UD-Q4_K_XL)
Size: 14.2 GB (Even bigger when you throw in Context Size and KV Cache, but that’s only during runtime)
Prompt Processing: 15.94 ± 0.01 t/s
Generation: 7.00 ± 0.04 t/s
PPL: 425.02 ± 5.19 (576 chunks, full wikitext-2 test set)
NPU Utilization: Generally 35-45% (Each Core)
CPU Utilization: Generally 25% in Total (0% Efficiency Cores, 50% Performance Cores)
(I could not get the NPU or CPU utilization numbers higher)
Step 1 - Update and make sure Python and Pip are installed
sudo apt update
sudo apt install python3 python3-pip -y
Step 2 - Make sure CMake is also installed
sudo apt install cmake
Step 3 - Clone, create a directory for, and build rk-llama.cpp
git clone https://github.com/invisiofficial/rk-llama.cpp
cd rk-llama.cpp
mkdir build && cd build
cmake .. -DLLAMA_RKNPU2=ON
make -j4
Step 4 - Create and activate a Virtual Environment
# 'cd ~' is optional
cd ~
python3 -m venv hf_env
source hf_env/bin/activate
If venv is not installed, follow the installation instructions it gives you. Mine looked something like this:
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt install python3.13-venv
Step 5 - Install the Hugging Face Hub
pip install huggingface_hub
Step 6 - Install the actual LLM file (this will probably take a while)
hf download unsloth/gemma-4-26B-A4B-it-qat-GGUF --include "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" --local-dir ~/rk-llama.cpp/models/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL
Step 7 - Set all governors to performance and increase the maximum number of open files allowed (THIS IS RESET EVERY TIME YOU REBOOT OR ENTER A NEW SSH SESSION)
#Setting everything to performance allows the LLM run faster.
echo performance | sudo tee /sys/bus/cpu/devices/cpu[0-7]/cpufreq/scaling_governor
echo performance | sudo tee /sys/class/devfreq/fb000000.gpu/governor
echo performance | sudo tee /sys/devices/platform/dmc/devfreq/dmc/governor
echo performance | sudo tee /sys/class/devfreq/fdab0000.npu/governor
ulimit -n 65536
Step 8 - Run the actual model
cd ~/rk-llama.cpp
# '4-7' refers to the performance cores on the ROCK 5C. Apparently running effieciency cores and performance cores at the same time makes it slower.
taskset -c 4-7 ./build/bin/llama-cli -t 4 --cpu-moe -m ./models/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf
- Step 7 has to be repeated every time you reboot the board or start a new SSH session.
- ‘apt upgrade’ can BREAK the wifi.
- You can monitor CPU and NPU usage by using cosmotop.
- The virtual envrionment only needs to be activated when installing LLM’s from hugging face, or when installing the actual Hugging Face Hub. It does not need to be activated when running the model.
- You can turn off thinking mode entirely by running taskset -c 4-7 ./build/bin/llama-cli -t 4 --cpu-moe --jinja --reasoning off -m ./models/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf (don’t forget to run cd ~/rk-llama.cpp) in step 8 instead.
- You can limit Context Size, KV Cache Size, and how many tokens the model is allowed to generate in a single response by throwing -c 8192 -n 2048 -ctk q8_0 -ctv q8_0 onto the back of the taskset command used to run the LLM.
-c caps the context window, -n caps how many tokens it can generate per response, and -ctk/-ctv quantizes the KV cache.