Recently, I tried out Google’s new Gemma 4 MoE model on the Radxa ROCK 5C hardware using the rk-llama.cpp ecosystem, and got some pretty good results (you can check them out here).
Now I plan on testing every single Gemma 4 LLM on the ROCK 5C hardware using rk-llama.cpp. I will post my results once I have finished.
Alright, I have finally tested every Gemma 4 model on the Radxa ROCK 5C 
I ended up testing both the Int-4 Quantized and Int-8 Quantized models on the board. Furthermore, I compared their speeds when they ran on the CPU only via llama.cpp, and when they ran on the CPU + NPU via rk-llama.cpp.
You can check out my exact results and numbers further down (and how to set them up yourself), but to sum things up: big dense models no run well, smaller models and MoE did. And whether you should run them on CPU ONLY or CPU + NPU depends on your priorities and model type.
For E2B and E4B, due to a missing tensor bug, you actually can’t run the int-4 versions with rk-llama, so your only option is to run them with CPU ONLY via normal Llama. Fortunately, the int-8 versions did not have this problem.
Long story short, both ran pretty well, and pretty fast (ESPECIALLY int-4 Gemma 4 E2B). And supposedly, if you use models from Unsloth, the int-4 or int-8 models are just as smart as the original model.
If you want faster raw generation speeds, you should use normal llama and run it on CPU ONLY. If you want TTFT times that are about twice as fast and generation speeds that are still good, you should use rk-llama and run it on CPU + NPU.
The large dense models 12B and 31B ran pretty slow regardless of what you did.
Running 31B on the ROCK 5C is just a really bad idea in general. It’s huge (even with int-4 Quantization), and very, very slow. Just don’t do it.
You could get away with running 12B, but its speeds are also going to be pretty slow regardless of what you do. For maximum results, you should run the int-4 Quantized model, on CPU ONLY.
MoE actually does pretty well speed-wise; size is a different story. If you want to learn more about it, go check out the results here.
Int-4 Quantized (2.61 GB)
CPU ONLY
Prompt Processing: 67.66 ± 0.04 t/s
Generation: 17.25 ± 0.03 t/s
(Had to use normal llama.cpp due to a tensor bug)
CPU + NPU
Due to a missing-tensor bug, the int-4 version of E2B cannot be loaded by rk-llama.cpp, and therefore the NPU cannot be utilized.
Int-8 Quantized (5.04 GB)
CPU ONLY
Prompt Processing: 39.35 ± 0.03 t/s
Generation: 8.85 ± 0.00 t/s
CPU + NPU
Prompt Processing: 69.73 ± 0.03 t/s
Generation: 7.13 ± 0.05 t/s
Int-4 Quantized (4.2 GB)
CPU ONLY
Prompt Processing: 34.96 ± 0.04 t/s
Generation: 8.81 ± 0.00 t/s
(Had to use normal llama.cpp due to a tensor bug)
CPU + NPU
The int-4 version of E4B also had a missing tensor bug (I know, amazing right?) and could not be loaded by rk-llama, and therefore the NPU could not be utilized.
Int-8 Quantized (8.18 GB)
CPU ONLY
Prompt Processing: 19.18 ± 0.02 t/s
Generation: 4.49 ± 0.00 t/s
CPU + NPU
Prompt Processing: 38.06 ± 0.01 t/s
Generation: 4.26 ± 0.02 t/s
Int-4 Quantized (6.7 GB)
CPU ONLY
Prompt Processing: 13.19 ± 0.02 t/s
Generation: 3.59 ± 0.00 t/s
CPU + NPU
Prompt Processing: 5.09 ± 0.00 t/s
Generation: 3.35 ± 0.01 t/s
Int-8 Quantized (13.63 GB)
CPU ONLY
Prompt Processing: 8.25 ± 0.02 t/s
Generation: 1.84 ± 0.00 t/s
CPU + NPU
Prompt Processing: 29.78 ± 0.54 t/s
Generation: 1.99 ± 0.00 t/s
Skipped since it was already tested here: Gemma 4 26B A4B (MoE) on Radxa ROCK 5C?
Speed was good; size was a different story.
Int-4 Quantized (17.3 GB)
CPU ONLY
Prompt Processing: 4.95 ± 0.00 t/s
Generation: 1.42 ± 0.00 t/s
CPU + NPU
Prompt Processing: 1.37 ± 0.00 t/s
Generation: 1.06 ± 0.00 t/s
Int-8 Quantized (32.6 GB)
The Int-8 Quantized version of 31B was too big to fit on the board.
Step 1 - Update and make sure Python and Pip are installed
sudo apt update
sudo apt install python3 python3-pip -y
Step 2 - Make sure CMake is also installed
sudo apt install cmake
Step 3.1 (If you want to use CPU + NPU) - Clone, create a directory for, and build rk-llama.cpp
git clone https://github.com/invisiofficial/rk-llama.cpp
cd rk-llama.cpp
mkdir build && cd build
cmake .. -DLLAMA_RKNPU2=ON
make -j4
Step 3.2 (If you want to use CPU ONLY) - Clone, create a directory for, and build normal llama.cpp
git clone https://github.com/ggml-org/llama.cpp llama.cpp-mainline
cd llama.cpp-mainline
mkdir build && cd build
cmake ..
make -j4
Step 4 - Create and activate a Virtual Environment
# 'cd ~' is optional
cd ~
python3 -m venv hf_env
source hf_env/bin/activate
If venv is not installed, follow the installation instructions it gives you. Mine looked something like this:
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt install python3.13-venv
Step 5 - Install the Hugging Face Hub
pip install huggingface_hub
Step 6 - Choose and download which model you want to use. DO NOT RUN ALL OF THEM
#Gemma 4 E2B (int-4)
hf download unsloth/gemma-4-E2B-it-qat-GGUF --include "gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf" --local-dir ~/models/gemma-4-E2B-it-qat-UD-Q4_K_XL
# NOTE: fails to load on rk-llama.cpp (missing-tensor bug). Use the normal llama.cpp build from Step 3.2 instead
#Gemma 4 E2B (int-8)
hf download unsloth/gemma-4-E2B-it-GGUF --include "gemma-4-E2B-it-Q8_0.gguf" --local-dir ~/models/gemma-4-E2B-it-Q8_0
#Gemma 4 E4B (int-4)
hf download unsloth/gemma-4-E4B-it-qat-GGUF --include "gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf" --local-dir ~/models/gemma-4-E4B-it-qat-UD-Q4_K_XL
# NOTE: also fails to load on rk-llama.cpp (missing-tensor bug). Use the normal llama.cpp build from Step 3.2 instead
#Gemma 4 E4B (int-8)
hf download unsloth/gemma-4-E4B-it-GGUF --include "gemma-4-E4B-it-Q8_0.gguf" --local-dir ~/models/gemma-4-E4B-it-Q8_0
#Gemma 4 12B (int-4)
hf download unsloth/gemma-4-12B-it-qat-GGUF --include "gemma-4-12B-it-qat-UD-Q4_K_XL.gguf" --local-dir ~/models/gemma-4-12B-it-qat-UD-Q4_K_XL
#Gemma 4 12B (int-8)
hf download unsloth/gemma-4-12b-it-GGUF --include "gemma-4-12b-it-UD-Q8_K_XL.gguf" --local-dir ~/models/gemma-4-12B-it-Q8_0
# NOTE: repo path is lowercase "12b"; this is Unsloth's dynamic UD-Q8_K_XL scheme, not a plain Q8_0
#Gemma 4 26B A4B (int-4)
If you'd like to run Gemma 4 MoE, go here instead: https://forum.radxa.com/t/gemma-4-26b-a4b-moe-on-radxa-rock-5c
#Gemma 4 31B (int-4)
hf download unsloth/gemma-4-31B-it-qat-GGUF --include "gemma-4-31B-it-qat-UD-Q4_K_XL.gguf" --local-dir ~/models/gemma-4-31B-it-qat-UD-Q4_K_XL
#Gemma 4 31B (int-8)
hf download unsloth/gemma-4-31B-it-GGUF --include "gemma-4-31B-it-Q8_0.gguf" --local-dir ~/models/gemma-4-31B-it-Q8_0
# NOTE: ~32.6GB file — not recommended, almost certainly won't fit given this board's disk/RAM (see benchmark section)
Step 7 - Set all governors to performance and increase the maximum number of open files allowed (THIS IS RESET EVERY TIME YOU REBOOT OR ENTER A NEW SSH SESSION)
#Setting everything to performance allows the LLM run faster.
echo performance | sudo tee /sys/bus/cpu/devices/cpu[0-7]/cpufreq/scaling_governor
echo performance | sudo tee /sys/class/devfreq/fb000000.gpu/governor
echo performance | sudo tee /sys/devices/platform/dmc/devfreq/dmc/governor
echo performance | sudo tee /sys/class/devfreq/fdab0000.npu/governor
ulimit -n 65536
Step 8.1 (If you want to use CPU + NPU) - Run the actual model with rk-llama.cpp
cd ~/rk-llama.cpp
# '4-7' refers to the performance cores on the ROCK 5C. Apparently running efficiency cores and performance cores at the same time makes it slower.
taskset -c 4-7 ./build/bin/llama-cli -t 4 -m ~/models/<model-folder>/<model-file>.gguf
Step 8.2 (If you want to use CPU ONLY) - Run the actual model with normal llama.cpp
cd ~/llama.cpp-mainline
#Again, '4-7' refers to the performance cores.
taskset -c 4-7 ./build/bin/llama-cli -t 4 -m ~/models/<model-folder>/<model-file>.gguf
- You can easily find the path to your downloaded model for Step 8 by running : find ~/models -name “gemma-4*.gguf”
- Step 7 has to be repeated every time you reboot the board or start a new SSH session.
- I’m pretty sure rk-llama and llama will run just fine on any Armbian OS as long as it’s a vendor version and not a current version.
- You can monitor CPU and NPU usage by using cosmotop.
- The virtual environment only needs to be activated when installing LLM’s from Hugging Face, or when installing the actual Hugging Face Hub. It does not need to be activated when running the model.
- You can turn off thinking mode entirely by using the following taskset commands in Step 8 instead:
8.1 Alternative
cd ~/rk-llama.cpp
taskset -c 4-7 ./build/bin/llama-cli -t 4 --jinja --reasoning off -m ~/models/<model-folder>/<model-file>.gguf
8.2 Alternative
cd ~/llama.cpp-mainline
taskset -c 4-7 ./build/bin/llama-cli -t 4 --jinja --reasoning off -m ~/models/<model-folder>/<model-file>.gguf
- You can limit Context Size, KV Cache Size, and how many tokens the model is allowed to generate in a single response by throwing -c 8192 -n 2048 -ctk q8_0 -ctv q8_0 onto the back of the taskset command used to run the LLM.
-c caps the context window, -n caps how many tokens it can generate per response, and -ctk/-ctv quantizes the KV cache.
Impressive! Respect.
Any luck on the VL part?
I was going to try out Moondream2 one of these days, once I can finally get my ESP32-S3 Sense to properly stream to my ROCK 5C, so I could also easily test out VL when I get to that point.