After spending the last few weeks exploring the QCS6490 AI stack, I finally managed to get Llama 3.2 1B running on the Hexagon NPU of the Radxa Dragon Q6A.
Like many others here, I initially went down the QAIRT/QNN/QAI Hub/ExecuTorch rabbit hole. I tried multiple SDK versions, compiled binaries, Qualcomm AI Hub, official documentation, and even spun up a cloud machine just to verify I wasn’t doing something wrong. It turned out the ecosystem is much trickier to navigate than it first appears.
The breakthrough came from TrailCurrent’s installation script:
Video (not mine, full credit to the creator and everyone who contributed to updating the binaries):
What worked
Radxa Dragon Q6A (QCS6490)
Armbian 26.2.4 (Ubuntu 24.04 LTS)
Kernel 6.18.2-current-qcs6490
Llama 3.2 1B running on the Hexagon NPU
Persistent Genie server exposed through an Ollama-compatible API
~12-13 tokens/sec generation
So this setup is not limited to the official Radxa OS. It also works on Armbian.
Remaining questions
Although the NPU inference works well, I noticed two apparent limitations:
Context window appears fixed at 1024 tokens
Generation appears limited to roughly 250 output tokens
If anyone has successfully built or run a 4K or 8K context version on the Q6A, I’d love to know how.
Also, has anyone managed to run any other LLM on the Hexagon NPU besides Llama 3.2 1B? Or perhaps a larger model (4B-9B)?
Could anyone successfully make the GPU work (through ROCm or Vulcan?) – I could now.
Hopefully this saves someone else a few weeks of trial and error.
From my experience Vulkan is slower than CPU when running Qwen_Qwen3.5-2B-Q8_0.gguf. I’ve used the MESA FreeAdreno drivers and the latest Radxa kernel.
Compiling llama.cpp following the instructions for Qualcomm Snapdragon hits a wall while trying to talk to the fastrpc driver. Qualcomm’s documentation and website is hard to navigate. Lllama.cpp points to some SDK provided as a docker image and the fact the SDK published by Qualcomm only offers a x86 docker image tells me not even them expect people to use their stuff.
Anyway I think I get around 6T/s on CPU with 8k context window on Qwen3.5-2B and a bit less (5.5ish) on Vulkan.
Were you able to leverage the NPU at all? With any other model?
Also, is there any Qairt compiled model (larger, 4-8B) that can run faster on the CPU, that you may be aware of?
Thats as far as I could go and while I’ve tested quants of other models including 8b it wasn’t very fast. Since it’s CPU you could try getting a better result by switching the CPU governor from the default schedutil to performance but I’m not sure how far it would help.
Should be easy to test. Any docker image for lamma-cpp made for raspberry pi 5 should work on the dragon since both are aarch64.
I was trying to build binaries that would leverage the NPU.
So far couldn’t get those to work.
But only one model: Qwen 2.5 1.7B model got successfully compiled…
Was giving close to 12-13 tokens per second… (Thought it was still falling back to the CPU)
But idk after some update it got corrupted again…
Now I am back with the gguf running at half the speed…
Radxa Dragon Q6A (QCS6490)