Hi everyone,
I’ve been working on getting Whisper speech recognition running with NPU acceleration on my Dragon Q6A, and wanted to share my findings and see if anyone else has had success.
The Goal
Build a voice assistant combining:
- Llama 3.2 on NPU (working great via genie-t2t-run!)
- Whisper on NPU for speech-to-text
The Problem
After extensive testing with Qualcomm’s qai-hub-models, I found that Whisper cannot currently run on the QCS6490 NPU:
-
Non-quantized Whisper models (
whisper_tiny,whisper_small) only support float precision, but QCS6490 NPU requires quantized (INT8) I/O -
Pre-quantized variant (
whisper_small_quantized) requires AIMET-ONNX, which has no aarch64 Linux wheels -
AI Hub website doesn’t offer QCS6490 as a download target for Whisper models
-
Downloaded ONNX models are QNN-precompiled and require the QNN Execution Provider, not available for aarch64 Linux
I’ve filed a feature request with Qualcomm: quic/ai-hub-models#281
Working Workaround (CPU)
In the meantime, Whisper works fine on CPU via HuggingFace Transformers:
# Setup (in a venv)
pip install "transformers[torch]" librosa soundfile
# Test
python3 -c "
from transformers import pipeline
pipe = pipeline('automatic-speech-recognition', model='openai/whisper-tiny', device='cpu')
result = pipe('your_audio.wav')
print(result['text'])
"
Performance on Q6A (CPU):
| Model | Parameters | Speed |
|---|---|---|
| whisper-tiny | 39M | ~1x realtime |
| whisper-small | 244M | ~2.3x realtime |
Not as fast as NPU would be, but usable for voice assistant applications.
Questions for the Community
-
Has anyone successfully run Whisper on the Q6A NPU through a different method?
-
Is there a way to manually quantize models for QCS6490 using the QAIRT SDK directly (outside of qai-hub-models)?
-
Are there alternative speech recognition models that already have quantized versions for QCS6490?
Full Guide
I’ve updated my Q6A AI quickstart guide with the working CPU-based Whisper setup if anyone wants to try it here: the approach combines Llama on NPU + Whisper on CPU for a complete voice assistant.
Thanks!