Pytorch NPU support

#1

Hi,

We are trying to load a small Llama model on to the NPU. The PyTorch is complaining it is not one of the supported devices:

RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, maia, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: npu

How do we load the model to run on the NPU in o6?

model = model.to(device=“cpu”) is working.

Thanks

lsmod | grep -i “npu|cix”
cix_dsp_rproc 28672 0

dpkg -l | grep cix
ii cix-audio-dsp 0.01-1 arm64 cix-audio-dsp package
ii cix-bt-driver 0.01-1 arm64 cix-bt-driver package
ii cix-common-misc 1.0.0 arm64 Used for OS common customize
ii cix-cpipe 0.01-1 arm64 cix-cpipe package
ii cix-csidma-driver 0.01-1 arm64 cix-csidma-driver package
ii cix-debian-misc 1.0.0 arm64 Used as debian os customize
ii cix-dpu-ddk 0.01-1 arm64 cix-dpu-ddk package
ii cix-edge-slm 0.01-1 arm64 cix-edge-slm package
ii cix-env 1.0.0 arm64 cix-env package
ii cix-ffmpeg 1.0.0 arm64 cix-ffmpeg package
ii cix-firmware 0.01-1 arm64 cix-firmware package
ii cix-gpu-driver 0.01-1 arm64 cix-gpu-driver package
ii cix-gpu-drm 0.01-1 arm64 cix-gpu-drm package
ii cix-gpu-umd 1.0.0-1 arm64 cix-gpu-umd package
ii cix-grubcfg 1.0-1 arm64 Cix grub configuration
ii cix-gstreamer 0.01-1 arm64 cix-gstreamer package
ii cix-hdcp2 0.01-1 arm64 cix-hdcp2 package
ii cix-irqbalance 1.0.0-1 arm64 cix-irqbalance package
ii cix-isp-driver 0.01-1 arm64 cix-isp-driver package
ii cix-isp-umd 0.01-1 arm64 cix-isp-umd package
ii cix-libglvnd 1.0.0-1 arm64 cix-libglvnd package
ii cix-llama-cpp 0.01-1 arm64 cix-llama-cpp package
ii cix-ltp 1.0.0 arm64 cix-ltp package
ii cix-mesa 1.0.0-1 arm64 cix-mesa package
ii cix-mnn 0.01-1 arm64 cix-mnn package
ii cix-noe-umd 0.01-1 arm64 cix-noe-umd package
ii cix-npu-driver 0.01-1 arm64 cix-npu-driver package
ii cix-npu-onnxruntime 0.01-1 arm64 cix-npu-onnxruntime package
ii cix-npu-umd 0.01-1 arm64 cix-npu-umd package
ii cix-opencv 0.01-1 arm64 cix-opencv package
ii cix-openssl 0.01-1 arm64 cix-openssl package
ii cix-optee 0.01-1 arm64 cix-optee package
ii cix-tools 0.01-1 arm64 cix-tools package
ii cix-unit-test 0.01-1 arm64 cix-unit-test package
ii cix-vpu-driver 0.01-1 arm64 cix-vpu-driver package
ii cix-whisper-cpp 0.01-1 arm64 cix-whisper-cpp package
ii cix-wlan 1.0.0 arm64 cix-wlan package
ii cix-xserver 1.0.0-1 arm64 cix-xserver package
ii cixbot 1.3.5 arm64
ii linux-headers-6.1.44-cix-build-generic 6.1.44-cix-build-generic-2 arm64 Linux kernel headers for 6.1.44-cix-build-generic on arm64
ii linux-image-6.1.44-cix-build-generic 6.1.44-cix-build-generic-2 arm64 Linux kernel, version 6.1.44-cix-build-generic
ii linux-image-6.1.44-cix-build-generic-dbg 6.1.44-cix-build-generic-2 arm64 Linux kernel debugging symbols for 6.1.44-cix-build-generic
ii linux-libc-dev:arm64 6.1.44-cix-build-generic-2 arm64 Linux support headers for userspace development

#2

pytorch doesn’t support NPU backends, so you have to compile the model using the vendors tools and export it to CIX format.

I haven’t played with LLM models yet (just CV) however there are some docs for Llama here. https://docs.radxa.com/en/orion/o6/app-development/artificial-intelligence/llama_cpp

#3

Thanks for the response. The o6 specifications claim that PyTorch is supported on this system. Does that mean PyTorch only supports CPU on this system?

#4

The Pytorch support means the compiler to CIX format can read .py model format.

1 Like