I just noticed that they are advertising the Zero 2 Pro and I’ve had a look out for this for a while. I have a project I’ve been working on with AI inference and have been using a Rock 5A in my prototype. I’m quite pleased with it and probably should just stick with it but interested to know how the Zero 2 Pro compares, as I would prefer the form factor, lower tdp and especially the price. I’m using yolov8 models, but happy to hear any experiences such as difficulty to get working and speeds. How does the 5 TOPS NPU compare with the RK3588s 6 TOPS NPU. Has anyone even got the NPU working either on the 2 Pro or another device. I will be getting one when I see them next available.
Zero 2 Pro NPU - Any Experiences?
NPU is currently on available in Android, since it requires the use of Amlogic SDK. Our Linux image is based on upstream Linux kernel and Debian packages, which does not have support for it.
Further, we currently have no experience developing NPU related functionality on this SoC ourselves.
I have no idea but would this work?
I’m keeping an eye on this as well, I would like to buy the Zero 2 Pro due to the competitive price and form factor. However I if the NPU doesn’t work, then it’s useless for me. That said, Khadas VIM3 is based on the same SOC and they were early on supporting the NPU and the software support seems to be mature now. They have their own SDK based on Amlogic SDK, which seems OK, however it might be that it only works on their image. Easiest option seems to be what Stuart linked to, which is an extension / delegate which add support for the inference on the NPU using more or less standard TFLite. That implementation is backed by VeriSilicon which is the company that made the NPU. (See https://docs.khadas.com/products/sbc/vim3/npu/start)
I would very much like to know if the tflite_vx_delegate can be made to work on the Zero 2 Pro. There might be a need to have the right kernel version and some dependencies in place to make the delegate part work correctly on the Zero 2 Pro. I have no idea how difficult that would be and it would be great if Radxa could make this a smooth process, it would also relieve the burden of making a unique SDK for the Zero 2 Pro.
That said, going from RK3588 to A311D, you would need to make some changes to your code and workflow, if the tflite_vx_delegate works, it seems pretty straightforward, instead of converting your model using RKNN, you would quantize the model in TensorFlow, save it as TFLite and then its pretty much business as usual for running a TFLite model.
The Khadas VIM3 image with working NPU is 4.9 kernel, it’s way too old and end of life.
https://9to5linux.com/linux-kernel-4-9-reaches-end-of-life-after-6-years-of-support
I was told someone is working on the NPU driver for the mainline kernel, so I think we will have to wait for the upstream driver available by the end of Dec.
1 Like
2 Likes
Tomeu did a nice writeup on performance improvements: https://blog.tomeuvizoso.net/2024/02/etnaviv-npu-update-16-nice-performance.html
Has the Zero2 been launched now?
Hi! There are any demo or sample running NPU on Android Device? I need to run OpenCV with the AMLogic NPU, any tips on this cenario? If i switch to the official DebianOS image from radxa i can acess the NPU?
Sorry no interest in or clue about Android. Radxa folks will need to comment. From Linux the NPU should just work (subject to using a recent enough kernel version).
1 Like
There are some NDK examples in Amlogic SDK, so you get some commands to run, but not an app with graphical user interface. I have asked Android engineer to upload our source code, since this is not publicly available right now.
Urghhhh you will need to port OpenCV to Android and to Amlogic NPU. The SDK provided demo code only supports running the following models: yoloface, yolov2, yolov3, and facenet.
There is also no documentation. Not even us have access to it.
Our current Debian system does not support NPU since this is a rather new development. We are looking to add this in the future.
1 Like
Ok thank you! There are any examples or samples running NPU on any OS like Ubuntu or Debian? Can you guys send me some examples via private messages?
It’s worth following Tomeu Vizoso’s blog on NPU topics:
1 Like
NPU is support on Zero2pro with Mesa TFlite delegate.
Please refer the new radxa documentation which mention how to install/usage of the NPU delegate and with the example of MobileNet V1 Object Recognition
1 Like
I have 3 Radxa zero 2 Pro and none are working with this image.
The first two are stuck before the kernel initialization, and the latest gets stuck after the kernel initialization at boot time. I have been struggling to use the npu on this board and I have finally come to the conclusion that for many of those boards the npu is malfunctioning or there is an hardware issue.
I had a similar issue, my zero 2 pro kept stuck after kernel initialization.
My solution was to not connect a monitor in the first boot, use serial console to setup, then everything works fine (almost).
The device still faces significant issues. Booting from a complete power-off state is successful only about 3 out of 10 times. Additionally, even when successfully booted, frequent freezes occur when attempting to utilize the npu. I suspect the problem lies with the driver rather than being a hardware issue.
By the way, using the provided Armbian image may lead to performance problems. It’s recommended to switch the CPU governor to ‘schedutil’ to avoid losing around 30% of performance during CPU inference. Furthermore, adding your user to the ‘render’ group eliminates the need for ‘sudo’ when accessing the NPU.
I will try to compile the newest kernel to see if it fixes the issue.
It is quite annoying that Radxa released such an unpolished product (the NPU is literally on the first page of the introduction!)
1 Like
@pitounet, thanks for your feedback, we are fixing this problem now,
but you also can access your device via serial console or SSH to use NPU.
Thanks for the response. I’ll try that. I really want to use a SBC with the form factor of the zero2 with NPU, as soon as you have found the fix, please let me know, I am willing to buy a new one if necessary.
Meanwhile, I’ll try with serial or console.
Regards
If you are experiencing problems: are you running the board with a heatsink and/or fan attached? The A311D chip needs cooling or it will throttle itself and that might explain some odd behaviour. NB: I would use the latest kernel and latest mesa versions to improve NPU performance and stability. NPU support is a relatively recent development so older kernels (anything lower than Linux 6.6.y at least) are likely to have bugs.