How much effort is it to run Coral - compatible models on built-in NPUs?

#1

Hi.
I played around with Google Coral recently and am considering making a small project or two.

However, the price of the accelerator is becoming less and less competitive, M.2. modules require adapters that also are expensive, and software support seems to be dwindling (can’t even install Python libraries above version 3.9).

Since I don’t have any experience with AI accelerators yet, especially the ones that are built into SoCs, I decided to ask…

How much effort would it take to make Coral - compatible models run on NPUs that Rockchip (RK3588S) and Amlogic (A311D) SoCs are equipped with?

I’m interested in both, since I haven’t decided which board to use yet.

#2

Go for the RK3588 as Rockchips SDK is much better than Amlogic’s SDK on the A311D.

The SDK supports converting TFLite models which the google coral use, however I wouldn’t waste my time with the coral as its very limiting and slow so the inference Models that can run on it are no longer the best (in terms of speed and accuracy). Also Tensorflow (TFLite) is an abandoned mess so your time is best spent learning pytorch and running the Models on the RK3588.

1 Like
#3

Correct me if I’m wrong, but Amlogic seems to have way better general Linux support currently. I saw somewhere that it has a significant level of mainline support with a lot of hardware features enabled, which Rockchip doesn’t have yet. It matters to me because I’d also like to have a good overall board if I decide that AI is not for me.

Regardless though… Is it really true that TFLite is such an abandoned mess? I assumed that if it’s something Google - made, then it would have good support. Even if Coral models aren’t the best, I’m still just a beginner in AI.

Are pytorch models supported by the NPUs?

Are there any pose-estimation and speech recognition models that would work?

#4

As for mainline support that is not something I care about, I am fine with Radxa’s Debian image. In the microcontroller/SoC/embedded space software support is quite ugly compared to desktop so depending on what you want is quite specific to each user.

Yes Tensorflow is abandoned mess, google is also well known for discontinuing products and has done so with the Coral. They have moved on from Tensorflow to JAX and the general consensus in the computer learning/vision community is to use pytorch for any new projects.

Pytorch models are supported by rknn-toolkit2. However its not too important as most frameworks support exporting to ONNX which rknn-toolkit2 can also use.

The RKNN Model Zoo has C++ and Python examples of yolov8-pose. I also create Go bindings and have ported the pose estimation code too.

I haven’t played with the speech recognition models, but there are some on the Model Zoo: Wav2Vec2, Whisper, and Zipformer.