Rockchip NPU update 4

#1

Kernel driver for the RK3588 NPU submitted to mainline
In the past few weeks I have been working on among other things a kernel driver for the NPU in the Rockchip RK3588 SoC, new from the ground up.

It is now fully working and after a good amount of polishing I sent it yesterday to the kernel mailing lists, for review. Those interested can see the code and follow the review process at this link.

The kernel driver is able to fully use the three cores in the NPU, giving us the possibility of running 4 simultaneous object detection inferences such as the one below on a stream, at almost 30 frames per second.

https://www.youtube.com/watch?v=DDccYn4wpnY&t=1s

The userspace driver is in a less polished state, but fully featured at this state. I will be working on this in the next few days so it can be properly submitted for review.

This is the first accelerator-only driver for an edge NPU submitted to the mainline kernel, and hopefully it can serve as a template for the next ones to come, as the differences among NPUs of different vendors are relatively superficial.

6 Likes
#2

Thanks for your hard work! Looking forward to be able to use this.

#3

Not mine hence the blockquote.

To be honest for my purpose I would much rather have bigger cores and a bigger GPU and use the die space of a NPU. Then far more models and training could be done than CCN detect models because common frameworks and avail source code have support.

I am just posting its gone mainline, but 3x 2 tops npu isn’t much use for me.

2 Likes
#4

good news and thanks for your hardwork!

#5

I think it would be nice if you posted a link to the original article: https://blog.tomeuvizoso.net/2024/06/rockchip-npu-update-4-kernel-driver-for.html?m=1

1 Like
#6

RKVDEC2 Driver Posted For Accelerated Video Decoding On Newer Rockchip SoCs

For years there has been the RKVDEC Linux media driver to provide accelerated video decoding on Rockchip SoCs. Being worked on now is RKVDEC2 for providing video decoding on the newer Rockchip SoCs.

RKVDEC2 patches were posted on Friday by Collabora’s Detlev Casanova. The initial hardware target with RKVDEC2 is on supporting video decode for Rockchip’s RK3588. The RK3588 SoC pairs a quad-core Arm Cortex-A76 and quad-core Arm Cortex-A55 with a Mali-G610 MC4 graphics, a basic NPU, and video accelerated decode for H.264 / H.265 / AV1 / AVS2.

Rockchip RK3588

The RKVDEC2 is being developed since with the newer SoCs is second-generation Rockchip video decode IP. There are dual video decoders on the RK3588 for supporting up to 8K@30 video but with the initial Linux driver only using one video decoder at a time is currently supported. This driver also is initially limited to H.264 even though HEVC/H.265, VP9, AV1, and AVS2 are also supported by the hardware.

The RKVDEC2 driver is based in part on the original RKVDEC driver. Those interested in this second-generation Rockchip video decode driver for Linux can find the initial patches under review on the kernel mailing list. Along with the forthcoming NPU driver, the RK3588 has been seeing a lot of nice upstream driver support work.