IMX415 + NPU demo on ROCK 5B

avaf · August 22, 2022, 7:29pm

This is a demo of Rock 5B with an IMX415 camera and NPU to detect objects in real-time.
FPS: 13 ~19
Image size: 1920x1080

rknn

Video:
https://drive.google.com/file/d/1t8MuSeTZjkzzqXmhvVqH7FbDSeDb6-BN/view?usp=sharing

Based on this:

setq · August 24, 2022, 7:45am

Thank you for sharing!

3djelly · August 24, 2022, 11:29pm

Are you downscaling the image from 1920x1080 to 640x640 before feeding to OpenCV?

If not, what FPS do you get on a 640x640 downscaled image?

avaf · August 25, 2022, 1:31pm

Yes.

On the upper left corner of the screenshot is the FPS.

grab a frame from the camera (1980x1080) - 30FPS
downscale to 640x640
run NPU
render it on screen (1920x1080) with sdl - (13 ~19 FPS)

Unfortunately, I left a 10ms delay on each frame while debugging the code. I think that slowed things down a bit. I will re-run the code without the delay. Maybe i can get 20 FPS or more.

avaf · August 25, 2022, 7:12pm

Little gain. I think it depends on the number of objects detected. There were some spikes of 20 FPS. (14 ~ 20 FPS)

That’s it for now!

3djelly · September 2, 2022, 9:20pm

Thanks for trying that out, I would have expected higher FPS as Google Coral’s USB at 4 TOPS gives FPS in the mid teens.
When my Rock Pi is delivered I will try it out and profile each step of the process in case there is some other bottle neck resulting in the lower than anticipated FPS.

avaf · October 8, 2022, 1:05pm

I have not seen any low-cost sbc with Google Coral’s USb performing above 20 fps when detecting many objects in real time. Can you share where you see that?

With some optimizations you get 25 fps, considering the camera is at 30 fps and the frames are 1920x1080.

It would be exciting to have a complete face detection demo and a benchmark ported to rknpu.
Boards are now being shipped.

3djelly · October 8, 2022, 7:32pm

Mid teens means around 14-15 FPS, I have not seen it go above 20 FPS either.

25 FPS is more of what I was expecting, so that is good to hear.

avaf · November 17, 2022, 2:50pm

Testing the demo on Debian X11 and kernel 5.10.110, pushing it to the limits:

IMX415 + NPU without performance governor

IMX415 + NPU with performance governor

IMX415 + HD720 (USB)

IMX415 + HD720 (USB) + H264 (USB - in YUV mode)

flyingRich · December 23, 2022, 6:55pm

I am using frigate NVR currently on the Rock 5B with a google coral. Is there a way to switch it to use the ROCK NPU?

avaf · December 23, 2022, 8:08pm

Please, check how to convert it here: https://github.com/rockchip-linux/rknn-toolkit2
I used radxa and rockchip (latest) model already converted.

Once you can convert it, or someone else, please let me know so i can try to run it and see how it compares.

Wenzhe_Ma · January 7, 2023, 1:28pm

This is fantastic. What OS did you use to run NPU detection? Thanks

avaf · January 7, 2023, 2:13pm

The first screenshot is from Ubuntu 22.04 and has no Desktop (CLI) 1920x1080, and the other with Graphical Desktop is Debian 11.5 (X11) which is lost when my eMMC died. I plan to redo everything with NVME when it arrives which can take some time. I think OS is irrelevant, you need a Lean and Mean distro with HW acceleration. Maybe with Wayland is possible to get better results.

@flyingRich
Did you convert the frigate tensorflow lite model?
I think it is this one:

github.com

blakeblackshear/frigate/blob/dev/frigate/detectors/plugins/edgetpu_tfl.py#L37


if detector_config.device is not None:
    device_config = {"device": detector_config.device}


edge_tpu_delegate = None


try:
    logger.info(f"Attempting to load TPU as {device_config['device']}")
    edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
    logger.info("TPU found")
    self.interpreter = tflite.Interpreter(
        model_path=detector_config.model.path or "/edgetpu_model.tflite",
        experimental_delegates=[edge_tpu_delegate],
    )
except ValueError:
    logger.error(
        "No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors."
    )
    raise


self.interpreter.allocate_tensors()

avaf · January 7, 2023, 9:55pm

Running the demo with SDL3.
Here is a pre-built package for Debian/Ubuntu:

Note that the API is changing, what works today may not work tomorrow. Porting sdl2 to sdl3 is getting harder.

avaf · January 11, 2023, 12:23pm

FYI,YOLOv8 version released. Has anyone tried that?

phiber · February 1, 2023, 4:30pm

I want to start playing with the NPU. Have you posted your code to github?
My plan is to get an RTSP stream from an IP camera, run it through the rock 5 NPU and output to another h264 stream on the local network.
I’m lacking documentation on how to do the first 2 steps

avaf · February 1, 2023, 10:41pm

No, I haven’t. I just recreated Debian 11 and tried to restore the same environment lost with the two failed eMMC. Still waiting for the m2 sata adapter and a way to convert the models (if that’s possible), i think i have to learn how to do it myself.

Regarding your RTSP project, i think the only way to achieve that is with gstreamer. GStreamer has a tone of documentation but zero practical examples.

Maybe this could be a start:

phiber · February 2, 2023, 12:15am

so far what I see is that by default, from repos there is no hardware support for gstreamer from radxa mirrors.

gstreamer1.0-rockchip1 : Depends: librockchip-mpp1 but it is not installable

To solve that, one could use rockchip mpp for gstreamer release which is from 2017 and won’t support h265 or any new hardware. That means the newest develop source must be used for compilation which will come with many more problems.
To build it from source, one must spend countless hours fixing every single problem that compilation will throw up. (I spent 2 hours so far, immediately after your reply).

So, rockchip specs are nice and all but currently there is no software support and the only official github repo has 154 issues and 3 week old commits which don’t help with anything.

I guess we need to wait a year or two to actually be able to use the hardware as advertised.

avaf · February 2, 2023, 12:58pm

There is a little gem here:

There you find all pre-built deb packages if you don’t want to spend time building/compiling and fixing all stuff.
Start with a minimal image that does not have any broken dependencies and then install the required packages. I think rkr1 was meant for kernel 5.10.66 ( ~ 72 - patches) and rkr4 for 5.10.110. But rkr1 will work on 5.10.110 with some minor features missing.

Care must be taken by the order you install the packages. And don’t make the mistake to install all at once and then reboot, install one package, reboot and check, repeat…

Tip:

librga
libmpp
xserver (X11)
etc…

phiber · February 2, 2023, 3:52pm

yeah, like i said nothing that’s installable and usable out of the box.
thanks for the link though