[showcase] NPU with CSI and USB cameras

ROCK 3 Series
1

These are my experiments with NPU to detect objects in real-time using MIPI CSI OV5647 and a USB camera to show how exposure affects the results. I am no expert in AI / NPU but i think there is room for some enhancements. The demo use OpenCV4.2, SDL2 (KMSDRM), and NPU. It is based on odroid-m1 example of how to detect objects in real-time with NPU.

note:
The lens used with OV5647 and the USB 2 MB sensor is the same.
OV5647 captures frames at 15 FPS while the USB at 5 FPS. I can capture frames (1920x1080) at 30 FPS with H264 on the USB camera but i could not make OpenCV work with H264. I tried FFMpeg which i am able to decode H264 in hw but OpenCV refuses to return the frame.

The setup:

  • Ubuntu 20.04 (debloated) CLI with mail blob
  • kernel 4.19.193
  • OpenCV 4.2
  • NPU

rock3a_cameras
rock3a_cameras1280×576 112 KB

Shooting with different exposure (ov5647 - $ 19.00 with IR):

https://drive.google.com/file/d/1ho9G7_mIKO5zEiqJ_RWL23NHrMXlAYRd/view

https://drive.google.com/file/d/1Zy_3FyCDZVAg3uSWj38Nz17GYbutzKqU/view

https://drive.google.com/file/d/1vAftvMvsmJb6qHXTjvXGtgkqGSWZdsfJ/view

Shooting with USB camera (2 MB - $ 30.00)

https://drive.google.com/file/d/19DEaJrXDEalCuttyX4ce3zZ2ccOCqY4k/view

How it is done:

  • grab a frame from the camera (1980x1080)
  • adjust to 640x640
  • run NPU
  • render on screen (1920x1080) with sdl2 using KMSDRM

The experiment is done using the following sources:

OV56447 driver port by @abel

For future enhancements, i am looking for removing OpenCV as it grabs the frame using gstreamer which seems to do some minor image conversion and image enhancements but has some memory leaks. I switched to pure v4l2 to fix the memory leak.

I opened an issue report with OpenCV, but i am not sure if the memory leak is on OpenCV or gstreamer.
If anyone happens to know how to fix it, please do so.

Cheers

5 Likes