These are my experiments with NPU to detect objects in real-time using MIPI CSI OV5647 and a USB camera to show how exposure affects the results. I am no expert in AI / NPU but i think there is room for some enhancements. The demo use OpenCV4.2, SDL2 (KMSDRM), and NPU. It is based on odroid-m1 example of how to detect objects in real-time with NPU.
note:
The lens used with OV5647 and the USB 2 MB sensor is the same.
OV5647 captures frames at 15 FPS while the USB at 5 FPS. I can capture frames (1920x1080) at 30 FPS with H264 on the USB camera but i could not make OpenCV work with H264. I tried FFMpeg which i am able to decode H264 in hw but OpenCV refuses to return the frame.
The setup:
- Ubuntu 20.04 (debloated) CLI with mail blob
- kernel 4.19.193
- OpenCV 4.2
- NPU
Shooting with different exposure (ov5647 - $ 19.00 with IR):
https://drive.google.com/file/d/1ho9G7_mIKO5zEiqJ_RWL23NHrMXlAYRd/view
https://drive.google.com/file/d/1Zy_3FyCDZVAg3uSWj38Nz17GYbutzKqU/view
https://drive.google.com/file/d/1vAftvMvsmJb6qHXTjvXGtgkqGSWZdsfJ/view
Shooting with USB camera (2 MB - $ 30.00)
https://drive.google.com/file/d/19DEaJrXDEalCuttyX4ce3zZ2ccOCqY4k/view
How it is done:
- grab a frame from the camera (1980x1080)
- adjust to 640x640
- run NPU
- render on screen (1920x1080) with sdl2 using KMSDRM
The experiment is done using the following sources:
OV56447 driver port by @abel
For future enhancements, i am looking for removing OpenCV as it grabs the frame using gstreamer which seems to do some minor image conversion and image enhancements but has some memory leaks. I switched to pure v4l2 to fix the memory leak.
I opened an issue report with OpenCV, but i am not sure if the memory leak is on OpenCV or gstreamer.
If anyone happens to know how to fix it, please do so.
Cheers