Use YoloV8 in RK3588 NPU

nickliu973 · May 4, 2023, 2:57am

I didn’t try directly run the yolov8 in rock5b.
You need to run the yolov8 in a PC and convert the model (some modifications) using RKNN toolkit2.
Then, run the converted model in rock5b.
You can use this command to clone the yolov8
https://github.com/ultralytics/ultralytics.git

You may want to try the yolov5 examples in the rknn-toolkit2 before working on YoloV8.

Dbenton · May 4, 2023, 1:20pm

I managed to get it just after posting, but yes still have reboot issues. I will try v5.

Dbenton · May 4, 2023, 1:30pm

Seems I have found a solution. Running the board using the Ameridroid power adapter, I had previously been using a USB-A to USB-C cable. Currently using a USB-C to USB-C and it trained all 3 epochs without a reboot.

nickliu973 · May 4, 2023, 6:26pm

Nice! If there is any problems, don’t hesitate to let me know.

arzen · May 5, 2023, 5:01am

convert yolov8 onnx to rknn, show error when run it on rk3588, error message:
failed to submit!, op id: 173, op name: Add:Add_243

convert script setting:

please help，thanks!

Luke_Mira · May 5, 2023, 6:27am

This is a beginner question.
Is the rk3588 model (.rknn) created on an AMD64 linux machine (installed rknn-toolkit2)?
Also, if you simulate it (.rknn) on Rock5b (installed rknpu2),
Do I have to run it on rock’s debian11 instead of armbian? Is it not possible to simulate on armbian?

1117 · May 5, 2023, 7:51am

Good day!
I tried exactly the same (yolov8s.pt to onnx, than to rknn)
And got these error http://joxi.ru/KAgPBgXhEwg59r

The same is with yolov5 and any version of yolov8

Please try to follow your insturtions now, maybe you will get the same negative result

nickliu973 · May 5, 2023, 2:16pm

Hi, can you share a screenshot of your onnx model using the https://netron.app/?
Also, please find the node Add_243 and take a screenshot of that node with the connected nodes.
Something like this:

nickliu973 · May 5, 2023, 2:21pm

Hi:) @Luke_Mira

Yes. Some people also managed to create the .rknn in rock 5B but I never tried that.

Do you mean run the neural network model in the physical NPU?
Sorry, I’m using my customized Ubuntu. I didn’t try it in debian or armbian.
Maybe ask the radxa admins could get a better answer?

nickliu973 · May 5, 2023, 2:25pm

Hi,
It looks like your script isn’t run the inference simulation.
Can you check the code of rknn is configured as following?
rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]], target_platform='rk3588')

Dbenton · May 5, 2023, 3:23pm

I’m attempting to use OpenCV for a webcam stream and feed it to a yolov8 model I have for face detection.

Without using OpenCV, I am able to run (in python):

 model= YOLO("mymodel.pt")
 model.predict(source="1", show=True)

However this has pretty bad frames and I wanted to give OpenCV a try for more customization.

I can use OpenCV and open the webcam and even show it. However, using:

ret, frame = cv2.read()
results = model(frame)

I receive an error:

The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

trying to use any() or all() results in an error that a file ‘None’ or ‘True’ does not exist. I’m not quite sure what’s happening, but I have found plenty of examples of OpenCV webcam frames being sent to YOLO this way without such an error.

nickliu973 · May 5, 2023, 3:49pm

Hi,
I didn’t try the YOLO+webcam before.
It looks like the camera device id, e.g., /dev/video20 is not properly specified in the both cases.
Can you check the camera device id is correct?

1117 · May 5, 2023, 4:34pm

I checked, all is correct, exactly from your post

Can you try to make rknn right now with your instruction?
From step 1 (git clone …ultralytics)

nickliu973 · May 5, 2023, 4:53pm

It’s working on my desktop.

1117 · May 5, 2023, 5:03pm

Can you share export.py and test.py files?
From ultralytics and rknn toolkit

Can you do the same with yolov5s.pt?

nickliu973 · May 5, 2023, 5:13pm

I missed your points.
If I understand correctly, you’re using the YoloV5/YoloV8s model generated from ultralytics, not the example one from rknn toolkit2.

For this step, you need replace the post-processing code, where the error comes from, in the test.py.

Dbenton · May 5, 2023, 6:01pm

Yes, the video source is correct. /dev/video1 is the source, and it can be accessed how I showed there. The first code snippet runs perfectly fine other than poor frames. I can access the video source using OpenCV just fine as well using:

cap = cv2.VideoCapture(1) or cap = cv2.VideoCapture('/dev/video1')

The issue is when passing the OpenCV frame to model().

1117 · May 5, 2023, 6:04pm

What do you mean?
What code i need to replace? Step4 http://joxi.ru/BA01G5WfMeBzer?
All of these is done

1117 · May 5, 2023, 6:05pm

Yes, i take onnx, that i got on step 2 and try to convert it to rknn

Can you do the same?

Got these error, when tried to execute rknn on rk3588 board

E RKNN: [02:30:45.402] failed to submit!, op id: 190, op name: Add:/model.24/Add_2, flags: 0x5, task start: 942, task number: 36, run task counter: 18, int status: 0
Traceback (most recent call last):
File “/home/sitnikov/YOLOv5-RK3588-Python/inference_npu.py”, line 218, in
input1_data = outputs[1]
IndexError: list index out of range

nickliu973 · May 5, 2023, 6:29pm

Hi, you may need to convert the cv.Mat to data structure for the model input.
This link has the example usage of using OpenCV.

github.com

ultralytics/ultralytics/blob/main/examples/YOLOv8-OpenCV-ONNX-Python/main.py

import argparse

import cv2.dnn
import numpy as np

from ultralytics.yolo.utils import ROOT, yaml_load
from ultralytics.yolo.utils.checks import check_yaml

CLASSES = yaml_load(check_yaml('coco128.yaml'))['names']

colors = np.random.uniform(0, 255, size=(len(CLASSES), 3))


def draw_bounding_box(img, class_id, confidence, x, y, x_plus_w, y_plus_h):
    label = f'{CLASSES[class_id]} ({confidence:.2f})'
    color = colors[class_id]
    cv2.rectangle(img, (x, y), (x_plus_w, y_plus_h), color, 2)
    cv2.putText(img, label, (x - 10, y - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2)

This file has been truncated. show original

Let me know if it works for you.