Use YoloV8 in RK3588 NPU

Step 1: follow the instruction to install the YoloV8 from https://github.com/ultralytics/ultralytics/tree/main/examples/YOLOv8-CPP-Inference.

Step 2: export the model to ONNX with using:

yolo export model=yolov8s.pt imgsz=640,640 format=onnx opset=12

The onnx file yolov8s.onnx will be generated.

Step 3, copy the onnx file to the

rknn-toolkit2/examples/onnx/yolov5/

Step 4. made three modifications of the test.py file

ONNX_MODEL = 'yolov8s.onnx'
RKNN_MODEL = 'yolov8s.onnx.rknn'
rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]], target_platform="rk3588")
 outputs = rknn.inference(inputs=[img])

Re-program pre- and post- processing code to fit the model into RK3588 NPU hardware.
(The post-processing code in the yolov8 can be copy-paste to the test.py)

Step 5, run the conversion

python test.py

and the yolov8s.onnx.rknn will be generated.

Step 6. Upload the rknn file to rk3588 and test it with the benchmark demo.

Step 7. Use the rknn_yolov5_demo as template to test the inference, disable the OEM post-processing code and program the one for YoloV8 as the dimension of inference output are different.

6 Likes

I think my hardware is too old for this, can you please share your yolov8s.onnx and yolov8s.onnx.rknn?

My attempt:

alex@svn:/apps/arm/rk3588/RKNN/ultralytics$ yolo export model=yolov8s.pt imgsz=640,640 format=onnx opset=12
Ultralytics YOLOv8.0.68 πŸš€ Python-3.8.0 torch-1.10.1+cu102 CPU
[W NNPACK.cpp:79] Could not initialize NNPACK! Reason: Unsupported hardware.
YOLOv8s summary (fused): 168 layers, 11156544 parameters, 0 gradients, 28.6 GFLOPs

PyTorch: starting from yolov8s.pt with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (21.5 MB)
/home/alex/.local/lib/python3.8/site-packages/onnx/mapping.py:27: FutureWarning: In the future `np.object` will be defined as the corresponding NumPy scalar.
  int(TensorProto.STRING): np.dtype(np.object)
module 'numpy' has no attribute 'object'.
`np.object` was a deprecated alias for the builtin `object`. To avoid this error in existing code, use `object` by itself. Doing this will not modify any behavior and is safe. 
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
/home/alex/.local/lib/python3.8/site-packages/onnx/mapping.py:27: FutureWarning: In the future `np.object` will be defined as the corresponding NumPy scalar.
  int(TensorProto.STRING): np.dtype(np.object)
ONNX: export failure ❌ 0.0s: module 'numpy' has no attribute 'object'.
`np.object` was a deprecated alias for the builtin `object`. To avoid this error in existing code, use `object` by itself. Doing this will not modify any behavior and is safe. 
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

onnx version:

alex@svn:/apps/arm/rk3588/RKNN/ultralytics$ pip show onnx
Name: onnx
Version: 1.9.0
Summary: Open Neural Network Exchange
Home-page: https://github.com/onnx/onnx
Author: ONNX
Author-email: onnx-technical-discuss@lists.lfai.foundation
License: Apache License v2.0
Location: /home/alex/.local/lib/python3.8/site-packages
Requires: numpy, protobuf, six, typing-extensions
Required-by: onnxoptimizer, rknn-toolkit2
alex@svn:/apps/arm/rk3588/RKNN/ultralytics$ pip show onnxoptimizer
Name: onnxoptimizer
Version: 0.2.7
Summary: Open Neural Network Exchange
Home-page: https://github.com/onnx/optimizer
Author: ONNX Optimizer Authors
Author-email: onnx-technical-discuss@lists.lfai.foundation
License: Apache License v2.0
Location: /home/alex/.local/lib/python3.8/site-packages
Requires: onnx
Required-by: rknn-toolkit2

numpy:

alex@svn:/apps/arm/rk3588/RKNN/ultralytics$ pip show numpy
Name: numpy
Version: 1.24.2
Summary: Fundamental package for array computing in Python
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: 
License: BSD-3-Clause
Location: /home/alex/.local/lib/python3.8/site-packages
Requires: 
Required-by: contourpy, h5py, jax, Keras-Preprocessing, matplotlib, ml-dtypes, onnx, onnxruntime, opencv-python, opt-einsum, pandas, rknn-toolkit2, scipy, seaborn, tensorboard, tensorflow, torchvision, ultralytics

Hi, @avaf.
The forum doesn’t allow me to upload a file larger than 4MB.

But I could install and export yolov8n (not yolov8s) in rk3588.

Probably the power issue, each time I tried to export the yolov8s in rk3588 board, my board would get restarted.

But i still need to convert onnx to rknn, is it possible to convert it to rknn in rk3588?

If possible, you can use kim dotcom mega (mega.nz) or perhaps googledrive.

Hi, @avaf,

Yes, the yolov8n.onnx can be converted to rknn in RK3588.
But you have to find the rknn-toolkit for the RK3588.
I saw someone did this in another forum but don’t remember where it is exactly.

A stretch run of the converted yolov8n (PC converted) is as:

The forum doesn’t allow mega.nz file to be uploaded.
And the compressed rknn file is also over 4MB.
image

No google drive. I want to limit the interaction within the forum only.
Thanks.

ock@rock5b:~/rockchip/RKNN$ git clone https://github.com/ultralytics/ultralytics --depth=1
Cloning into 'ultralytics'...
remote: Enumerating objects: 248, done.
remote: Counting objects: 100% (248/248), done.
remote: Compressing objects: 100% (223/223), done.
remote: Total 248 (delta 36), reused 102 (delta 20), pack-reused 0
Receiving objects: 100% (248/248), 620.63 KiB | 4.89 MiB/s, done.
Resolving deltas: 100% (36/36), done.
rock@rock5b:~/rockchip/RKNN$ cd ultralytics/
rock@rock5b:~/rockchip/RKNN/ultralytics$ pip install -r requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Collecting matplotlib>=3.2.2
  Downloading matplotlib-3.7.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (11.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.3/11.3 MB 2.9 MB/s eta 0:00:00
Collecting numpy>=1.21.6
  Downloading numpy-1.24.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 26.3 MB/s eta 0:00:00
Collecting opencv-python>=4.6.0
  Downloading opencv_python-4.7.0.72-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (40.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.4/40.4 MB 21.0 MB/s eta 0:00:00
Requirement already satisfied: Pillow>=7.1.2 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 8)) (9.0.1)
Requirement already satisfied: PyYAML>=5.3.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 9)) (5.4.1)
Requirement already satisfied: requests>=2.23.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 10)) (2.25.1)
Collecting scipy>=1.4.1
  Downloading scipy-1.10.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (30.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 30.9/30.9 MB 2.7 MB/s eta 0:00:00
Collecting torch>=1.7.0
  Downloading torch-2.0.0-1-cp310-cp310-manylinux2014_aarch64.whl (74.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.3/74.3 MB 13.2 MB/s eta 0:00:00
Collecting torchvision>=0.8.1
  Downloading torchvision-0.15.1-cp310-cp310-manylinux2014_aarch64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 22.3 MB/s eta 0:00:00
Collecting tqdm>=4.64.0
  Downloading tqdm-4.65.0-py3-none-any.whl (77 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.1/77.1 KB 12.1 MB/s eta 0:00:00
Collecting pandas>=1.1.4
  Downloading pandas-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (11.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 28.4 MB/s eta 0:00:00
Collecting seaborn>=0.11.0
  Downloading seaborn-0.12.2-py3-none-any.whl (293 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 293.3/293.3 KB 31.6 MB/s eta 0:00:00
Collecting psutil
  Downloading psutil-5.9.4.tar.gz (485 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 485.8/485.8 KB 32.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting thop>=0.1.1
  Downloading thop-0.1.1.post2209072238-py3-none-any.whl (15 kB)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3/dist-packages (from matplotlib>=3.2.2->-r requirements.txt (line 5)) (2.4.7)
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 31.6 MB/s eta 0:00:00
Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3/dist-packages (from matplotlib>=3.2.2->-r requirements.txt (line 5)) (2.8.1)
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.39.3-py3-none-any.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 32.2 MB/s eta 0:00:00
Collecting packaging>=20.0
  Downloading packaging-23.0-py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.7/42.7 KB 7.1 MB/s eta 0:00:00
Collecting contourpy>=1.0.1
  Downloading contourpy-1.0.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (283 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 283.7/283.7 KB 33.3 MB/s eta 0:00:00
Collecting filelock
  Downloading filelock-3.11.0-py3-none-any.whl (10.0 kB)
Collecting networkx
  Downloading networkx-3.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 31.6 MB/s eta 0:00:00
Collecting typing-extensions
  Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting jinja2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 KB 18.5 MB/s eta 0:00:00
Collecting sympy
  Downloading sympy-1.11.1-py3-none-any.whl (6.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.5/6.5 MB 24.1 MB/s eta 0:00:00
Collecting python-dateutil>=2.7
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 KB 28.9 MB/s eta 0:00:00
Collecting tzdata>=2022.1
  Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.8/341.8 KB 34.7 MB/s eta 0:00:00
Collecting pytz>=2020.1
  Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 502.3/502.3 KB 35.3 MB/s eta 0:00:00
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.7->matplotlib>=3.2.2->-r requirements.txt (line 5)) (1.16.0)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (26 kB)
Collecting mpmath>=0.19
  Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 KB 34.8 MB/s eta 0:00:00
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml) ... done
  Created wheel for psutil: filename=psutil-5.9.4-cp310-abi3-linux_aarch64.whl size=278808 sha256=4214dc365f2af49265df4fd669c41695d14c31c529fd3b696082b9242e726d5a
  Stored in directory: /home/rock/.cache/pip/wheels/8c/f0/2c/0d7d6e64319efa308609a6844a955ef94aa797388843265de0
Successfully built psutil
Installing collected packages: pytz, mpmath, tzdata, typing-extensions, tqdm, sympy, python-dateutil, psutil, packaging, numpy, networkx, MarkupSafe, kiwisolver, fonttools, filelock, cycler, scipy, pandas, opencv-python, jinja2, contourpy, torch, matplotlib, torchvision, thop, seaborn
Successfully installed MarkupSafe-2.1.2 contourpy-1.0.7 cycler-0.11.0 filelock-3.11.0 fonttools-4.39.3 jinja2-3.1.2 kiwisolver-1.4.4 matplotlib-3.7.1 mpmath-1.3.0 networkx-3.1 numpy-1.24.2 opencv-python-4.7.0.72 packaging-23.0 pandas-2.0.0 psutil-5.9.4 python-dateutil-2.8.2 pytz-2023.3 scipy-1.10.1 seaborn-0.12.2 sympy-1.11.1 thop-0.1.1.post2209072238 torch-2.0.0 torchvision-0.15.1 tqdm-4.65.0 typing-extensions-4.5.0 tzdata-2023.3
rock@rock5b:~/rockchip/RKNN/ultralytics$ pip install -e '.[dev]'
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/rock/rockchip/RKNN/ultralytics
  Preparing metadata (setup.py) ... done
Requirement already satisfied: Pillow>=7.1.2 in /usr/lib/python3/dist-packages (from ultralytics==8.0.69) (9.0.1)
Requirement already satisfied: PyYAML>=5.3.1 in /usr/lib/python3/dist-packages (from ultralytics==8.0.69) (5.4.1)
Requirement already satisfied: matplotlib>=3.2.2 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (3.7.1)
Requirement already satisfied: numpy>=1.21.6 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (1.24.2)
Requirement already satisfied: opencv-python>=4.6.0 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (4.7.0.72)
Requirement already satisfied: pandas>=1.1.4 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (2.0.0)
Requirement already satisfied: psutil in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (5.9.4)
Requirement already satisfied: requests>=2.23.0 in /usr/lib/python3/dist-packages (from ultralytics==8.0.69) (2.25.1)
Requirement already satisfied: scipy>=1.4.1 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (1.10.1)
Requirement already satisfied: seaborn>=0.11.0 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (0.12.2)
Collecting sentry_sdk
  Downloading sentry_sdk-1.19.1-py2.py3-none-any.whl (199 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.2/199.2 KB 3.9 MB/s eta 0:00:00
Requirement already satisfied: thop>=0.1.1 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (0.1.1.post2209072238)
Requirement already satisfied: torch>=1.7.0 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (2.0.0)
Requirement already satisfied: torchvision>=0.8.1 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (0.15.1)
Requirement already satisfied: tqdm>=4.64.0 in /home/rock/.local/lib/python3.10/site-packages (from ultralytics==8.0.69) (4.65.0)
Collecting check-manifest
  Downloading check_manifest-0.49-py3-none-any.whl (20 kB)
Collecting coverage
  Downloading coverage-7.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (228 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 228.8/228.8 KB 24.1 MB/s eta 0:00:00
Collecting mkdocs-material
  Downloading mkdocs_material-9.1.6-py3-none-any.whl (7.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 29.7 MB/s eta 0:00:00
Collecting mkdocstrings[python]
  Downloading mkdocstrings-0.21.2-py3-none-any.whl (26 kB)
Collecting pytest
  Downloading pytest-7.2.2-py3-none-any.whl (317 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.2/317.2 KB 36.4 MB/s eta 0:00:00
Collecting pytest-cov
  Downloading pytest_cov-4.0.0-py3-none-any.whl (21 kB)
Requirement already satisfied: packaging>=20.0 in /home/rock/.local/lib/python3.10/site-packages (from matplotlib>=3.2.2->ultralytics==8.0.69) (23.0)
Requirement already satisfied: contourpy>=1.0.1 in /home/rock/.local/lib/python3.10/site-packages (from matplotlib>=3.2.2->ultralytics==8.0.69) (1.0.7)
Requirement already satisfied: python-dateutil>=2.7 in /home/rock/.local/lib/python3.10/site-packages (from matplotlib>=3.2.2->ultralytics==8.0.69) (2.8.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/rock/.local/lib/python3.10/site-packages (from matplotlib>=3.2.2->ultralytics==8.0.69) (1.4.4)
Requirement already satisfied: cycler>=0.10 in /home/rock/.local/lib/python3.10/site-packages (from matplotlib>=3.2.2->ultralytics==8.0.69) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /home/rock/.local/lib/python3.10/site-packages (from matplotlib>=3.2.2->ultralytics==8.0.69) (4.39.3)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3/dist-packages (from matplotlib>=3.2.2->ultralytics==8.0.69) (2.4.7)
Requirement already satisfied: pytz>=2020.1 in /home/rock/.local/lib/python3.10/site-packages (from pandas>=1.1.4->ultralytics==8.0.69) (2023.3)
Requirement already satisfied: tzdata>=2022.1 in /home/rock/.local/lib/python3.10/site-packages (from pandas>=1.1.4->ultralytics==8.0.69) (2023.3)
Requirement already satisfied: sympy in /home/rock/.local/lib/python3.10/site-packages (from torch>=1.7.0->ultralytics==8.0.69) (1.11.1)
Requirement already satisfied: jinja2 in /home/rock/.local/lib/python3.10/site-packages (from torch>=1.7.0->ultralytics==8.0.69) (3.1.2)
Requirement already satisfied: filelock in /home/rock/.local/lib/python3.10/site-packages (from torch>=1.7.0->ultralytics==8.0.69) (3.11.0)
Requirement already satisfied: typing-extensions in /home/rock/.local/lib/python3.10/site-packages (from torch>=1.7.0->ultralytics==8.0.69) (4.5.0)
Requirement already satisfied: networkx in /home/rock/.local/lib/python3.10/site-packages (from torch>=1.7.0->ultralytics==8.0.69) (3.1)
Collecting tomli
  Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting build>=0.1
  Downloading build-0.10.0-py3-none-any.whl (17 kB)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from check-manifest->ultralytics==8.0.69) (59.6.0)
Collecting mkdocs>=1.4.2
  Downloading mkdocs-1.4.2-py3-none-any.whl (3.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 26.5 MB/s eta 0:00:00
Collecting mkdocs-material-extensions>=1.1
  Downloading mkdocs_material_extensions-1.1.1-py3-none-any.whl (7.9 kB)
Collecting pygments>=2.14
  Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 34.6 MB/s eta 0:00:00
Collecting pymdown-extensions>=9.9.1
  Downloading pymdown_extensions-9.11-py3-none-any.whl (239 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.9/239.9 KB 24.4 MB/s eta 0:00:00
Requirement already satisfied: colorama>=0.4 in /usr/lib/python3/dist-packages (from mkdocs-material->ultralytics==8.0.69) (0.4.4)
Collecting requests>=2.23.0
  Downloading requests-2.28.2-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 KB 10.4 MB/s eta 0:00:00
Collecting regex>=2022.4.24
  Downloading regex-2023.3.23-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (768 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 769.0/769.0 KB 31.7 MB/s eta 0:00:00
Collecting markdown>=3.2
  Downloading Markdown-3.4.3-py3-none-any.whl (93 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 93.9/93.9 KB 13.7 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (195 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 195.9/195.9 KB 28.5 MB/s eta 0:00:00
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.23.0->ultralytics==8.0.69) (2020.6.20)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests>=2.23.0->ultralytics==8.0.69) (1.26.5)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.23.0->ultralytics==8.0.69) (3.3)
Requirement already satisfied: MarkupSafe>=1.1 in /home/rock/.local/lib/python3.10/site-packages (from mkdocstrings[python]->ultralytics==8.0.69) (2.1.2)
Collecting mkdocs-autorefs>=0.3.1
  Downloading mkdocs_autorefs-0.4.1-py3-none-any.whl (9.8 kB)
Collecting mkdocstrings-python>=0.5.2
  Downloading mkdocstrings_python-0.9.0-py3-none-any.whl (38 kB)
Collecting attrs>=19.2.0
  Downloading attrs-22.2.0-py3-none-any.whl (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.0/60.0 KB 8.1 MB/s eta 0:00:00
Collecting exceptiongroup>=1.0.0rc8
  Downloading exceptiongroup-1.1.1-py3-none-any.whl (14 kB)
Collecting pluggy<2.0,>=0.12
  Downloading pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting iniconfig
  Downloading iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.9/140.9 KB 20.2 MB/s eta 0:00:00
Collecting pyproject_hooks
  Downloading pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)
Collecting markdown>=3.2
  Downloading Markdown-3.3.7-py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.8/97.8 KB 8.3 MB/s eta 0:00:00
Collecting mergedeep>=1.3.4
  Downloading mergedeep-1.3.4-py3-none-any.whl (6.4 kB)
Collecting watchdog>=2.0
  Downloading watchdog-3.0.0-py3-none-manylinux2014_aarch64.whl (82 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.0/82.0 KB 6.8 MB/s eta 0:00:00
Requirement already satisfied: click>=7.0 in /usr/lib/python3/dist-packages (from mkdocs>=1.4.2->mkdocs-material->ultralytics==8.0.69) (8.0.3)
Collecting ghp-import>=1.0
  Downloading ghp_import-2.1.0-py3-none-any.whl (11 kB)
Collecting pyyaml-env-tag>=0.1
  Downloading pyyaml_env_tag-0.1-py3-none-any.whl (3.9 kB)
Collecting griffe>=0.24
  Downloading griffe-0.26.0-py3-none-any.whl (88 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.3/88.3 KB 12.2 MB/s eta 0:00:00
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.7->matplotlib>=3.2.2->ultralytics==8.0.69) (1.16.0)
Requirement already satisfied: mpmath>=0.19 in /home/rock/.local/lib/python3.10/site-packages (from sympy->torch>=1.7.0->ultralytics==8.0.69) (1.3.0)
Installing collected packages: watchdog, urllib3, tomli, regex, pyyaml-env-tag, pygments, pluggy, mkdocs-material-extensions, mergedeep, markdown, iniconfig, griffe, exceptiongroup, coverage, charset-normalizer, attrs, sentry_sdk, requests, pytest, pyproject_hooks, pymdown-extensions, ghp-import, pytest-cov, mkdocs, build, ultralytics, mkdocs-material, mkdocs-autorefs, check-manifest, mkdocstrings, mkdocstrings-python
  Running setup.py develop for ultralytics
Successfully installed attrs-22.2.0 build-0.10.0 charset-normalizer-3.1.0 check-manifest-0.49 coverage-7.2.3 exceptiongroup-1.1.1 ghp-import-2.1.0 griffe-0.26.0 iniconfig-2.0.0 markdown-3.3.7 mergedeep-1.3.4 mkdocs-1.4.2 mkdocs-autorefs-0.4.1 mkdocs-material-9.1.6 mkdocs-material-extensions-1.1.1 mkdocstrings-0.21.2 mkdocstrings-python-0.9.0 pluggy-1.0.0 pygments-2.14.0 pymdown-extensions-9.11 pyproject_hooks-1.0.0 pytest-7.2.2 pytest-cov-4.0.0 pyyaml-env-tag-0.1 regex-2023.3.23 requests-2.28.2 sentry_sdk-1.19.1 tomli-2.0.1 ultralytics urllib3-1.26.15 watchdog-3.0.0
rock@rock5b:~/rockchip/RKNN/ultralytics$ python3 -m pip list | grep numpy
numpy                      1.24.2
rock@rock5b:~/rockchip/RKNN/ultralytics$ python3 -m pip list | grep onnx
rock@rock5b:~/rockchip/RKNN/ultralytics$ pip install onnx
Defaulting to user installation because normal site-packages is not writeable
Collecting onnx
  Downloading onnx-1.13.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (13.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.8/13.8 MB 2.2 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.16.6 in /home/rock/.local/lib/python3.10/site-packages (from onnx) (1.24.2)
Collecting protobuf<4,>=3.20.2
  Downloading protobuf-3.20.3-cp310-cp310-manylinux2014_aarch64.whl (918 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 918.4/918.4 KB 30.8 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=3.6.2.1 in /home/rock/.local/lib/python3.10/site-packages (from onnx) (4.5.0)
Installing collected packages: protobuf, onnx
Successfully installed onnx-1.13.1 protobuf-3.20.3
rock@rock5b:~/rockchip/RKNN/ultralytics$ pip install onnxruntime
Defaulting to user installation because normal site-packages is not writeable
Collecting onnxruntime
  Downloading onnxruntime-1.14.1-cp310-cp310-manylinux_2_27_aarch64.whl (4.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 3.3 MB/s eta 0:00:00
Collecting coloredlogs
  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 KB 5.9 MB/s eta 0:00:00
Requirement already satisfied: sympy in /home/rock/.local/lib/python3.10/site-packages (from onnxruntime) (1.11.1)
Requirement already satisfied: numpy>=1.21.6 in /home/rock/.local/lib/python3.10/site-packages (from onnxruntime) (1.24.2)
Requirement already satisfied: protobuf in /home/rock/.local/lib/python3.10/site-packages (from onnxruntime) (3.20.3)
Collecting flatbuffers
  Downloading flatbuffers-23.3.3-py2.py3-none-any.whl (26 kB)
Requirement already satisfied: packaging in /home/rock/.local/lib/python3.10/site-packages (from onnxruntime) (23.0)
Collecting humanfriendly>=9.1
  Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 KB 12.9 MB/s eta 0:00:00
Requirement already satisfied: mpmath>=0.19 in /home/rock/.local/lib/python3.10/site-packages (from sympy->onnxruntime) (1.3.0)
Installing collected packages: flatbuffers, humanfriendly, coloredlogs, onnxruntime
Successfully installed coloredlogs-15.0.1 flatbuffers-23.3.3 humanfriendly-10.0 onnxruntime-1.14.1
rock@rock5b:~/rockchip/RKNN/ultralytics$ python3 -m pip list | grep torch
torch                      2.0.0
torchvision                0.15.1
rock@rock5b:~/rockchip/RKNN/ultralytics$ yolo export model=yolov8s.pt imgsz=640,640 format=onnx opset=12
Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8s.pt to yolov8s.pt...
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 21.5M/21.5M [00:00<00:00, 33.8MB/s]
Ultralytics YOLOv8.0.69 πŸš€ Python-3.10.6 torch-2.0.0 CPU
YOLOv8s summary (fused): 168 layers, 11156544 parameters, 0 gradients, 28.6 GFLOPs

PyTorch: starting from yolov8s.pt with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (21.5 MB)

ONNX: starting export with onnx 1.13.1 opset 12...
================ Diagnostic Run torch.onnx.export version 2.0.0 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

ONNX: export success βœ… 2.1s, saved as yolov8s.onnx (42.8 MB)

Export complete (5.2s)
Results saved to /home/rock/rockchip/RKNN/ultralytics
Predict:         yolo predict task=detect model=yolov8s.onnx imgsz=640 
Validate:        yolo val task=detect model=yolov8s.onnx imgsz=640 data=coco.yaml 
Visualize:       https://netron.app
rock@rock5b:~/rockchip/RKNN/ultralytics$ find . -name *.onnx
./yolov8s.onnx
rock@rock5b:~/rockchip/RKNN/ultralytics$

Unfortunately, i can’t continue from here, if anyone wants to convert and share the final rknn in the proper platform, please do so, here is the file (mega.nz):
https://mega.nz/file/kbB3QCpC#7YB2Gnz5WsF4F-2oXBztK03SSiELrVRFmI4SadKgNPM

Thanks for sharing the instructions and thanks to everyone for interacting, please don’t take things personally, we’re here to have fun.

Let’s have a beer together one day, cheers.

@jack, @avaf I may consider to post the code, but this forum needs to exclude this raciest, LGBT-hate and bully @JonGroff first.

2 Likes

Y’all are funny where’s my popcorn :popcorn::joy::joy::joy::popcorn:

1 Like

Dropbox? file.io uploadnow.io maybe?

great. I’m looking forward to it.
i want to connect this to my camera rtsp stream and output an rtsp stream with boxes around objects that the model recognizes.

1 Like

Thanks so much for sharing @nickliu973!

I’ve been working on reproducing this following your instructions and am close but still running into some trouble.

Since others are interested in the models, I’ll share what I’ve got here in the hopes that it helps someone else.

I’m doing everything from Docker:

If you want to export the YOLOv8 (yolov8s) model in ONNX & RKNN format:

docker buildx bake \
  --set='*.context=https://github.com/milas/rock5-toolchain.git#rknn:rknn' \
  'https://github.com/milas/rock5-toolchain.git#rknn' \
  yolov8

That will result in the following in your current directory:

./out/rknn
β”œβ”€β”€ yolov8s.onnx
└── yolov8s.rknn

NOTE: This will ONLY work from an AMD64 host, as the β€œfull” rknn-toolkit2 only has a x86_64 Python wheel. The rknn-toolkit-lite2 is for ARM64/aarch64, but I haven’t tried using it currently.


I also built the rknn_benchmark and have an image with the yolov8s model and benchmark:

docker run --rm -it --ipc=host --privileged \
  docker.io/milas/rknn-benchmark:yolov8s \
  rknn_benchmark /sdk/models/yolov8s.rknn 5 7

NOTE: This will ONLY work on a Rock 5 (RK3588) device.

This is currently where I’m at, but it fails for me:

rknn_api/rknnrt version: 1.4.0 (a10f100eb@2022-09-09T09:07:14), driver version: 0.8.2
total weight size: 11893952, total internal size: 6150080
total dma used size: 26640384
model input num: 1, output num: 1
input tensors:
index=0, name=images, n_dims=4, dims=[1, 480, 640, 3], n_elems=921600, size=921600, w_stride = 640, size_with_stride=921600, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
output tensors:
index=0, name=output0, n_dims=4, dims=[1, 84, 6300, 1], n_elems=529200, size=529200, w_stride = 0, size_with_stride=529200, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=2.493242
custom string:
Warmup ...
0: Elapse Time = 69.47ms, FPS = 14.39
1: Elapse Time = 57.34ms, FPS = 17.44
E RKNN: [17:33:58.876] failed to submit!, op id: 12, op name: Add:/model.2/m.0/Add, flags: 0x5, task start: 3410, task number: 3, run task counter: 0, int status: 0
rknn run error -1

EDIT: This is what’s showing up in the kernel logs when it happens:

err: [2023-04-16T20:00:25.726217426Z]: RKNPU: failed to wait job, task counter: 0, flags: 0x5, ret = 0, elapsed time: 6135901us
err: [2023-04-16T20:00:25.832819426Z]: RKNPU: job timeout, flags: 0x0, irq status: 0x0, raw status: 0xc0000000, require mask: 0x300, task counter: 0x0, elapsed time: 6242507us
info: [2023-04-16T20:00:25.939628426Z]: RKNPU: soft reset

I saw different results on different runs, e.g. it sometimes makes it further than that but sometimes fails even earlier. I started trying different core masks and actually locked up the kernel at one point and had to power cycle :grimacing:


I’m still iterating on the Docker build setup; once I get things working and that improved, I’ll make a new thread with the details. If you’re feeling adventurous, take a look at the rknn/yolo targets in docker-bake.hcl on the rknn branch.

2 Likes

Hi, @milas.

Thanks for the message.
Sorry, it will take me a bit while to get back to this thread.
I did not try the rknn docker before.
I’m wondering if the docker is running in simulator, debug or physical model of the RK3588 NPU?

The message shows the output dimension is [84,6300] instead of the [84, 8400] which I was used.
Did you make any change to the model?

The error message implies the scheduling issue of the rknn runtime.
It indicates the add node in the model 2 of the neural network caused the problem which is weird as this data path should not cause the data sync problem.


The issue I came across in physical mode of the RK3588 NPU is the add model in the model.22 (post-processing stage of the bounding-box generation) and the issue is consistent in different runs.
And the solution is to remove the node which causes the issue and re-program it input the post-processing runtime.

Can you try to run the model in the physical mode and let me know the issue?

Running on a physical Rock 5B (retail version from Allnet).

I did not - using the stock model and trying to follow your instructions. I might have messed something up though, thanks for the tip - I will look at this again!

hi everyone. anybody made progress past the last error @nickliu973 mentioned?

@nickliu973 could you please share the v8 test.py file? thanks

What is this nice UI from a screenshot?

Hi, please check this web: https://netron.app/

1 Like

You can post your issue and I will help you.

I’m trying to follow the guide here and immediately i’m met with this:

image

I was able to install ultralytics using PIP but attempting to run anything related to Yolov8 in python is restarting the board. I’m using the ameridroid branded power adapter.