Has anyone achieved F1 close to ONNX after quantizing custom-trained model to axmodel on AX-M1?

1

Hi,

I’ve been trying to deploy several custom-trained models on AX650N using Pulsar2 4.2, but all of them show significant F1 drop after INT8 quantization (ranging from -27% to -51%).

I’ve already tried: - Balanced calibration dataset - MSE calibration method - Mixed precision (FP32 for sensitive layers) - Larger calibration size

Models tested: YOLO11, YOLOv8, YOLO26, RT-DETR, MobileNetV2

Has anyone managed to get axmodel F1 within ~5-10% of the original ONNX on a custom dataset? If so, what was your approach?

Thank you

2

Hi, @Pokemon_Go_playonthe

Did you remove the postprocessing node in your custom trained onnx model? If not, the accuracy will drop a lot.

you could have a look how to remove the postprocessing node in this link:

after that you can use Pulsar2 to convert it

best,
Morgan

1 Like