ONNX Vision Exchange

ONNX is the shared graph format that lets a PyTorch detector run in other runtimes without rewriting the model.

PyTorch graph exports to ONNX with NMS inside and runs anywhere: 120 ms Python falls to 38 ms runtime

Why Does This Exist?

Teams train in PyTorch but deploy on cameras, browsers, and phones that do not run Python training loops. Rewriting each detector per chip wastes months. ONNX exists as the exchange purpose: export once to a shared operator graph, then run in ONNX Runtime, TensorRT, or OpenVINO. Input is a trained detector. Output is a versioned file plus input shape contract.

Runtimes differ. TensorRT compiles for NVIDIA. OpenVINO tunes Intel. This page is the exchange step that feeds both, with opset, shapes, and NMS placement.

Basics first: a graph holds operators like Conv, Resize, and NMS with fixed versions called opsets. Newer opsets support more vision ops but older runtimes may refuse them.

Think of It Like This

A PDF for models

Word writes, any printer prints PDF. PyTorch writes, any runtime runs ONNX. Fonts embed like weights embed. The analogy stops at editability: PDFs reflow poorly, and ONNX graphs also resist training edits, so keep the training script as source.

How It Actually Works

Export with fixed batch 1 and size 320 by 320, opset 17 for NMS support. Keep NMS inside the graph so boxes filter on device. Verify by comparing 200 images: max box IoU drift under 0.001 and class match 100 percent. Then feed the file to the target compiler. Use dynamic batch only if the runtime builds explicit shape profiles.

Custom ops break the assumption. A novel activation with no ONNX mapping exports as a custom node that runtimes reject. Replace customs with standard composites before export or ship a plugin.

Worked export check

PyTorch baseline runs 120 ms per 320 frame with Python overhead. ONNX Runtime runs 38 ms on the same GPU. Speedup is 120 divided by 38 which equals 3.16. Of 200 checks, 199 match exactly and 1 shifts by 0.0006 IoU from rounding, a pass. Dynamic batch without profiles runs 44 ms, so fixed shapes stay.

Watch Out For

Resize mode mismatches that shift boxes

PyTorch bilinear align corners true versus ONNX false moves small boxes by 2 pixels. The symptom is export parity on classes with mAP down 3 points. Fix it by pinning resize mode at export and by checking box centers, not just labels.

The Quick Version

ONNX exchanges detectors across runtimes without rewrites.
Export fixed shapes with NMS inside at a known opset.
Our file cut 120 ms to 38 ms, 3.16 times faster.
Verify 200 images for box drift under 0.001.
Replace custom ops or runtimes will reject the graph.