TensorRT Vision Compile

TensorRT compiles ONNX graphs into fused NVIDIA kernels with calibrated INT8, so detectors hit frame rate on Jetson and server GPUs.

ONNX builds a fused INT8 engine by calibration: 38 ms falls to 16 ms with mAP loss 0.8 points

Why Does This Exist?

NVIDIA GPUs hold tensor cores that sprint on INT8 but stall on loose Python graphs. Shipping raw ONNX leaves half the speed unused. TensorRT exists as the compile purpose: fuse layers, pick fast kernels per chip, calibrate INT8 scales, and emit an engine tuned to exact shapes. Input is ONNX plus calibration images. Output is a chip locked engine.

Exchange feeds it. ONNX carries the graph. Inference tuning sets shapes. This page is the NVIDIA compile with profiles and calibration.

Basics first: an engine binds to chip, batch, and size. A Jetson Orin engine will not run on a T4. Calibration sets INT8 scales from 500 representative images.

Think of It Like This

A race shop tune for one track

A stock car runs anywhere. A race shop tunes gearing, fuel, and aero for one track and one weather, then trailers the car there. TensorRT tunes kernels for one chip and one image size. The analogy stops at portability: the tuned car still drives elsewhere slowly, while the engine refuses other chips outright.

How It Actually Works

Parse ONNX at opset 17. Set optimization profiles for batch 1 to 8 and 320 by 320. Enable FP16 then INT8 with entropy calibration on 500 diverse frames. Build with workspace 2 GB. Serialize the plan file. At boot, deserialize in 2 seconds instead of rebuilding in 4 minutes. Verify mAP on 1,000 holdout frames.

Shape drift breaks the assumption. A 640 frame into a 320 engine fails or pads wastefully. Systems build one engine per deployed size and reject others at the gate.

Worked compile gain

ONNX runtime runs 38 ms. TensorRT FP16 runs 22 ms. INT8 runs 16 ms. Gain over ONNX is 38 divided by 16 which equals 2.375. mAP moves 0.412 to 0.404, a 0.008 drop. Calibration with only daylight drops night by 3 points, so the 500 must span day and night.

Watch Out For

Calibrating on daylight only

Day frames set narrow scales that clip night headlights into white blobs. The symptom is day parity with night misses. Fix it with balanced calibration and by checking per lighting splits, not global mAP.

The Quick Version

TensorRT fuses and calibrates ONNX into chip locked engines.
Profiles fix batch and size, calibration fixes INT8 scales.
Our 320 frame fell 38 ms to 16 ms, 2.38 times faster.
mAP held within 0.8 points on balanced calibration.
Build one engine per size or shape drift fails.