TVM Compiler Vision

TVM autotunes vision kernels per chip from a shared graph, so one model runs fast on odd CPUs, GPUs, and accelerators.

Relay graph searches 800 kernel trials per chip: 88 ms falls to 41 ms on ARM with tuned cache

Why Does This Exist?

Warehouses hold mixed chips: ARM cameras, x86 registers, old GPUs. Hand tuning each wastes quarters. TVM exists as the portable compile purpose: import ONNX into Relay, search kernel schedules per chip with Ansor, and ship tuned runtimes. Input is a graph plus target string. Output is a tuned library per chip.

Single vendor differs. TensorRT wins NVIDIA. OpenVINO wins Intel. This page is the heterogeneous route when fleets mix silicon.

Basics first: a schedule splits loops, tiles caches, and vectorizes. Search tries hundreds of schedules per operator and keeps the fastest measured, not predicted.

Think of It Like This

A tailor for mixed uniforms

One pattern must fit school, team, and work uniforms in different fabrics. The tailor measures each group once, cuts accordingly, and reuses patterns. TVM measures each chip once with trials, then reuses tuned kernels. The analogy stops at drift: bodies stay stable, while chip firmware updates can invalidate tuned timings.

How It Actually Works

Import ONNX to Relay with 320 fixed shape. Set target such as llvm for ARM with neon. Run Ansor with 800 trials on device, not host estimates. Export tuned library plus params. Cache by target string and graph hash. Verify numerical parity within 1e-4 on 100 frames before timing.

Host tuning breaks the assumption. x86 trial winners lose on ARM caches by 30 percent. Systems tune on device or on farm twins of the exact chip.

Worked tune run

Untuned Relay runs 88 ms on ARM A72. Tuned runs 41 ms. Gain is 88 divided by 41 which equals 2.15. Search costs 800 times 2 seconds which equals 1,600 seconds once, then each of 2,000 cameras saves 47 ms per frame for millions of frames.

Watch Out For

Shipping one tune to all chip steppings

A72 rev 2 cache differs from rev 3 and tuned tiles miss. The symptom is fleet variance with same binary. Fix it by keying tunes to chip stepping and by reusing nearest only afterbench.

The Quick Version

TVM searches kernel schedules per chip from Relay.
Tune on device with hundreds of trials, cache by target.
Our ARM run fell 88 ms to 41 ms, 2.15 times faster.
Verify numerics before trusting speed wins.
Key tunes to stepping or fleet variance surprises.