FPGA ASIC Vision Accelerators

Why Does This Exist?

GPUs win on flexibility and lose on determinism. A traffic camera that must answer every frame in 20 ms cannot tolerate driver jitter, batching waits, or thermal clocks. FPGAs and ASICs exist for that corner: the CNN becomes a fixed pipeline of multiply accumulates, line buffers, and on chip weights, with one pixel per clock after fill.

Inputs are tools, not frameworks. FINN, hls4ml, Vitis AI, and vendor compilers take ONNX and emit streaming RTL or bitstreams. Operator fusion is automatic here since Conv plus BatchNorm plus ReLU become one stage. Inference optimization still sets the model size first, since gates cannot page to DRAM mid frame for free.

Basics first: resources are DSP slices for MACs, BRAM for line buffers and weights, and LUTs for control. Clocks sit near 150 to 300 MHz. Power stays near 3 to 10 watts for cameras. Recompiles take hours, not seconds.

Think of It Like This

How It Actually Works

Unroll to meet the deadline. A 3 by 3 INT8 conv needs 9 MACs per output pixel. Unrolling 32 output channels needs 288 MACs since 32 times 9 equals 288. At 150 MHz that stage gives 43.2 GMAC per second since 288 times 150 million equals 43.2 billion. A 1 GMAC detector then clears in 23.1 ms since 1 divided by 43.2 equals 0.0231 seconds. Add line buffer depth: a 3 by 3 over 320 width needs 2 times 320 pixels buffered per channel.

Fix the size at synthesis. Height, width, channels, and INT8 scales become constants in gates. A new resolution rebuilds the bitstream. Keep one validated size per camera SKU and version bitstreams like models, since a 320 build will not run 640 inputs.

Stream, never batch. Pixels enter over AXI Stream, fill line buffers once, then exit as boxes with frame sync. There is no batch queue to amortize, which is why latency is deterministic within a clock plus DRAM stall budget.

Worked gate math

MACs needed are 288 since 32 times 9 equals 288. Throughput is 43.2 GMAC per second since 288 times 150 million equals 43,200 million. A 1 GMAC model needs 23.1 ms since 1000 divided by 43.2 equals 23.15. Line buffer is 640 pixels since 2 times 320 equals 640 per channel stage.

Code

The point is the unroll check before synthesis. It recomputes MACs, throughput, and frame time.

1import numpy as np2
3
4def stage_gmac_per_s(macs: float, clock_mhz: float) -> float:5    return macs * clock_mhz / 1000.0  # 1 MAC per clock per unit6
7
8def frame_ms(gmac_model: float, gmac_per_s: float) -> float:9    return 1000.0 * gmac_model / gmac_per_s  # model size over throughput10
11
12print(32 * 9)  # -> 28813print(stage_gmac_per_s(288.0, 150.0))  # -> 43.214print(frame_ms(1.0, 43.2))  # -> 23.148148148148145

Size the stage for worst case light plus worst case density. Average case math misses night rain frames that fill edge buffers.

Watch Out For

The Quick Version

• FPGA and ASIC fix CNNs into gate pipelines for deterministic low watt latency.
• Unroll channels to meet math: 32 times 9 is 288 MACs per stage.
• At 150 MHz that is 43.2 GMAC per second, so 1 GMAC clears in 23.1 ms.
• Fix input size at synthesis and version bitstream with model plus scales.
• Use gates when every frame needs the same deadline on watts, not for flexible fleets.

What to Read Next

• Inference Optimization Vision
• Operator Fusion Vision
• ONNX Vision Exchange