EfficientFormer Speeds Attention on Phones

EfficientFormer keeps 4D convs early and slim 3D attention late with latency driven sizing, so phones hit live ViT speed.

Pool mixers early plus attention late cut phone latency 3 times

Why Does This Exist?

Phone GPUs run convs fast but reshape heavy attention slowly. Plain ViTs miss live budgets by multiples.

EfficientFormer sizes by latency. It fits efficient architectures, rivals mobilevit sizing, and serves image classification live.

Think of It Like This

Bikes for alleys plus cars for highways

You bike narrow alleys where cars stall, then drive highways where bikes crawl. Each leg uses the faster vehicle.

Pool mixers are bikes, late attention cars. Where it stops: biking highways wastes time, just as pooling late loses accuracy.

How It Actually Works

Stages 1-2 use 4D MetaBlocks with pool mixing (no attention). Stages 3-4 flatten to 3D and run slim attention with small MLPs. Search widths by iPhone ms, not FLOPs.

Worked example

$56 \times 56$ pool mixing costs $3136$ ops linear. Late $14 \times 14 = 196$ attention costs $196^2 = 38$ K pairs. All-attention $56$ would cost $9.8$ M, about $256$ times more. Latency $12$ ms to $4$ ms per block in paper phone runs, $3$ times faster at $79$ percent.

Watch Out For

Sizing by FLOPs instead of phone ms

Pool versus attention FLOPs misrank phone speed. Symptom: low FLOP model slower on device. Fix: search with on-device latency as EfficientFormer does.

The Quick Version

EfficientFormer pools early in 4D and attends late in 3D.
Latency search picks widths for phones, not FLOPs.
Fixture late $38$ K pairs beat early $9.8$ M by 256 times.
Phone blocks drop $12$ ms to $4$ ms at $79$ percent.
Size by measured ms before publishing speed claims.