MaxViT Splits Attention Into Grids

MaxViT alternates blocked local attention with dilated grid attention, so every layer sees detail and far layout at linear cost.

Blocked windows handle detail while dilated grids connect far cells at window cost

Why Does This Exist?

Window attention is cheap but blind beyond its pane. Full attention sees all but costs quadratic maps.

MaxViT gets both. It builds on convolutional layers, serves image classification backbones, and feeds object detection pyramids.

Think of It Like This

Neighbors then district reps

Each street meets locally, then one rep per street meets on a city grid. Local needs settle fast and city patterns still spread.

Blocks are streets, grids are rep meetings. Where it stops: reps compress detail, just as grid attention samples sparsely.

How It Actually Works

Each MaxViT block has MBConv, blocked attention, then grid attention. Blocked splits $H \times W$ into $P \times P$ windows ( $P = 7$ ) with $49$ tokens each. Grid samples every $G$ -th cell into $G \times G$ groups, linking far spots. Both cost linear in pixels.

Worked example

$56 \times 56 = 3136$ tokens split into $(56/7)^2 = 64$ windows of $49$ . Blocked pairs are $64 \times 49^2 = 64 \times 2401 = 153{,}664$ . Grid with $G = 8$ makes $64$ groups of $49$ for the same $153{,}664$ . Sum $307{,}328$ versus full $9.8$ M, about $32$ times cheaper with global reach.

Watch Out For

Using tiny windows that starve the grid

Window $4$ fragments textures and grid links carry too little. Symptom: noisy small maps. Fix: keep $7$ or $8$ windows as in the paper before tuning grids.

The Quick Version

MaxViT alternates blocked local and dilated grid attention plus MBConv.
Blocked handles $49$ -token detail, grid links far cells.
Fixture $307$ K pairs replace $9.8$ M, about 32 times cheaper.
Both steps stay linear in pixel count.
Keep windows near 7 to protect texture.