MViT Pools Attention Across Scales

MViT pools keys and values while widening channels stage by stage, so transformers shrink maps like CNNs for video and images.

Pooled keys shrink 3136 to 196 while channels double for cheap pyramids

Why Does This Exist?

ViT keeps width and resolution fixed, so video tokens explode. CNNs shrink space while growing channels, which video needs.

MViT copies that schedule. It extends convolutional layers staging, backs image classification pyramids, and scales to video via pretrained models.

Think of It Like This

Summary notes that get shorter but denser

Early lecture notes copy full sentences. Later pages condense to keywords but use richer terms. Length drops while meaning per line rises.

Pooling is condensing, channel growth is richer terms. Where it stops: over-condensing loses names, just as heavy pooling drops small motion.

How It Actually Works

Each stage pools $K$ and $V$ with stride $s$ via conv or maxpool, while $Q$ keeps full length early then pools later. Channels double as resolution halves. Attention is $softmax(QK_{pool}^T/\sqrt{d})V_{pool}$ plus residual pooling on $Q$ for shape match.

Worked example

Stage 1 has $56 \times 56 = 3136$ queries. Pool $K$ stride $8$ to $7 \times 7 = 49$ ? MViT uses stride $4$ here to $14 \times 14 = 196$ . Pairs are $3136 \times 196 = 614{,}656$ versus $9.8$ M full, about $16$ times cheaper. Channels grow $96$ to $192$ to $384$ to $768$ while maps shrink, matching ResNet logic.

Watch Out For

Pooling queries and keys with the same stride

Pooling $Q$ early throws away positions dense heads need. Symptom: coarse boxes only. Fix: pool $K,V$ aggressively but keep $Q$ full until later stages as the paper schedules.

The Quick Version

MViT pools $K,V$ while widening channels per stage like CNNs.
Queries stay fuller early, keys shrink fast.
Fixture $3136 \times 196$ costs 16 times less than full $9.8$ M.
Channels $96$ to $768$ grow as maps $56$ to $7$ shrink.
Pool $K,V$ first and $Q$ later to protect locations.