PVT Builds Pyramids for Transformers

PVT shrinks keys with spatial reduction so transformers output pyramid maps, fitting dense tasks like detection and segmentation.

Spatial reduction cuts 3136 keys to 49, so attention drops 64 times while keeping pyramids

Why Does This Exist?

Plain ViT outputs one $14 \times 14$ map. Detection needs $56 \times 56$ down to $7 \times 7$ pyramids, but full attention on $56 \times 56$ costs $3136^2$ pairs.

PVT keeps pyramids affordable. It feeds object detection and semantic segmentation backbones past image classification only use.

Think of It Like This

Reading a map at four zoom levels

You keep country, region, city, and street maps of one area. For a country question you scan the small map, for a street question you open the large one.

Stages are zooms, reduction is folding the large map before comparing. Where it stops: folding blurs alleys, just as reduction blurs tiny objects.

How It Actually Works

Four stages output strides $4$ , $8$ , $16$ , $32$ . Stage $i$ has $H_i W_i$ queries. Spatial reduction with ratio $R_i$ reshapes keys to $H_i/R_i \times W_i/R_i$ , so keys drop by $R_i^2$ . Attention is $softmax(QK^T/\sqrt{d})V$ with $K$ reduced.

Worked example

Stage 1 on $224$ input gives $56 \times 56 = 3136$ tokens. Full pairs are $3136^2 = 9{,}834{,}496$ . With $R = 8$ , keys reshape to $7 \times 7 = 49$ . Pairs drop to $3136 \times 49 = 153{,}664$ . Ratio $9{,}834{,}496 / 153{,}664 = 64$ . Four stages then emit $56$ , $28$ , $14$ , $7$ maps for dense heads.

Watch Out For

Setting reduction so high small objects vanish

Ratio $8$ on early stages erases $4$ -pixel lights. Symptom: good classification with weak small AP. Fix: use smaller $R$ early ( $8,4,2,1$ schedule from the paper) and keep high resolution for small splits.

The Quick Version

PVT emits four pyramid maps at strides 4, 8, 16, and 32 for dense tasks.
Reduction ratio $R$ cuts keys by $R^2$ before attention.
Fixture 3136 to 49 keys cuts pairs 64 times from 9.8M to 0.15M.
Early stages need smaller $R$ to protect tiny objects.
Pyramids let transformers replace CNN backbones directly.