Hierarchical Transformers Shrink Maps

Hierarchical transformers halve maps while doubling channels, giving CNN-like pyramids that dense heads reuse directly.

Four stages halve maps 56 to 7 while doubling channels for dense heads

Why Does This Exist?

Single-scale ViT forces detectors to upsample one map. Small and large objects need different resolutions natively.

Hierarchies fix it. They generalize convolutional layers staging, back object detection and semantic segmentation pyramids.

Think of It Like This

Photo prints from contact sheet to poster crop

You keep wallet, print, poster, and billboard crops of one negative. Small faces need the large print, crowds need the small one.

Stages are print sizes, channels are ink depth. Where it stops: too many enlargements blur grain, just as too many merges blur detail.

How It Actually Works

Merge $2 \times 2$ neighbors per stage, halving $H,W$ and doubling $C$ . Apply windowed or reduced attention per stage to hold linear cost. Emit four maps for feature pyramids.

Worked example

Start $56 \times 56 \times 96 = 301{,}056$ values. Stage 2 is $28 \times 28 \times 192 = 150{,}528$ . Stage 3 $14 \times 14 \times 384 = 75{,}264$ . Stage 4 $7 \times 7 \times 768 = 37{,}632$ . Each halves values while doubling depth, so total stays near $564$ K with four usable scales.

Watch Out For

Merging without overlap on thin structures

Non-overlapping $2 \times 2$ merges split one-pixel lines across stages. Symptom: wire or pole gaps in masks. Fix: use overlapping or padded merges when thin structures matter.

The Quick Version

Hierarchical stages halve resolution while doubling channels per step.
Four maps at 56, 28, 14, and 7 feed pyramid heads directly.
Fixture totals stay near 564K values across stages.
Windowed attention keeps each stage near linear cost.
Overlap merges when thin lines must survive.