Pyramids Detect Every Scale Once

A feature pyramid reuses the backbone once and fuses strong low-resolution semantics into high-resolution maps, so one detector reads tiny and huge objects together.

A backbone runs once while a top down path upsamples strong semantics and adds them into high resolution maps, so small and large heads read together

Why Does This Exist?

A backbone built for classification outputs one low-resolution summary, which answers what is present but not where each size lives. A 12-pixel bird vanishes in a stride-32 map while a 300-pixel bus overflows a stride-8 map, so no single level serves both. The old fix ran the whole backbone on three resized copies of the image, paying roughly three times the compute.

A feature pyramid (Lin et al., 2017) runs the backbone once and builds the scales inside the network. A top-down path upsamples semantically strong maps and adds them into high-resolution lateral maps, giving every level both detail and meaning. The same Faster R-CNN or RetinaNet heads then read small objects off fine levels and large objects off coarse levels in that single pass.

Think of It Like This

An architect's drawing set

An architect never draws a door handle on the site plan or the whole building on the hinge detail sheet. The site plan carries context at coarse scale and the detail sheet carries edges at fine scale, and red markup notes flow from the site plan down onto each detail sheet so every page stays oriented.

The bottom-up backbone draws the sheets, the top-down path carries the markup downward, and the lateral 1x1 connections are the registration marks aligning them. Where it stops: markup cannot invent a door nobody measured, just as upsampling cannot invent a 12-pixel bird the fine map never resolved.

How It Actually Works

The bottom-up backbone yields $C_3$ , $C_4$ , $C_5$ at strides 8, 16, and 32. Each lateral 1x1 convolution projects its map to 256 channels, the top map sets $P_5$ , and each lower level adds its lateral map to the 2x upsampled parent: $P_4 = \text{lateral}(C_4) + \text{up}(P_5)$ , then a 3x3 convolution smooths aliasing. Heads attach per level, so anchors on $P_3$ stay small and anchors on $P_5$ grow large, and each object is assigned to the level whose resolution fits it.

Worked example

Take an $800 \times 800$ image. Grids are $800 / 8 = 100$ for $P_3$ , $800 / 16 = 50$ for $P_4$ , and $800 / 32 = 25$ for $P_5$ . A 12-pixel bird spans $12 / 8 = 1.5$ cells on $P_3$ but only $12 / 32 = 0.38$ cells on $P_5$ , so it is readable on $P_3$ and a smudge on $P_5$ . A 300-pixel bus spans $300 / 32 \approx 9.4$ cells on $P_5$ , a comfortable window. The $C_5$ lateral maps $2048 \times 256$ with $2048 \times 256 = 524{,}288$ weights, and one backbone pass replaces three image-pyramid passes.

Code

The point of this snippet is the grid arithmetic, not a network: pure Python recomputes the level sizes, the cell spans, and the lateral weight count.

1size = 8002for stride in (8, 16, 32):3    print(size // stride)4# -> 1005# -> 506# -> 257
8print(round(12 / 8, 2), round(12 / 32, 2), round(300 / 32, 2))9# -> 1.5 0.38 9.3810
11print(2048 * 256)12# -> 524288

$P_3$ offers a 100-wide grid where the bird covers 1.5 cells, $P_5$ offers a 25-wide grid where the bus covers 9.38 cells, and the lateral projection costs 524288 weights.

Watch Out For

Attaching heavy heads to every level for free

The symptom is inference that slows sharply after adding the pyramid, because five heads each score thousands of anchors. Fix it by sharing lightweight heads across levels and tuning per-level anchor counts before blaming the backbone.

Expecting upsampling to rescue invisible objects

The symptom is small-object AP that barely moves after adding the pyramid on a low-resolution input. Fix it by checking that $P_3$ actually resolves the object: a 12-pixel target on stride 8 is marginal, and no top-down semantics can invent detail the fine map never captured.

The Quick Version

One backbone pass yields strides 8, 16, and 32 with grids 100, 50, and 25 on an 800-pixel image.
Lateral 1x1 maps project each level to 256 channels, costing $524{,}288$ weights at $C_5$ .
Top-down 2x upsampling plus addition gives every level both detail and semantics.
The 12-pixel bird reads on $P_3$ at 1.5 cells, the 300-pixel bus reads on $P_5$ at 9.38 cells.
Heads attach per level, replacing three backbone passes with one.