Anchor Boxes

Preset box shapes tiled across the image give the detector a starting guess, so it only predicts small nudges instead of raw coordinates from scratch.

Nine preset templates per grid cell compete for each object, and the network predicts small nudges that turn the best-fitting template into the final box

Why Does This Exist?

Asking a network for four raw numbers, $(x, y, \text{width}, \text{height})$ , anywhere in a high-resolution image is an unstable task. Targets range from 10-pixel specks to 1000-pixel giants in any corner, and the gradients thrash across those scales. Training converges slowly, if at all.

Anchor boxes reframe the question. Tile the image with thousands of preset reference shapes, then ask a much easier question per box: is there an object in this one, and what small nudge would fit it exactly? Predicting bounded offsets from a roughly-right template stabilizes the gradients, which is why this idea sits inside Faster R-CNN, SSD, and YOLO alike.

This page covers the template mechanism: tiling, matching, offset math, and decoding. The detection pipeline around it is object detection, the cleanup step afterward is non-maximum suppression, and the movement away from templates is anchor-free detection.

The one idea you need first. Intersection over Union scores box overlap: area shared divided by area covered, from 0 (disjoint) to 1 (identical). Anchors use it both to claim objects during training and to delete duplicates at inference.

Think of It Like This

Buying a suit off the rack

A tailor keeps a rack of ready-made suits: small, medium, large, tall, wide. When you walk in, she does not cut fabric from a raw bolt. She finds the rack suit closest to your build, then alters only the difference: an inch in at the waist, half an inch out at the cuff.

Each part maps. The rack suits are the anchor shapes, your body is the ground-truth object, picking the closest suit is IoU matching, and the alterations are the predicted offsets. Small alterations are fast and safe; draping a whole suit from nothing is slow and error-prone.

Where it stops: the tailor measures you directly, while the network never sees the object except through the anchor it was assigned. A customer between two rack sizes still gets a wearable suit; an object between two anchor shapes can get missed entirely, which is why anchor shapes are tuned to the dataset.

How It Actually Works

The image is divided into a grid matching the backbone's feature map, say $13 \times 13$ cells. Each cell center carries several anchors spanning scales and aspect ratios such as 1:1, 1:2, and 2:1. With 3 scales times 3 ratios, that is 9 anchors per cell, or $13 \times 13 \times 9 = 1521$ boxes scored across the image.

During training, each ground-truth object is claimed by the anchor overlapping it best: that anchor is positive, anchors below an IoU around 0.3 are background, and the band between is ignored by the loss. For a positive anchor with center $(x_a, y_a)$ , width $w_a$ , and height $h_a$ , and a true box $(x, y, w, h)$ , the network learns offset targets, not coordinates:

t_x = \frac{x - x_a}{w_a}, \quad t_y = \frac{y - y_a}{h_a}

t_w = \log\left(\frac{w}{w_a}\right), \quad t_h = \log\left(\frac{h}{h_a}\right)

In plain words: $t_x$ and $t_y$ say how many anchor-widths to shift the center, while the logarithms turn width and height scaling into symmetric targets, so stretching 10 percent and shrinking 10 percent are equal and opposite. At inference the equations run in reverse: predicted offsets warp each static anchor into a box, and overlapping winners survive non-maximum suppression.

Worked example: one anchor becomes a prediction

Anchor center $(150, 150)$ , width $50$ , height $100$ . The network predicts offsets $(0.1, -0.2, 0.0, 0.1)$ :

x = 150 + 0.1 \times 50 = 155, \quad y = 150 - 0.2 \times 100 = 130

w = 50 \times \exp(0.0) = 50.0, \quad h = 100 \times \exp(0.1) \approx 110.52

The prediction is $(155, 130, 50.0, 110.52)$ : nudged right, pulled up, width untouched, height stretched about 10 percent. Note how ordinary the network's outputs are: four small numbers near zero, exactly the regime gradients like.

Code

The point of this snippet is the decoding step: how four small network outputs warp a static anchor into a final box.

1import numpy as np2
3anchors: np.ndarray = np.array([[150.0, 150.0, 50.0, 100.0]])  # x, y, w, h4offsets: np.ndarray = np.array([[0.1, -0.2, 0.0, 0.1]])  # tx, ty, tw, th5
6x_a, y_a, w_a, h_a = anchors[:, 0], anchors[:, 1], anchors[:, 2], anchors[:, 3]7tx, ty, tw, th = offsets[:, 0], offsets[:, 1], offsets[:, 2], offsets[:, 3]8
9pred: np.ndarray = np.stack(10    [x_a + tx * w_a, y_a + ty * h_a, w_a * np.exp(tw), h_a * np.exp(th)],11    axis=-1,12)13print(np.round(pred, 2))  # -> [[155.   130.    50.   110.52]]

A $t_w$ of $0.0$ leaves the width at $50.0$ , while $t_h = 0.1$ stretches the height through $\exp(0.1) \approx 1.105$ , the same numbers the worked example derived by hand.

Watch Out For

Anchor shapes that ignore the dataset

Square anchors plus horizontal priors on a pedestrian dataset force every prediction through enormous offsets, defeating the whole purpose. The symptom is slow convergence and ragged boxes on the dominant shape, for example tall thin people detected with clipped heads.

Fix it by clustering the training boxes, typically k-means on widths and heights, and setting anchor scales and ratios from the cluster centers before training starts. Re-run the clustering whenever the dataset changes: anchors tuned for COCO rarely suit aerial or shelf imagery.

Background anchors drowning the loss

A dense tiling scores ten thousand boxes of which a handful hold objects. Averaged naively, background gradients swamp the positives and the model learns to predict nothing everywhere. The symptom is high loss that barely moves plus near-zero recall.

Fix it with focal loss, which down-weights easy background boxes, or hard negative mining, which caps the negative-to-positive ratio in each batch. One of the two is standard in every anchor-based detector, including the YOLO architecture.

The Quick Version

Anchor boxes are preset reference shapes tiled across a grid, so the network predicts small offsets instead of raw coordinates.
A 13 by 13 grid with 9 anchors per cell scores 1521 boxes, spanning scales and aspect ratios at every location.
Each ground-truth object trains the anchor overlapping it best, matched by Intersection over Union.
Offsets are proportional center shifts plus log-scale size multipliers, which keep targets small and symmetric.
Decoding inverts the math at inference, and non-maximum suppression removes the duplicates.
Tune anchor shapes to the dataset by clustering, and counter background imbalance with focal loss or hard negative mining.