Region Proposal Networks

A Region Proposal Network scans the shared feature map with ready-made anchor boxes and flags the few regions worth a closer look, so the detector never slides over empty background.

Why Does This Exist?

Early detectors found objects by brute force: sliding windows or selective search evaluated thousands of candidate boxes per image, most of them empty background. That proposal step ran outside the network and dominated runtime, so high-accuracy detection couldn't run anywhere near real time.

The Region Proposal Network (RPN), introduced with Faster R-CNN, moved proposing inside the network. It reuses the backbone's feature map, scores a set of ready-made anchor boxes at every spot, and hands only the promising regions to the second stage. Proposing got nearly free, and two-stage detection became practical.

This page covers the anchors, the two prediction heads, and the cleanup step. The full detector that consumes the proposals is Faster R-CNN; the box format itself is on the anchor boxes page.

Think of It Like This

The gallery scout with sticky notes

An appraiser must value paintings in a huge cluttered warehouse. Walking every aisle with a flashlight takes forever, so a scout runs ahead and sticks glowing notes only where something painting-like hangs, ignoring blank walls. The appraiser then visits just the notes and does the careful work of naming each painting and tracing its frame.

The scout is the RPN and the appraiser is the second stage. The analogy stops where the scout's ignorance starts: the scout never names a painting, and the RPN never outputs a class, only object-or-background plus a box fix.

How It Actually Works

The anchors

The RPN looks at the backbone feature map, not raw pixels. A small $3 \times 3$ window slides over that map, and at every position it evaluates $k$ predefined anchor boxes spanning scales and aspect ratios (3 scales $\times$ 3 ratios gives $k = 9$ ). Here $k$ is just the anchor count per spot. Anchors are guesses about shape, not predictions; the network's job is to score and adjust them.

The two heads

Two tiny sibling layers, usually $1 \times 1$ convolutions, read each window. The classification head outputs an objectness score per anchor: the probability it holds any object versus background, with no notion of dog, car, or person. The regression head outputs four offsets per anchor, $(\Delta x, \Delta y, \Delta w, \Delta h)$ , shifting and resizing the anchor onto the object. Offsets use center-x, center-y, width, height, each normalized so one head works for all anchor sizes.

Training labels each anchor by IoU (intersection over union) with the ground-truth boxes: IoU above about $0.7$ is positive, below about $0.3$ is negative, and anything between is ignored. The loss is a multi-task sum, and the regression term counts only for positive anchors, since empty background has no box to fit.

Worked example: labeling three anchors by IoU

Take a ground-truth box $G = [0, 0, 10, 10]$ with area $100$ , where coordinates are $[x1, y1, x2, y2]$ . Score three anchors:

$P = [0, 0, 9, 9]$ , area $81$ , fully inside $G$ . Intersection $81$ , union $100$ , IoU $= 81/100 = 0.81$ . Positive: it trains both heads.
$M = [3, 3, 13, 13]$ , area $100$ . Intersection $[3, 3, 10, 10]$ has area $7 \times 7 = 49$ ; union $100 + 100 - 49 = 151$ ; IoU $= 49/151 = 0.3245$ . Ignored: it trains neither head.
$N = [20, 20, 30, 30]$ , no overlap. IoU $= 0.0$ . Negative: it trains only the objectness head toward background.

For the positive anchor, the regression target is computed from centers and sizes. $P$ has center $(4.5, 4.5)$ , width $9$ ; $G$ has center $(5, 5)$ , width $10$ . The targets are $t_x = (5 - 4.5)/9 = 0.0556$ and $t_w = \log(10/9) = 0.1054$ , with $t_y$ matching $t_x$ and $t_h$ matching $t_w$ .

Overlapping survivors are then trimmed with non-maximum suppression: group overlapping boxes, keep the highest objectness score, drop the rest. Thousands of raw proposals shrink to roughly two thousand worth classifying.

Code

1def iou(a, b):2    x1, y1 = max(a[0], b[0]), max(a[1], b[1])3    x2, y2 = min(a[2], b[2]), min(a[3], b[3])4    inter = max(0, x2 - x1) * max(0, y2 - y1)5    area_a = (a[2] - a[0]) * (a[3] - a[1])6    area_b = (b[2] - b[0]) * (b[3] - b[1])7    return inter / (area_a + area_b - inter)8
9def label(score):10    if score >= 0.7:11        return "positive"12    if score < 0.3:13        return "negative"14    return "ignored"15
16g = [0, 0, 10, 10]17for name, box in [("P", [0, 0, 9, 9]), ("M", [3, 3, 13, 13]), ("N", [20, 20, 30, 30])]:18    s = round(iou(g, box), 4)19    print(name, s, label(s))20# -> P 0.81 positive21# -> M 0.3245 ignored22# -> N 0.0 negative

Watch Out For

Default anchors that miss your objects

COCO-sized default anchors suit people and cars, not tiny ships in satellite images. The symptom is proposals that never tighten no matter how long you train, because no anchor starts near the true shape. Plot your ground-truth box sizes and ratios first, then set anchor scales to match; clustered box dimensions beat defaults.

Expecting the RPN to name the object

The classification head is binary, object versus background, so a high objectness score says nothing about which class is inside. The symptom is code that reads RPN scores as detections and reports "dog 0.99" for a box the second stage would call background. Treat RPN output as regions to examine, and take class names only from the second-stage head.

The Quick Version

The RPN reuses the backbone feature map to propose regions, replacing slow selective search.
Nine anchors per spot cover the usual scale and aspect-ratio guesses.
The objectness head scores foreground versus background; it never names a class.
The regression head outputs four normalized offsets per anchor, trained on positives only.
IoU decides each anchor's fate: about 0.7 and up is positive, about 0.3 and below is background.
Non-maximum suppression keeps the best box per overlap group before stage two.