Visual explainer

Intersection over Union and Mean Average Precision

Measure box overlap, match ranked detections, construct precision and recall, and distinguish interpolated AP from averages across classes and IoU thresholds.

Measure Shared Area

Intersection over union measures shared area relative to the combined area.

Intersection over union (IoU) measures localization overlap. Average precision (AP) summarizes how well confidence ranks correct detections across recall; mean AP averages those AP entries under a declared protocol. Neither is simply average confidence.

The solid box A spans (0,0) to (4,4) and dashed B spans (2,0) to (6,4). The hatch marks their intersection: 2×4=8. Each box has area 16; the union is 16+16−8=24, so IoU is 8/24=1/3. These are continuous box coordinates: do not add one to edge lengths.

Count The Union Once

The union includes both boxes without counting their intersection twice.

Shift B diagonally by (2,2) instead. The intersection becomes 2×2=4, giving union 16+16−4=28 and IoU 1/7. The mint perimeter follows the actual union. The dashed enclosing rectangle has area 36 and includes two empty corners; it is not the denominator. This beat changes the offset while retaining the four-unit box side.

Declare The Overlap Threshold

The same overlap passes a loose threshold and fails a stricter one.

For a new candidate with IoU 0.60, a threshold of 0.50 permits a match and 0.75 rejects it. The two bars have identical lengths; only the threshold moves. Matching also requires the right image and class and an eligible ground truth. An overlap cutoff is separate from the confidence cutoff used to retain predictions.

Match Each Object Once

Only the first eligible detection can claim an ordinary ground-truth object.

At a fixed IoU threshold, process predictions in descending confidence. Both candidates here overlap the same ordinary object sufficiently. Score 0.9 claims it as a true positive (TP); the later 0.8 duplicate becomes a false positive (FP). This isolates ordinary, nonignored instances. Crowd annotations and ignored regions need the evaluator's special rules. COCO's evaluator implements those distinctions.

Read The Ranked List

Confidence ranks detections before their true and false positives are accumulated.

Use two ground-truth objects and three predictions: scores [0.9,0.8,0.7], statuses [TP,FP,TP]. The last prediction finds the second object. Bar height encodes score; circles and the cross encode match status. A high score can be wrong. Evaluate this same list in the next three beats, with no crowd, ignore, area filtering or detection cap effects.

Accumulate Precision And Recall

A false positive lowers precision without increasing recall.

Precision is TP/(TP+FP) and recall is TP/2 for these two objects. After ranks 1, 2 and 3, (recall,precision) is (0.5,1), (0.5,0.5) and (1,2/3). The second point moves straight down: adding a false positive finds no new object. Dashed segments show rank order, not a prescribed interpolation between operating points.

Form The Precision Envelope

Interpolated precision takes the best precision available at each recall or higher.

For each recall level r, take the maximum measured precision among points with recall at least r. That is 1 through recall 0.5, then 2/3 through recall 1. The low second point is below this envelope. A continuous area convention gives 0.5×1 + 0.5×2/3 = 5/6; a sampled convention can give a slightly different result. Always name the AP convention rather than treating every PR area calculation as interchangeable.

Sample COCO Recall Levels

COCO averages interpolated precision at 101 recall thresholds.

COCO uses recall levels 0,0.01,…,1. In this fixture, 51 levels through 0.50 receive precision 1 and the remaining 50 receive 2/3. Their mean is (51 + 50×2/3)/101 = 253/303 ≈ 0.8349835. The drawing shows every tenth tick for legibility; brackets name the full sample counts. This is one class at one IoU threshold, not the complete COCO summary.

Average The AP Entries

Mean average precision gives equal weight to the evaluated class entries.

A separate two-class example has AP values 0.8 and 0.4; their mean is (0.8+0.4)/2=0.6. Class frequency does not set bar width or averaging weight. COCO's headline AP also averages over IoU thresholds 0.50,0.55,…,0.95, with specified area and detection limits. AP50 fixes only the 0.50 threshold. Categories without evaluable ground truth are excluded rather than assigned an invented zero. Keep the full protocol with any comparison.

Where It Breaks

$The same pixel shift removes a larger fraction of overlap from a small object.$

The same pixel shift removes a larger fraction of overlap from a small object.

Shift a four-by-four box one pixel horizontally: intersection 12, union 20, IoU 0.6. Shift a twenty-by-twenty box by the same pixel: intersection 380, union 420, IoU 19/21≈0.905. Both panels use ten drawing units per image pixel. At threshold 0.75 the small prediction fails while the large one passes, despite the same absolute displacement.

An aggregate mAP can hide poor small-object performance, class failures or annotation uncertainty. Inspect size and class slices alongside the summary. Box IoU also says nothing about the pixels inside the box; instance masks address that different output.

The Quick Version

IoU divides intersection area by union area.
Union excludes empty corners of an enclosing rectangle.
Matching thresholds change acceptance without changing overlap.
Duplicate detections cannot repeatedly claim an ordinary object.
Confidence ranks predictions before match statuses are accumulated.
False positives lower precision without increasing recall.
Interpolated precision uses the best remaining precision.
COCO averages precision at 101 recall thresholds.
Mean AP depends on classes, overlap thresholds and evaluation settings.
Small boxes can fail overlap thresholds after tiny absolute shifts.