COCO Detection Segmentation

Everyday scenes with boxes, masks and captions per image, the default ground for detection and segmentation work.

COCO pairs each everyday photo with boxes, masks and captions, so one image trains detection, segmentation and captioning

Why Does This Exist?

Classification labels one photo. Real scenes hold many objects at many sizes, partly hidden, with context that matters. COCO gathered everyday photos and labeled boxes, instance masks, keypoints and captions on the same images, so detection and segmentation share one ground. Scoring uses mAP across thresholds and sizes.

This page covers task family, I/O roles and appropriate use. Box matching math is in IoU.

Think of It Like This

A family photo with everyone tagged

One holiday photo holds kids, dogs, chairs and cake, some half hidden behind others. You draw a box around each person, trace each dog exactly, and write one caption for the scene.

That triple annotation is COCO. Boxes for finding, masks for tracing, captions for describing, all on the same picture.

How It Actually Works

Purpose spans detection, instance segmentation, keypoints and captioning. Input is one natural scene. Outputs are box lists, mask polygons and caption strings depending on the task. Appropriate use is benchmarking detectors and segmenters on everyday scenes. Small objects and crowded scenes are the known hard slices, so report by size rather than one headline alone.

Toy counting walkthrough

Take 10 scene photos with 3 labeled objects each, so 30 boxes total. A detector returns 27 boxes, 24 matching truth at the chosen IoU line. Precision is $24 / 27 \approx 0.889$ and recall is $24 / 30 = 0.8$ . The gap names the failure: mostly misses rather than false alarms.

Code

Count matches the way scoring starts, before averaging over thresholds.

1import numpy as np2
3
4def pr_counts(matched: int, returned: int, total: int) -> tuple[float, float]:5    prec = matched / max(returned, 1)  # of what you returned, how much was right6    rec = matched / max(total, 1)  # of what existed, how much you found7    return round(prec, 3), round(rec, 3)8
9
10print(pr_counts(24, 27, 30))11# -> (0.889, 0.8)

Precision near 0.889 with recall 0.8 points at misses first.

Watch Out For

Tuning to one threshold on crowded scenes

A single IoU line hides small-object misses behind large-object hits. The symptom is a good headline with weak crowd performance. Fix it by reporting across thresholds and sizes, as COCO-style averaging does.

The Quick Version

COCO pairs one scene photo with boxes, masks and captions for several tasks at once.
Inputs are scenes, outputs are box lists, masks or captions by task.
Toy 24 of 27 returned and 30 total gives precision about 0.889 and recall 0.8.
Use it to benchmark detectors and segmenters on everyday scenes.
Report by size and threshold, because crowds and small objects fail first.