Data Labeling Tools

Labeling tools turn raw photos into trusted training data with drawing UX, double review, and agreement scores that decide what ships.

One thousand boxes pass three annotators at 0.82 IoU agreement and only the 180 disputed ones reach expert review

Why Does This Exist?

Models learn whatever the labels say, including the mistakes. A box drawn loosely around two cars teaches that two cars are one object. At scale these slips dominate errors more than architecture choices, yet teams often treat labeling as unskilled clicking in a generic tool.

Labeling tools exist to make quality measurable. They provide the right drawing mode per task, route work through reviewers, and score agreement so bad batches never reach training. This page covers the tooling: queues, UX, consensus, and export. Label quality practice itself lives in labelling and annotation, and dataset strategy lives in data-centric AI.

Think of It Like This

A print shop with proofreaders

Writers draft pages, proofreaders mark corrections, and only flagged pages return to the author. Nobody re-reads clean pages twice.

Photos are drafts, annotators are writers, and reviewers are proofreaders. Agreement scores decide which pages count as clean. The analogy stops at skill: proofreaders need language, while box reviewers need to see tight edges at a glance.

How It Actually Works

Work flows in four stages. Import batches images with task templates: boxes for detection, polygons for segmentation, keypoints for pose. Annotators draw with shortcuts like edge snapping and interpolation across video frames. A second pass reviews by sampling or by full double labeling on hard slices. Finally the tool exports versioned JSON with annotator IDs, timestamps, and agreement stats attached.

Worked example

One batch holds $1{,}000$ boxes across $400$ street images, each drawn by $3$ annotators. Mean pairwise box IoU lands at $0.82$ . The rule auto-accepts pairs above $0.75$ and routes the rest to an expert: $180$ boxes fall short and get adjudicated, while $820$ ship untouched. Review cost drops by over four fifths while every shipped box carries a measured agreement instead of a hope.

Code

Mean pairwise IoU over three boxes, the gate behind the routing rule.

1def box_iou(a: tuple[float, float, float, float], b: tuple[float, float, float, float]) -> float:2    """IoU of two xyxy boxes."""3    ix = max(0.0, min(a[2], b[2]) - max(a[0], b[0]))4    iy = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))5    inter = ix * iy  # overlap area6    ua = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter7    return inter / ua if ua > 0 else 0.08
9
10print(round(box_iou((10, 10, 50, 50), (12, 12, 52, 52)), 2))  # -> 0.82

Watch Out For

Accepting defaults for every class

Small distant pedestrians need different IoU gates than large near cars, but teams set one threshold everywhere. The symptom is clean car scores hiding sloppy person boxes that later wreck detection metrics.

Set gates per class and size band. Review the worst band first, not a random sample.

The Quick Version

Labeling tools provide drawing modes, review queues, and agreement scoring in one loop.
Double labeling plus adjudication beats single-pass clicking on hard slices.
Worked numbers: $1{,}000$ boxes, $0.82$ mean IoU, $180$ routed to experts.
Auto-accept above $0.75$ IoU and spend experts only on disputes.
Set agreement gates per class, and export annotator IDs with every batch.