Open Images Broad Labels

Web photos with many boxes, visual relationships and partial labels across hundreds of classes, built for broad open-world detection.

Open Images pairs web photos with many boxes and relationship triples, so broad vocabularies train under partial labels

Why Does This Exist?

Closed vocabularies with full exhaustiveness do not match web photos, where many objects appear and only some get labeled. Open Images collected web photos with boxes across hundreds of classes plus relationship and attribute notes, accepting partial labeling to reach breadth. It suits open-vocabulary testing better than narrow full sets.

This page covers purpose, partial-label contract and appropriate use. Scoring basics are in mAP.

Think of It Like This

A market list with some items unchecked

You list one hundred market stalls but only tick the ones you visited. Unticked does not mean absent, only unchecked.

Open Images works that way. A missing box often means unlabeled rather than truly absent, so training must not punish every blank.

How It Actually Works

Purpose is broad detection with relationships. Input is one web photo, outputs are box lists with class names plus optional relationship triples. Appropriate use is large-vocabulary detection and relationship checks. Respect positive-only and verified-negative signals where provided, and verify the current release before quoting class counts.

Toy partial-label walkthrough

Take 10 photos where 6 boxes are labeled positive and 4 plausible objects are unlabeled. Treating all 4 blanks as negatives gives $4 / 10 = 0.4$ false-negative pressure from annotation gaps alone. Ignoring unverified blanks and training on verified signals removes that invented penalty.

Code

Masked counting that skips unverified blanks.

1import numpy as np2
3
4def masked_hits(pred: np.ndarray, verified: np.ndarray) -> float:5    sel = pred[verified == 1]  # only score verified entries, skip unknowns6    return float(sel.mean()) if sel.size else 0.07
8
9print(f"{masked_hits(np.array([1, 0, 1, 0]), np.array([1, 1, 0, 0])):.2f}")10# -> 0.50

One of two verified entries hits, so 0.50 without punishing unknowns.

Watch Out For

Punishing unlabeled positives as background

Treating every unlabeled region as negative teaches the model to miss real objects. The symptom is falling recall on rare classes after more training. Fix it with verified-negative sampling and label-aware losses.

The Quick Version

Open Images maps web photos to many boxes plus relationships across a broad vocabulary.
Input is a photo, outputs are boxes with optional triples under partial labels.
Toy 4 blanks in 10 give 0.4 invented penalty if scored as negatives.
Use it for broad and open-world detection checks.
Score only verified entries and verify release counts officially.