Open Vocabulary Detection

Detect any object named in words, not just trained boxes. Text queries steer proposals to rare and new things.

Why Does This Exist?

Closed detectors miss rare things like axolotls because training lacks them. Open detectors pretrain on grounding phrases plus boxes, then answer new phrases in object detection form. LVIS rare AP measures the tail that matters.

This residual page frames the task; methods live in GLIP, Grounding DINO, OWL-ViT, and YOLO-World.

The one idea you need first.

Box feature $b_i$ scores phrase $t_c$ by cosine. LVIS splits classes into frequent, common, and rare; rare AP counts new names. Proposals must stay class-free to catch the tail.

Think of It Like This

A search party briefed in words, not photos

A party seeks a red scooter with a cracked mirror described only in words. Members stop any match, not just memorized faces. New descriptions work without retraining the team.

Phrases are briefings, proposals are passersby, and similarity is the stop rule. Where it stops: people ask follow-ups, while detectors score once.

How It Actually Works

Generate class-free proposals $b_i$ . Encode phrases $t_c$ . Score $s_{ic} = \mathrm{sim}(b_i, t_c)$ and keep top boxes per phrase after NMS. Pretrain on Objects365 plus GoldG grounding mixes; test on LVIS rare and ODinW suites.

In plain words: proposals find things, language names them, and rare AP grades new names.

Worked example

Two boxes score red scooter at $0.36$ and $0.21$ . Threshold $0.25$ keeps one. Truth IoU $0.74$ passes. Rare-class AP over $337$ LVIS rare names rises from $8.1$ closed to $21.4$ open in a typical run. The $13.3$ jump is the tail opening.

Watch Out For

Proposals tuned only to base classes

Base-only proposals drop novel shapes before language sees them. Rare recall then caps low despite good text heads. Train class-free proposals on broad boxes and check rare recall in object detection harnesses.

The Quick Version

Open detection boxes any phrase-named object.
Phrase similarity replaces fixed class rows.
LVIS rare AP grades unseen names.
Grounding pretraining lifts the tail.
Class-free proposals protect novel shapes.