OWL-ViT Open Token Boxes

Start from CLIP patches, pool tokens into boxes, and score each box against text names. Open phrases work with light tuning.

OWL-ViT pools patch tokens into open-vocabulary box scores

Why Does This Exist?

CLIP sees whole scenes but no boxes. OWL-ViT adds light box heads on frozen-ish patch tokens with bipartite matching, keeping text scoring. Open-vocabulary detection then runs from a CLIP start with little box data.

This page covers token pooling only. Heavy fusion detectors are the Grounding DINO page.

The one idea you need first.

Patch tokens $p_i$ from ViT pool into box $b$ by attention. Score $s = \mathrm{sim}(b, t_c)$ for text $c$ . Bipartite matching pairs predictions to truth one-to-one per image.

Think of It Like This

Cookie cutters on a printed sheet

A printed sheet shows many shapes. Small cutters punch out candidate cookies, and name cards score each cookie by look. New cards work without rebaking the sheet.

Patches are the sheet, pooling is the cutter, and text cards are names. Where it stops: dough cuts cleanly, while token pools blur edges.

How It Actually Works

Init from CLIP $f$ plus text $h$ . Add box MLP plus pooling on $p_i$ . Tune on Objects365 plus Visual Genome with box $L_1$ plus GIoU plus text contrast per matched pair. At test time feed new names as text and keep top boxes.

In plain words: CLIP gives eyes, pooling gives hands, and text gives the name list.

Worked example

One hundred pools, three truths. Matched IoUs $0.81, 0.66, 0.74$ , mean $0.737$ . Text scores $0.41$ true versus $0.18$ best other. Loss sums box $0.3$ plus class $0.5$ . One-shot queries with one example image per new class lift rare AP $6$ points in standard runs.

Watch Out For

Small boxes lost in coarse tokens

Sixteen-pixel patches smear tiny items and recall sags on object detection small splits. Upscale inputs or use windowed features before blaming text heads.

The Quick Version

OWL-ViT pools CLIP patch tokens into boxes.
Text names score each box by similarity.
Bipartite matching trains one-to-one pairs.
Light tuning opens a CLIP start to detection.
One example per class enables one-shot queries.