GLIP Grounded Detection

Treat every box label as words and align regions to those words during training. Detection and grounding become one job.

GLIP trains detection as phrase grounding over words and regions

Why Does This Exist?

Detection data is dear while grounding phrases are many. GLIP rewrites both as word-region matching over GoldG $0.8$ M pairs plus Objects365 boxes. One head learns nouns plus phrases, lifting open-vocabulary detection and visual grounding together.

This page covers word-region unification only. Later query designs are the Grounding DINO page.

The one idea you need first.

Region $r_i$ meets word $w_j$ in score $s_{ij}$ . Grounding loss matches many words per image; detection loss matches few classes. Deep fusion lets words shape regions early.

Think of It Like This

Labeling museum photos with full captions, not tags

A curator tags photos with single words like dog, then upgrades to full captions naming the brown dog left of the bench. Every noun becomes training, not just the class tag.

Tags are detection labels, captions are grounding text, and alignment is the catalog. Where it stops: museum captions stay clean, while web phrases add noise.

How It Actually Works

Encode regions $R$ and words $W$ with cross-attention fusion. Score $S = R W^T$ . Pay classification over words plus box $L_1$ and GIoU on matched pairs. Mix detection batches with grounding batches; prompts list classes as a sentence at test time.

In plain words: sentences replace class lists, and fusion puts language into boxes.

Worked example

Caption person in red helmet, regions $3$ , words $5$ . True links $(r_1, red)$ , $(r_1, helmet)$ . Scores $0.5$ true versus $0.1$ rest. Loss $-\log(0.5/0.9) = 0.588$ per link early, falling to $0.2$ as fusion sharpens. LVIS rare AP rises from $9$ to $20$ in standard runs.

Watch Out For

Prompt format drift at test time

Training sentences differ from test lists and scores sag $5$ points. Comma lists versus periods shift token shapes. Keep prompt igetool formatting fixed between train and test in object detection harnesses.

The Quick Version

GLIP unifies detection and grounding as word-region matching.
GoldG 0.8M pairs plus detection data train one head.
Deep fusion lets words shape region features.
Test classes arrive as sentence prompts.
Rare-class detection rises with phrase data.