Grounding DINO Open Boxes

Guide DINO queries with text so each query hunts a phrase-named object. Language picks which image spots become boxes.

Grounding DINO fuses text queries with image features for open boxes

Why Does This Exist?

Plain DINO queries learn dataset-common spots and miss rare phrases. Grounding DINO picks queries by text-image similarity, then fuses both streams through deformable attention. New phrases retrieve their own boxes in open-vocabulary detection tests.

This page covers text-guided queries only. Word-region pretraining basics are the GLIP page.

The one idea you need first.

Decoder queries $q$ start from image spots most similar to text $t$ . Fusion layers mix $q$ with image $v$ plus text $t$ . Box plus phrase scores train jointly.

Think of It Like This

Sniffer dogs briefed with a scent cloth

Dogs run free without scent, but a cloth with the target smell focuses the search. Each team quarters only ground matching the scent.

Queries are dogs, text cloth is the phrase, and similarity picks the field. Where it stops: dogs tire, while queries score all spots at once.

How It Actually Works

Encode image $v$ and text $t$ . Select top spots by $v \cdot t$ as queries. Decode with self plus image plus text cross-attention. Pay box $L_1$ plus GIoU plus phrase classification. Pretrain on Objects365 plus GoldG plus Cap4M mixes.

In plain words: text selects where to look, fusion reads what sits there, and boxes name the pick.

Worked example

Nine hundred queries reduce to $100$ by text similarity. True box IoU $0.78$ for top query versus $0.31$ for a text-blind pick. Loss $L_1$ $0.12$ plus GIoU $0.25$ plus class $0.4$ totals $0.77$ . Text selection cuts the $0.31$ misses that blind queries keep.

Watch Out For

Long phrases confusing query choice

Twenty-word phrases dilute similarity and queries scatter. Grounding on visual grounding longs sags. Shorten to nouns plus relations or weight nouns higher in the text encoder.

The Quick Version

Grounding DINO guides queries with text similarity.
Image plus text fusion shapes each box.
Box L1 plus GIoU plus phrase loss train jointly.
Broad grounding mixes supply open phrases.
Text selection lifts rare-object recall.