Visual Grounding Phrases

Given a phrase like the red mug left of the laptop, draw the one box it means. Relations and attributes pick the target among lookalikes.

Visual grounding links a phrase to exactly one box in the photo

Why Does This Exist?

Detection lists all boxes of fixed classes, but users point with words. Grounding takes free text plus a photo and returns the one referred box. Object detection gives candidates; language picks among them using color, size, and left-of relations.

This page covers phrase-to-box only. Multi-object open vocab is the open-vocabulary pages, and fine phrase tests are referring expressions.

The one idea you need first.

Input is phrase $t$ plus photo $x$ ; output is box $b = [x,y,w,h]$ . Score is IoU with truth, counted right at $0.5$ . RefCOCO sets hold such pairs with lookalike distractors.

Think of It Like This

Finding your bag by a spoken clue in a hall

A hall holds forty black bags. A friend says yours is the small one left of the pillar with a red tag. Color, size, and relation together single it out where class alone fails.

Bags are boxes, the clue is the phrase, and IoU is the handshake on the right bag. Where it stops: halls stay still, while photos add viewpoint and occlusion.

How It Actually Works

Encode phrase to $t$ and photo regions to $r_i$ . Fuse with cross-attention and score each region, or regress $b$ directly as numbers. Train with $L_1$ plus GIoU on boxes plus matching loss on phrase-region pairs. Relations like left of enter through position features.

In plain words: $t$ queries regions, boxes carry positions, and IoU judges the pick.

Worked example

Truth $[100, 150, 80, 80]$ , prediction $[110, 155, 80, 80]$ . Overlap $70 \times 75 = 5250$ . Union $6400 + 6400 - 5250 = 7550$ . IoU $5250/7550 = 0.695$ , above $0.5$ so counted right. Dropping the relation left of would pick a second mug at IoU $0.05$ and fail.

Watch Out For

Ignoring relations and picking any match

Color-only models pick the wrong red mug when two sit apart. Accuracy on relation splits sags $15$ points. The fix is position-aware fusion plus relation-heavy tests from referring expressions.

The Quick Version

Visual grounding maps one phrase to one box.
Attributes plus spatial relations disambiguate lookalikes.
Boxes train with L1 plus GIoU; matches train contrastively.
IoU half on RefCOCO sets is the standard score.
Relations need position features, not words alone.