Referring Expression Tests

Long phrases name one object among many lookalikes using clothes, spots, and order. Models must read every clue to place the box.

Referring expressions pick one person among lookalikes by wording

Why Does This Exist?

Short phrases underdetermine the target when five people wear jackets. RefCOCOg collects long phrases with ordinals like second from left plus attributes. Visual grounding systems prove full reading here, beyond class plus color.

This page covers long-phrase comprehension only. Basic phrase boxes are the grounding page.

The one idea you need first.

Phrase length averages $8$ to $12$ words in RefCOCOg versus $3$ in early sets. Ordinal $o$ plus attribute $a$ plus relation $r$ must all hold for the pick. Score stays IoU $0.5$ per phrase.

Think of It Like This

Picking one penguin by a long zoo note

Forty penguins huddle and a note says the third from the left with a chipped beak behind the tall rock. Each clue cuts the set until one remains. Skipping any line leaves three birds tied.

Words are cuts, boxes are birds, and IoU checks the final point. Where it stops: zoo notes stay true, while web phrases add ambiguity and error.

How It Actually Works

Encode long phrase with text transformer $h$ keeping word order. Encode regions $r_i$ with positions. Fuse deeply over layers so ordinals meet layout. Score regions or regress the box with $L_1$ plus GIoU. Train on RefCOCO, RefCOCO+, and RefCOCOg splits that stress position-free versus position-rich wording.

In plain words: depth matters because second from left needs counting, not bag words.

Worked example

Four people, phrase second from left in blue. Scores $[0.2, 0.4, 0.9, 0.3]$ . Softmax gives $0.149, 0.182, 0.442, 0.227$ after exponentials $1.22, 1.49, 3.60, 1.35$ over $8.16$ total. Pick three scores $0.442$ and IoU $0.72$ passes. Dropping ordinal flips the pick to person two at IoU $0.08$ .

Watch Out For

Bag-of-words models failing ordinals

Averaged word vectors ace colors but flop second from left. Ordinal splits lag $20$ points. The fix is order-aware text encoders plus position features in object detection regions.

The Quick Version

Referring tests use long phrases with attributes, ordinals, relations.
One box among lookalikes is the output.
Order-aware language plus positions matter.
RefCOCOg stresses the hardest wording.
IoU half scores each phrase pick.