Scene Graph Generation from Images

Scene graphs turn detected objects into subject-predicate-object triples so questions about relations become lookups.

Why Does This Exist?

Object detection lists nouns without verbs. Who rides what and what sits on what needs edges. Scene graph generation detects $N$ objects then scores predicates for pairs, outputting triples like person riding bike. Retrieval and VQA then query the graph instead of pixels.

Think of It Like This

Labeling a party photo with string and name tags

You stick name tags on every guest, then tie labeled strings between pairs: holding, talking to, standing near. Queries like who holds cake follow strings instead of scanning faces again.

Objects are guests and predicates are strings. The analogy stops at exhaustiveness: photos need only salient strings, not all $N^2$ pairs.

How It Actually Works

Detector gives boxes $b_i$ with features $f_i$ . Pair $(i, j)$ forms edge feature $[f_i, f_j, \text{spatial}]$ scoring $50$ predicates. Training uses ranking over the long tail since near dominates riding $100$ to $1$ . Recall@50 counts how many true triples appear in the top $50$ edges.

Worked example

Three objects give $3 \times 2 = 6$ directed pairs. Pair person-bike scores riding $0.82$ , near $0.60$ , holding $0.35$ . Pair bike-ground scores on $0.91$ . Top- $50$ keeps riding and on while near drops to rank $8$ . Recall@50 of $0.4$ means two of five true triples surfaced, typical for this long-tailed task.

Watch Out For

Predicate bias toward vague near

Without reweighting, near wins every pair at $0.7$ while riding stalls at $0.3$ . The symptom is graphs full of near and on. Reweight by inverse frequency or evaluate mean recall per predicate.

The Quick Version

Scene graphs add predicate edges to detected object nodes.
Fifty predicates cover most relations with a heavy long tail.
Ranking losses beat plain classification for rare relations.
Do not score all $N^2$ pairs blindly; prune by overlap and salience.
Recall at $50$ is the standard metric, not accuracy.