Scene Understanding for Robotics

Scene understanding builds a working model of surfaces, objects, and free space a robot can query for support, clearance, and reachability.

Pixels become surfaces and objects linked by support and clearance relations

Why Does This Exist?

A list of detections cannot say whether the table holds the book, whether the gap fits the arm, or which handle to pull. Robots need relations: support, containment, clearance, reachability. Scene understanding exists to lift pixels into that queryable model, where each fact enables or blocks an action. Without it, planning guesses geometry from labels and collides.

For masks see semantic segmentation and for the AR variant see the AR scene page in this wave.

Think of It Like This

Reading a room before carrying a ladder

You note the table holds boxes, the doorway clears your shoulders, and the rug slips. You carry differently. The robot builds the same notes as a graph: nodes for surfaces and objects, edges for support and clearance. The analogy stops at glances: the robot measures each edge in centimeters.

How It Actually Works

Segmentation plus depth yields surfaces and objects with poses. A relation head predicts support (A holds B), containment, and clearance volumes. Affordance heads label graspable or pushable parts. The result is a scene graph the planner queries: is the top clear, does the shelf bear 2 kg, which path keeps 10 cm clearance. Updates run incrementally as the arm moves objects.

Worked example

A book 200 mm wide rests on a table with 50 mm free beside it. The gripper needs 85 mm opening. Support says table holds book, clearance says 50 mm is short of 85 mm, so side grasp fails while top grasp with 20 mm finger thickness succeeds. The planner picks top grasp without trial and error. One relation check saves a failed execution.

Watch Out For

Support assumed from contact alone

Touching pixels do not prove support: a poster touches a wall without bearing weight, and a leaning plank touches without stability. The symptom is confident lifts of unliftable structures. The fix requires geometry plus physics cues such as normals, overlap area, and mass priors before asserting support.

The Quick Version

Scene understanding outputs relations, not just labels.
Support, clearance, and affordance gate actions.
Graphs keep queries fast as scenes change.
Measure edges in physical units, not pixels.
Update incrementally during manipulation.