Semantic Scene Completion in 3D
Semantic completion fills occluded voxels behind visible surfaces and labels each as chair, wall, or free space.
Why Does This Exist?
Scene reconstruction maps what cameras saw. Robots grasp what they cannot see: chair backs, room corners, occluded table legs. Semantic scene completion (SSC) takes one depth view and outputs a labeled voxel grid including hidden cells. Occupancy prediction does this for driving; SSC covers indoor semantics.
Think of It Like This
Finishing a half-seen chess piece behind a book
One photo shows half a rook behind a book. You know rooks are symmetric and wooden, so you carve the hidden half to match. The visible half constrains; the prior completes.
Depth is the visible half and shape priors finish the piece. The analogy stops at labels: chess rules give the name while SSC also predicts chair versus table.
How It Actually Works
Input TSDF from one view feeds a 3D CNN outputting labels per voxel over grids at m. Loss weights hidden voxels -fold since visible ones are easy. IoU measures geometry; mIoU measures geometry plus labels.
Worked example
A chair front occupies visible voxels labeled chair at mean score. Behind it, hidden voxels predict chair at and predict wall at . Geometry IoU reaches while mIoU sits at because two hidden legs flip to table. That gap is the label cost of hallucination.
Watch Out For
Hallucinated backs trusted for grasping
A completed chair back offset cm breaks force closure. The symptom is grasps that slip on hidden sides. Plan grasps on visible voxels and treat completed ones as obstacles only.
The Quick Version
- SSC labels hidden voxels from one depth view plus priors.
- Geometry IoU and semantic mIoU report shape versus labels separately.
- Hidden voxels train at higher weight than visible ones.
- Do not grasp hallucinated backs; use them as obstacles.
- Indoor grids near at m cover rooms.