Scene Understanding for AR

Scene understanding for AR maps floors, walls, and occluders so virtual chairs sit on floors and hide correctly behind real sofas.

A floor plane fit with 1.2 cm residual anchors a chair that stays put while a 6 cm fit lets it drift and sink

Why Does This Exist?

Unanchored AR floats: chairs hover, labels swim, hands pass through tables. Rooms vary in light and clutter while phones vary in sensors. One frame depth jitters too much to trust.

Scene understanding for AR exists to stabilize the world model. Input is RGB plus depth or motion frames. Output is planes, mesh, lighting, and anchors. The headset frame is AR and VR; depth is depth estimation. Pixel labels are semantic segmentation.

Use it when virtual content must sit, hide, and light like real objects. Use face filters alone when playfulness outranks placement truth.

Think of It Like This

Taping a poster with a level and stud finder

You find the stud, level the poster, check the door swing will not tear it, and note window light so colors match. Skipping one step leaves a crooked poster torn by Friday.

Planes are the level line. Mesh is the stud map. Lighting is the window note. The analogy stops at motion: posters stay put while users pace, so anchors must survive relocalization that tape never faces.

How It Actually Works

Fit dominant planes with robust regression over depth points, keep those with residual under $2$ cm and wide support. Build a coarse mesh for occlusion, estimate one directional light plus ambient for shading match, then attach anchors to planes, not raw camera poses. Contract is frames in, planes plus mesh plus light out. A floor fit at $1.2$ cm residual holds a chair within $1$ cm across a walk. A $6$ cm fit lets it sink $4$ cm and slide.

Worked example

A living room scan gives $2000$ floor points. Robust fit finds a plane with $1.2$ cm root mean residual and $85$ percent inliers. A virtual chair leg at that plane stays planted while the user circles $3$ m. A competing fit on rug plus floor mixes heights to $6$ cm residual; the same chair sinks $4$ cm into the rug and drifts $5$ cm sideways. Occlusion test against the sofa mesh hides $30$ percent of the chair correctly when viewed from the door.

Watch Out For

Textureless floors that invent planes

Blank rugs return sparse depth with $5$ cm noise. The fitter hallucinates tilt. The symptom is chairs leaning like tired tourists. Fix it with wider baselines, inertial priors, and by refusing anchors when inliers fall short.

Lighting ignored until compositing looks fake

A noon estimate used at dusk paints noon shading on dusk rooms. The symptom is glowing objects users call stickers. Fix it by re estimating light per session and dimming with exposure.

The Quick Version

AR scene models hold planes, mesh, light, and anchors.
Robust fits under $2$ cm residual make sittable floors.
The fixture holds at $1.2$ cm and sinks at $6$ cm residual.
Occlusion mesh hides virtual parts behind real ones.
Blank floors and stale light are the first failures to gate.