Augmented and Virtual Reality
AR and VR vision tracks heads, hands, and rooms so virtual objects sit still in real space and respond to touch without making you sick.
Why Does This Exist?
A virtual cup must sit on a real table while you walk around it. A ms lag lets it swim and your stomach notices before your eyes name the lag. Hands must grab it with occlusion that respects fingers.
AR and VR vision exists to hold that illusion. Input is headset cameras plus motion sensors. Output is head pose, hand pose, and room mesh at display rate. Motion between frames is optical flow; the clip context is video understanding. Lens and pose geometry is camera calibration.
Use it when virtual content must anchor to rooms and bodies. Use plain 3D viewers when the camera never moves with the user.
Think of It Like This
A stagehand in the dark moving props with you
A stagehand watches your steps in the dark and slides a chair so it stays under the spotlight as you circle. The hand must move before you notice lag, and must hide the chair leg behind you when you pass.
Tracking is watching steps. Mapping is knowing chair spots. The analogy stops at sickness: a late stagehand merely looks clumsy while late photons make viewers ill, so budgets bind harder.
How It Actually Works
Fuse visual tracking with inertial sensing: cameras fix drift while gyros fill fast spins. Build a room mesh from depth or multiview, then render virtual objects with depth test so real hands occlude correctly. Contract is sensor stream in, poses plus mesh plus composited frame out. Budget motion to photon near ms: ms tracking, ms render, ms display. At ms a degree per second turn slips degrees, visible as swim.
Worked example
A user turns degrees per second. At ms lag the virtual cube slips degrees, near the comfort limit. At ms it slips degrees and swims. Occlusion test places a cm cube behind a hand at cm: depth says hand wins, so those pixels keep hand color. Missing depth paints cube over fingers and breaks touch instantly.
Watch Out For
Textureless walls that starve tracking
White meeting rooms give cameras nothing to hold. Visual pose drifts and the inertial guess walks away. The symptom is slow scene slide during still stares. Fix it with depth assists, wider fields of view, and by warning when feature counts fall.
Wrong depth that breaks touch
A cm depth error at cm flips finger versus cube order. The symptom is cubes drawn over fingertips users swear they touched. Fix it with per device depth calibration and by biasing toward hand when uncertain.
The Quick Version
- AR and VR anchor virtual objects to tracked rooms and bodies.
- Contract is sensor stream in, poses plus mesh plus frame out.
- The fixture holds degree slip at ms against at ms.
- Depth tested occlusion sells touch; missing depth breaks it.
- Feature poor rooms and depth bias are the first failures to check.