Embodied and Interactive Vision

Cameras that move can look closer, circle around, and poke things. Embodied vision studies perception tied to action: seeing in order to do, and doing in order to see better.

The sense-predict-act loop closes on the world, and view A wins with 0.6 bits of gain against 0.3

Why Does This Exist?

Internet vision is a spectator sport: fixed photos, fixed frames, no second chances. A model that misreads an occluded mug in a photo simply fails. A robot facing the same mug leans left, and the handle appears. Passivity turns answerable questions into errors; action recovers them for the price of movement.

Embodied vision studies that trade. Perception chooses actions (active vision), actions need predicted futures (video prediction), and both need a world that pushes back (physical reasoning, plus pixel-based control in RL from pixels and imitation from video). This page maps the loop. Agent-scale embodiment (general embodied AI, action models, world models) belongs to its generative-AI owners, and navigation routes to the visual navigation page when its wave lands (see donor table).

The one idea you need first. The sense-act loop runs observe, update beliefs, choose the action that best reduces uncertainty or advances the goal, execute, and repeat. Beliefs are distributions, not labels: a $50/50$ object identity is a state the loop can act on (move to disambiguate), where a classifier would just guess.

Think of It Like This

A tourist with a camera, not a bolted security camera

A security camera watches the plaza from its bracket: whatever crosses its fixed frame is all it ever knows, and a pillar hides the same corner forever. A tourist with a camera steps sideways around the pillar, zooms into the plaque, and circles the statue to read every face. Same plaza, different perception — the difference is legs.

Embodied agents are tourists. The camera moves to where evidence lives instead of waiting for evidence to visit. The price is decisions (where to step) and risk (steps cost time and battery), which is why the loop optimizes information per unit of movement rather than information alone.

How It Actually Works

The loop has four beats. Sense: gather the current view plus proprioception. Update: fold observations into a belief state (object poses, identity odds, scene layout). Predict: roll candidate actions forward — what would the camera see from there, what would the arm feel. Act: execute the best candidate, then re-sense. Active vision owns the choice rule (expected gain minus motion cost); prediction owns the forward roll; control (RL, imitation) owns the execution policy.

The currency is uncertainty reduction per cost. A view is worth taking when its expected information gain exceeds its movement price by the best margin available.

Worked example

Identity uncertainty starts at $0.8$ bits (a blurry object, several candidates). Two candidate views:

View A (lean left, see the handle): uncertainty falls to $0.2$ bits. Gain $= 0.8 - 0.2 = 0.6$ bits at cost $1$ move: $0.6$ bits per move.
View B (zoom from here): uncertainty falls to $0.5$ bits. Gain $= 0.8 - 0.5 = 0.3$ bits at cost $1$ move: $0.3$ bits per move.

Take view A: double the gain per move. If A's cost rose to $3$ moves (a long walk), its rate falls to $0.2$ per move and B wins — the loop prices information like freight, not like treasure.

Code

The point of this snippet is the choice rule: gains over costs, argmax over views.

1def gain_per_cost(before: float, after: float, cost: float) -> float:2    return (before - after) / cost  # bits of uncertainty per move3
4
5views = {"A lean left": (0.8, 0.2, 1.0), "B zoom": (0.8, 0.5, 1.0)}6rates = {v: gain_per_cost(b, a, c) for v, (b, a, c) in views.items()}7print({v: round(r, 2) for v, r in rates.items()})  # -> {'A lean left': 0.6, 'B zoom': 0.3}8print(max(rates, key=rates.get))  # -> A lean left

Watch Out For

Simulation legs that never walked outside

Policies trained in clean simulators exploit simulator trivia: perfect odometry, texture-free walls, deterministic contact. Deployed, the same policy wobbles on carpet and misjudges every shiny handle. The sense-act loop amplifies the gap because errors compound across steps.

Randomize aggressively (friction, lighting, sensor noise) and validate on real floors early. A loop closed only in simulation is a rehearsal mistaken for opening night.

Acting without predicting

Choosing views by immediate gain alone walks into dead ends: the informative close-up behind a door that then blocks retreat. Greedy information maximization with no forward model is exploration without a map.

Pair every choice rule with a predictive rollout, however short. The prediction pages exist because action without foresight spends movement learning what a one-step lookahead would have told it.

The Quick Version

Embodied vision closes the loop: sense, update beliefs, predict candidate actions, act, repeat.
The worked fixture prices view A at $0.6$ bits per move against $0.3$ for view B, so A wins.
Movement reprices information: triple A's cost and the ranking flips.
Simulation-only loops compound sim-to-real gaps across steps; randomize and test on real floors.
Greedy gain without predictive rollout walks into dead ends; pair choice with foresight.