Embodied Vision Language Acts

Follow spoken orders through rooms using live camera views. Words plus sight become moves that reach goals.

Embodied vision-language turns words plus views into actions

Why Does This Exist?

Chat about rooms does not move robots through them. Embodied nets close the loop: read order $t$ , see view $v_t$ , pick move $a_t$ , and repeat to goals. Room-to-Room sets grade success weighted by path length, linking object tracking to action.

This page covers instruction-to-action only. Pure video QA is the prior page.

The one idea you need first.

Policy $\pi(a_t \mid t, v_t, h)$ maps order plus view plus history $h$ to moves. Success rate plus SPL weight detours. Sim trains, real halls test the gap.

Think of It Like This

Guiding a friend by phone through a new house

A friend hears go past the kitchen to the red chair and narrates views back. You correct left versus right at each door until the chair appears. Words plus sight become steps.

Orders are guidance, views are narration, and stops end calls. Where it stops: phones carry human common sense, while policies hold only trained patterns.

How It Actually Works

Encode order $t$ once and views $v_t$ per step with ViT plus text towers. Track history $h$ with transformers. Decode moves from turn, move, and stop sets. Train by imitation on human paths plus fine RL on success. Test success plus SPL on unseen houses.

In plain words: $t$ sets goals, $v_t$ shows doors, and $h$ remembers turns.

Worked example

Order length $12$ words, path $9$ steps. Success $1$ with path $11$ versus best $9$ gives SPL $1 \times 9/11 = 0.818$ . Fleet mean success $52$ percent, SPL $0.47$ . The $0.05$ gap to shortest shows mostly direct routes with some wander.

Watch Out For

Sim-only routes failing in real halls

Clean sim textures plus exact moves do not match slips and glare. Real success falls $20$ points. Domain randomize plus test in real halls before claiming navigation skill.

The Quick Version

Embodied nets map orders plus views to moves.
History tracks turns across steps.
Room-to-Room success plus SPL grade routes.
Imitation starts, RL refines.
Sim-to-real gaps need real-hall tests.