Egocentric Vision

Head-mounted cameras see the world the way wearers do: hands first, objects close, motion constant. Egocentric vision reads that shaky first-person stream for assistance and memory.

Memory queries shortlist 50 frames from 28800, so the heavy model reads minutes instead of the whole day

Why Does This Exist?

Third-person models watch the actor; wearable assistants are the actor. A head-mounted camera sees hands occluding objects, violent head turns, and hours of mundane footage between key moments — every assumption of curated clips (stable framing, centered subjects, trimmed actions) fails at once. Porting exocentric models scores poorly and nobody knows which violation hurts most.

Egocentric vision rebuilds for the forehead: hand-object interaction as the primary signal, gaze as attention labels, anticipation of the wearer's next action, and episodic memory over day-long streams ("where did I leave the keys?"). AR glasses and human-demonstration robotics wait on it. The backbone view is video foundation models; the memory view is long-video modeling.

The one idea you need first. Episodic memory queries ask the past: given hours of first-person footage and a language query, return the answer with visual evidence. Systems split it into cheap retrieval (shortlist candidate minutes with light embeddings) and heavy reading (run the strong model on seconds, not hours). The split is the only way day-long horizons fit real budgets.

Think of It Like This

A cooking show filmed from the chef's forehead

A tripod cooking show frames hands, pan, and ingredients in stable thirds — every tutorial assumption. Strap the camera to the chef's forehead: hands balloon to half the frame, pans whip across the kitchen between glances, ingredients hide behind fingers at the key second. Same recipe, hostile footage.

Egocentric models are edited for the forehead cut: they expect giant hands (primary actors, not occluders), whip-pans (motion to survive, not blur to reject), and fingers over the thing that matters (infer around them). Tripod-trained models reviewing this footage complain about quality; forehead-native ones get cooking.

How It Actually Works

Pipelines stabilize lightly (electronic, not cinematic — over-smoothing erases hand micro-motion), detect hands and active objects per frame, and track gaze for attention supervision. Action models consume short windows for "what now" plus long context for "what next" (anticipation heads predict seconds ahead). Memory systems index the day with cheap clip embeddings, retrieve top minutes per query, and read densely only there. Audio rides along: kitchens and workshops narrate themselves.

Benchmarks (Ego4D-style: thousands of hours, hundreds of wearers) grade hands, gaze, anticipation, and episodic queries separately — no single number covers the forehead.

Worked example

"Where did I leave the keys?" over an $8$ -hour day at $1$ fps: $8 \times 3{,}600 = 28{,}800$ frames. Cheap retrieval shortlists $50$ candidate frames; the heavy reader inspects those. Reduction: $28{,}800 / 50 = 576$ .

The heavy model prices $50$ frames instead of $28{,}800$ — a $576\times$ discount that turns an impossible query into a seconds-long one. Retrieval misses cap the ceiling (a key frame unshortlisted is unanswerable), so recall at the shortlist stage is the metric that decides the product.

Code

The point of this snippet is the funnel: day-scale footage narrows to a readable handful before heavy models engage.

1def memory_funnel(hours: float, fps: int = 1, shortlist: int = 50) -> tuple:2    frames = int(hours * 3600 * fps)3    return frames, round(frames / shortlist)4
5
6print(memory_funnel(8.0))  # -> (28800, 576)

Watch Out For

Bystanders who never consented

Forehead cameras record everyone nearby: faces, screens, keypads, children — continuously, in private spaces. Datasets blur faces after the fact, but deployed assistants stream first and ask never. Regulation and norms both bite here, differently per room and country.

Design privacy in: on-device processing, bystander downsampling, recording indicators, and auto-expiry for non-queried footage. A memory product that remembers everyone else's day is a liability wearing a feature costume.

Motion that defeats frame thinking

Head turns whip frames into directional blur; hand occlusions remove the object at the key instant. Frame-based detectors and classifiers trained on stable footage collapse — not slightly, but to near chance on fast segments.

Model motion natively (event-like differencing, blur-robust features, temporal context spanning the whip) and evaluate on motion-heavy slices separately. Average accuracy over mostly-static benchmarks hides the failure where the product lives.

The Quick Version

Egocentric video is hands-first, shaky, and hours long — every curated-clip assumption fails.
Memory queries funnel $28{,}800$ frames to $50$ for heavy reading: a $576\times$ discount.
Hands are primary actors, gaze is free attention supervision, anticipation heads predict next actions.
Bystander privacy needs on-device processing and auto-expiry, not after-the-fact blurs.
Evaluate motion-heavy slices separately; averages over static footage hide the real failures.