Video Object Segmentation Tracks Masks

Video segmentation carries a first-frame mask through time, repainting the same object in every later frame.

A first-frame mask seeds memory, and frame 20 still matches at IoU 0.81 by reading stored keys instead of segmenting fresh

Why Does This Exist?

Per-frame segmentation flickers. The same dog gets a new shape every frame, edges boil, and small parts vanish between cuts.

Video segmentation keeps identity through time. It extends instance segmentation, reuses region ideas from semantic segmentation, and needs the motion memory of video understanding.

Think of It Like This

Tracing a runner with carbon paper

Trace a runner once, then slide carbon paper under later frames so the outline follows the body instead of redrawing it.

It fails when the runner passes behind a pole. The carbon keeps the old shape floating where the body no longer shows.

How It Actually Works

Store keys $K$ and values $V$ from past masks, where $K$ holds appearance and $V$ holds mask shape. For current frame query $q$ , read weights $w = \text{softmax}(qK^\top / \sqrt{d})$ , where $d$ is key size. The mask is $m = wV$ , a blend of stored shapes weighted by look-alike scores.

Worked example

Frame 20 scores memory slots $[2.0, 1.0, 0.0]$ . Exponentials are $[7.39, 2.72, 1.00]$ summing to $11.11$ , giving weights $[0.665, 0.245, 0.090]$ . Blending stored masks with those weights yields IoU 0.81 against truth. Fresh per-frame segmentation on the same blurred frame scores 0.62. Memory adds 0.19.

Watch Out For

Letting memory fill with bad masks

One wrong mask poisons later reads because $V$ stores shape. The symptom is a blob that follows nothing after occlusion. Gate memory writes by confidence and keep the clean first frame pinned.

The Quick Version

Video segmentation propagates masks with $m = wV$ memory reads.
Fixture weights $[0.665, 0.245, 0.090]$ blend past shapes to IoU 0.81.
Memory beats fresh segmentation 0.81 to 0.62 on blurred frames.
Identity persists through motion that breaks single images.
Pin clean memory and gate writes to survive occlusion.