SAM 2 Tracks Masks Through Video

SAM 2 adds memory to SAM, so point prompts on one frame propagate consistent masks across video in real time.

Six frame memory keeps IoU 0.86 through occlusions live

Why Does This Exist?

SAM cuts per frame but flickers: frame $t$ dog differs from $t+1$ with no memory. Video editing needs stable tracks.

SAM 2 streams memory. It extends segment anything, serves video object segmentation, and aids instance tracking.

Think of It Like This

A flipbook with onion skin memory

Animators see six faint prior pages under the current sheet. Lines stay consistent even when a hand covers part for a frame.

Memory bank is onion skin, prompts are first sketch. Where it stops: long covers erase memory, just as long occlusions drift tracks.

How It Actually Works

Encode frame to features. Attend to memory bank of $6$ past frame plus mask embeddings. Decode mask plus IoU plus occlusion score. Update bank FIFO. Hiera encoder keeps speed.

Worked example

Frame $10$ dog IoU $0.90$ . Frames $11$ - $13$ occluded $40$ percent, memory holds shape. Frame $14$ IoU $0.86$ , a drop of $0.04$ only. Without memory per-frame SAM falls to $0.61$ , a $0.29$ gap. Six frames at $64 \times 64 \times 64$ use about $1.5$ M values, small versus re-encoding.

Watch Out For

Trusting tracks through long full occlusions

Thirty-frame hides flush memory with background. Symptom: ID switches on return. Fix: re-prompt after long covers and cap memory reliance to short gaps.

The Quick Version

SAM 2 adds streaming memory to promptable masks for video.
Six past frames guide current decoding via attention.
Fixture IoU $0.90$ to $0.86$ holds through occlusion versus $0.61$ alone.
Hiera encoder keeps live speed.
Re-prompt after long full covers.