CamShift Tracking

CamShift is mean shift that resizes itself. The window grows, shrinks, and tilts each frame to fit the color mass it just found.

After mean shift converges, image moments resize and rotate the window, so the next frame starts from a box already fitted to the target.

Why Does This Exist?

Mean shift tracking climbs to the right center but keeps the starting window size forever. A person walking toward the camera doubles in size within seconds: the fixed box ends up framing only the torso, the centroid stalls, and the track degrades exactly when the target matters most. Restarting with a bigger box by hand is not tracking.

CamShift (Continuously Adaptive Mean Shift, Bradski 1998) closes the loop with image moments. After mean shift converges, it measures the color mass inside the window (zeroth moment), sets the window size from that area, and fits orientation from second moments. The box breathes with the target: approaching faces get bigger boxes, rotating hands get tilted ones. It became the default head and hand tracker in early OpenCV demos for good reason.

Think of It Like This

A spotlight that frames the actor

A theater spotlight operator keeps the lead actor centered (that is mean shift) but also widens the beam when the actor spreads their arms and narrows it for a soliloquy. The operator judges size from how much lit actor fills the current beam: spillover means widen, black edges mean tighten.

Moments are that judgment. The zeroth moment is how much actor fills the beam. The operator's widening rule is CamShift's resize step. A fixed-beam operator loses the actor's hands; CamShift keeps the whole performance framed.

How It Actually Works

1. Mean shift, then measure

Run standard mean shift to convergence inside the current window on the backprojection map. Then compute moments of the probability pixels: $M_{00} = \sum I$ (total mass), $M_{10} = \sum x I$ , $M_{01} = \sum y I$ , with centroid $(M_{10}/M_{00}, M_{01}/M_{00})$ .

2. Worked moments

Backprojection patch $\begin{bmatrix}0 & 1 & 0 \\ 1 & 5 & 2 \\ 0 & 1 & 0\end{bmatrix}$ over $x, y \in \{0, 1, 2\}$ . Total mass $M_{00} = 10$ . $M_{10} = 1 \cdot 7 + 2 \cdot 2 = 11$ , so $x_c = 1.1$ . $M_{01} = 1 \cdot 8 + 2 \cdot 1 = 10$ , so $y_c = 1.0$ . The centroid sits slightly right of center, pulled by the $2$ on the right edge, and the window width scales with $\sqrt{M_{00}}$ , growing as the target approaches.

3. Resize, rotate, repeat

Set the next window's size from $M_{00}$ and its angle from second-order central moments, then start the next frame's mean shift there. Orientation comes free, which fixed mean shift can never recover. Hue histograms in HSV (from color spaces) supply the usual backprojection, ignoring brightness.

Watch Out For

Moments trust all mass equally

A same-colored object entering the window adds mass that swells and drags the box toward the intruder. CamShift adapts to whatever is inside, friend or foe. Constrain the search area with a motion gate or pair it with a detector that re-seeds the window on drift.

Lighting shifts rewrite hue identity

Hue is stabler than RGB but not immune: auto-white-balance swings and sodium lamps remap the target's histogram until the backprojection goes dark. If tracks die at dusk or under mixed lighting, the color model aged out, and the histogram needs refreshing or a less color-pure feature.

The Quick Version

CamShift adds a moment-based resize and rotate step after each mean shift convergence.
The example patch holds mass $10$ with centroid $(1.1, 1.0)$ , pulling the window rightward.
Window size tracks $\sqrt{M_{00}}$ , so approaching targets grow their own boxes.
Second moments give orientation that fixed mean shift cannot recover.
Same-color intruders corrupt the mass, so gate the search area in cluttered scenes.