Mean Shift Tracking

Mean shift tracking re-centers the box wherever the target's colors pile up most densely, climbing the similarity hill one frame at a time with no motion model at all.

The window climbs the backprojected color-similarity hill until it sits centered on the densest patch of target colors.

Why Does This Exist?

Some targets have no stable shape but unmistakable colors: a red jersey in a green field, a yellow taxi in gray traffic. Motion models like the Kalman filter predict where the box goes but say nothing about what it looks like, so they drift when detections fail. What you want is a tracker that locks onto the appearance itself and follows it wherever it slides.

Mean shift tracking (Comaniciu, Ramesh, and Meer, 2000) does exactly that. Learn the target's color histogram once, backproject it each frame into a probability map of "target-color-ness," and climb to the map's peak starting from last frame's box. No training, no motion model, a few dozen iterations of arithmetic. The clustering cousin that climbs density peaks in feature space is mean shift; this page is its tracking application in image space.

Think of It Like This

Finding the warmest spot in a pool

You step into a pool blindfolded looking for the warm inflow. You feel the temperature around your feet, shuffle toward the warmer side, feel again, shuffle again. Each step uses only local warmth, yet you end at the vent. You never mapped the pool.

The pool is the backprojection map, warmth is target-color probability, and shuffling is the mean shift step: move the window to the weighted center of what it currently covers. The vent is the similarity peak. And like the pool, a second warm vent (a same-colored rival) captures you if you start closer to it.

How It Actually Works

1. Learn the color model

From the initial box, build a histogram $q$ of the target's colors (typically hue, weighted by saturation to ignore gray pixels, with an Epanechnikov kernel stressing the center). This histogram is the tracker's whole memory of what it is following.

2. Backproject and climb

Each frame, replace every pixel with the probability its color belongs to the target, forming a backprojection map. Starting at the old box center, repeatedly shift the window to the weighted centroid of the map inside it until it stops moving. That fixed point is the mode: the densest concentration of target colors nearby.

3. Score with Bhattacharyya

Similarity between target model $q$ and candidate $p$ is the Bhattacharyya coefficient $\rho = \sum_u \sqrt{p_u q_u}$ , with distance $d = \sqrt{1 - \rho}$ . Worked: $p = [0.5, 0.3, 0.2]$ , $q = [0.45, 0.35, 0.2]$ . Then $\rho = \sqrt{0.225} + \sqrt{0.105} + \sqrt{0.04} \approx 0.474 + 0.324 + 0.200 = 0.998$ , so $d \approx \sqrt{0.0016} \approx 0.04$ : near-identical. A candidate drifting onto background scores visibly worse, which is how trackers detect loss.

Watch Out For

Same-colored rivals steal the window

The tracker follows colors, not identity. A second red jersey crossing the target's path merges the peaks, and the window may latch onto the wrong player with full confidence. If IDs swap whenever lookalikes meet, color alone is insufficient and you need motion or appearance features.

Fixed window size in a zooming world

Classic mean shift never resizes: an approaching target outgrows the box and the centroid stalls at the box's edge. CamShift exists precisely to fix this by adapting the window from the color mass each frame.

The Quick Version

Mean shift tracking climbs the backprojected target-color map to its peak each frame.
The target model is one histogram learned from the initial box; no training phase.
Bhattacharyya distance scores candidates: the example pair sits at $d \approx 0.04$ , near-identical.
No motion model means it survives erratic movement but dies on same-colored distractors.
Fixed window size is the hard limit; CamShift adapts it.