Multi-Object Tracking Explained
Multi-object tracking detects everything in each frame and then solves the seating puzzle of which box continues which identity.
Why Does This Exist?
Counting pedestrians, analyzing plays and monitoring traffic all need every target held separately through crossings and occlusions. Detectors alone cannot do this: they rename the world each frame. Multi-object tracking (MOT) exists to maintain the cast list, starting tracks for newcomers, bridging short disappearances, and killing tracks that truly left. The parent map is object tracking; the one-target sibling is single-object tracking.
Think of It Like This
A coat check with identical coats
A coat check tags each coat on arrival, but half the coats are black and identical. When owners return in a rush, tags fall off some items. The attendant matches by tag (motion prediction), fabric wear (appearance embedding) and memory of who arrived when (track age). Handing one black coat to the wrong owner is an identity switch: both people leave happy and the books are wrong. MOT systems keep the same three books and fear the same mistake.
How It Actually Works
Each frame, detections arrive and existing tracks predict forward with a Kalman filter. A cost matrix blends motion distance (Mahalanobis or IoU) with appearance distance, and the Hungarian algorithm picks the globally cheapest detection-to-track assignment. Unmatched detections seed new tracks after a confirmation period; unmatched tracks coast on prediction for a grace window before deletion.
A worked assignment
Two tracks and two detections produce matching costs of 0.3 for track 0 to detection 0, 0.2 for track 1 to detection 1, with cross costs 0.9 and 0.8. The straight assignment totals 0.3 + 0.2 = 0.5 while the crossed one totals 0.9 + 0.8 = 1.7, so the Hungarian algorithm keeps identities uncrossed. When both cross costs drop near the straight ones, an identity switch becomes likely and appearance features must break the tie.
Watch Out For
Counting detector wins as tracking wins
A stronger detector removes misses and false alarms, which inflates MOTA while association stays identical. The symptom is celebration over a detector upgrade disguised as a tracker upgrade. Fix it by reporting identity-switch counts and IDF1 beside MOTA, since those expose association quality directly.
The Quick Version
- MOT detects all targets per frame, then assigns detections to tracks optimally.
- Kalman prediction plus Hungarian assignment is the classic association loop.
- Appearance embeddings rescue matches where motion alone is ambiguous.
- Track birth confirmation and death grace periods tame flicker.
- Report MOTA with IDF1 and switch counts or detector gains will mislead you.