Object Tracking with Kalman Filtering and SORT

Keep Identity Across Frames

SORT tracks objects by predicting their boxes, associating new detections with those predictions, and correcting matched states. It combines a Kalman motion model with one-to-one assignment. Detecting two boxes in each frame does not by itself say which box belongs to A or B.

Squares mark detections; paths mark proposed continuity across time. Horizontal position indexes frames and vertical position is schematic image location. The SORT method defines the box state and lifecycle. The following numbers are synthetic examples, not default tuning values or benchmark results.

Represent The Box State

SORT uses state (u,v,s,r,u̇,v̇,ṡ): center coordinates, box area, width/height aspect ratio, and rates for center and area. Aspect ratio is held constant by the motion model. A width-four, height-two box centered at (4,3) has s=8 and r=2. Recover width as √(sr)=4 and height as s/width=2. The center dot and arrow distinguish position from extent; the arrow indicates a motion component without assigning a numerical speed in this beat.

Predict With Constant Velocity

Switch to a one-dimensional center slice with x=10 pixels and velocity 2 pixels per frame. One frame later, x⁻=10+2×1=12. The dashed box is a prediction, not an observation. Box sizes are schematic here; the center displacement is drawn at one hundred units per image pixel. Abrupt acceleration or camera motion can invalidate this simple extrapolation.

Propagate The Uncertainty

For the position–velocity slice use F=[[1,1],[0,1]], prior covariance P=diag(4,1) and process covariance Q=diag(1,0). Covariance prediction gives P⁻=FPFᵀ+Q=[[6,1],[1,1]]. The position variance rises from 4 to 6 pixel squared. Intervals show marginal mean ± one standard deviation, so their radii are 2 and √6≈2.449 pixels. Both rows use forty drawing units per pixel. These are uncertainty intervals, not object widths or guaranteed bounds.

Compare An Associated Observation

Suppose association has supplied observation z=13 pixels for this track. The innovation is z−x⁻=13−12=1 pixel. The circle marks the prediction, the square the observation, and the arrow isolates their difference at one hundred forty drawing units per pixel. Association decides which observation to use; the Kalman update determines how strongly it changes the state. The global association step is examined below.

Weight The Correction

Let observation variance be R=3 pixel squared and use predicted position variance 6. The position gain is Kx=6/(6+3)=2/3. Corrected position is 12+(2/3)×1=12.6667; its variance becomes (1−2/3)×6=2. The narrower interval uses radius √2, with both rows at sixty drawing units per pixel.

In the full two-state fixture, H=[1,0] gives gain vector [2/3,1/9]ᵀ, so velocity also changes from 2 to 19/9≈2.1111. Updated covariance is [[2,1/3],[1/3,8/9]]. The diagram displays only the position marginal. See state estimation for the general recursion.

Build Overlap Costs

Return to two-dimensional boxes, all width four and height two. A one-pixel horizontal shift gives intersection 6 and union 10, hence IoU 0.6. A three-pixel shift gives intersection 2 and union 14, hence IoU 1/7≈0.143. Dashed boxes are predictions; solid boxes are observations. A minimization cost such as 1−IoU prefers larger overlaps. This synthetic comparison changes geometry from the preceding scalar filter fixture. It uses no appearance evidence.

Assign Globally Rather Than Greedily

For a separate illustrative affinity matrix, rows are tracks T1/T2 and columns are detections D1/D2. Taking the largest entry first chooses T1–D1 at 0.60 and leaves T2–D2 at 0.10: total 0.70. The off-diagonal assignment scores 0.50+0.55=1.05 while using each row and column once. Global linear assignment finds the best total under this objective. The numbers illustrate assignment algebra, not the previous beat's box coordinates.

A minimum IoU also rejects weak associations. The reference implementation has an unambiguous-match shortcut and otherwise solves assignment before rejecting below-threshold matches; threshold handling should not be confused with an arbitrary claim that every matrix entry is always eligible.

Manage Birth And Expiry

An unmatched detection at frame 2 starts the lower track. The upper track is observed at frames 1 and 2, missed at 3, then expires at 4 under this toy policy: allow one missed frame and delete after two consecutive misses. Squares denote hits, a dotted circle denotes a miss, and a cross denotes deletion. Confirming a new track can require repeated hits; initialization and reporting policy are distinct. This drawing explains lifecycle logic rather than specifying every implementation default.

Where It Breaks

The shaded-by-hatching interval hides A long enough for its track to expire. A later detection starts ID C. The broken trace shows lost identity even if the returning box is geometrically plausible. Basic SORT has no appearance-based re-identification mechanism to prove that C is the old A.

Simple motion and overlap make association economical, but long occlusions, crowded crossings and missed detections can break continuity. Use appearance cues or richer temporal reasoning when identity recovery matters. Detection AP alone cannot measure this failure; it scores detections without testing long-term identity.

The Quick Version

• Tracking links detections into identities across frames.
• SORT models box center, area, aspect ratio and motion rates.
• Constant velocity predicts the next box center.
• Covariance prediction propagates motion uncertainty.
• Innovation compares the associated observation with the prediction.
• Kalman gain weights corrections according to uncertainty.
• Box overlap supplies geometric association costs.
• Global assignment respects one-to-one detection and track constraints.
• Track policies govern births, confirmation and expiry.
• Long occlusions can replace an old identity with a new one.