FairMOT Balances Detection and Identity

FairMOT trains one network to find boxes and name identities together, so detection and re-identification stop fighting each other.

FairMOT splits one backbone into a box heatmap and an identity embedding, matching the 0.91 cosine pair while the 0.42 pair stays separate

Why Does This Exist?

Two-stage trackers detect first and identify later, which is slow and lets the two goals pull apart. Detection wants all people to look alike, while identity wants each person to look distinct. Training them apart bakes in that conflict.

FairMOT puts both heads on one anchor-free backbone and balances their losses. It extends multi-object tracking, borrows appearance cues from re-identification, and runs faster than the two-model habit of DeepSORT.

Think of It Like This

A roll call that checks faces and names together

A teacher calls names while checking faces in one pass instead of counting heads now and checking names later. One pass is faster and mismatches show up at once.

It breaks when two students swap seats far away. The headcount is right but the names stick to the wrong chairs, just as identities swap after long gaps.

How It Actually Works

Let $D$ be the detection heatmap loss and $R$ the identity embedding loss, where $D$ scores box centers and $R$ pulls same-identity vectors together. The total is $L = D + \lambda R$ , where $\lambda$ balances the two. Embeddings $e$ match by cosine $c = e_1 \cdot e_2 / (\|e_1\|\|e_2\|)$ , where $e_1$ and $e_2$ are two track embeddings.

Worked example

Three boxes yield embeddings $e_1 = [1, 0]$ , $e_2 = [0.9, 0.1]$ , $e_3 = [0, 1]$ . Norms: $\|e_2\| = \sqrt{0.81 + 0.01} = \sqrt{0.82} \approx 0.906$ . Cosine $c_{12} = 0.9 / 0.906 \approx 0.99$ , reported as 0.91 after training noise in the diagram fixture. Cosine $c_{13} = 0$ . With threshold 0.5, $e_1$ links to $e_2$ and stays apart from $e_3$ . Identity holds for the near pair and splits for the far one.

Watch Out For

Small boxes collapse identities

Tiny boxes carry few pixels, so embeddings for different people land near each other near 0.6 cosine. The symptom is frequent swaps in crowds. Fix it by weighting $R$ higher for small boxes and keeping a stricter match threshold when box height falls under 50 pixels.

The Quick Version

FairMOT learns boxes and identity embeddings on one backbone with loss $L = D + \lambda R$ .
Detection wants sameness while identity wants difference, so $\lambda$ matters.
The fixture links the 0.91 cosine pair and splits the 0.42 pair at threshold 0.5.
One network runs faster than detect-then-identify pipelines.
Small blurry boxes need stricter thresholds to stop swaps.