Siamese Trackers Follow Any Target

A siamese tracker learns a similarity check once, then follows any new target by matching its first-frame patch against each later frame.

A siamese tracker scores every search patch by cross-correlation with the first-frame template, and the 5.0 patch wins over the 1.0 distractor

Why Does This Exist?

Tracking has to follow an object it has never seen before. A detector knows fixed classes, but here the target is defined at test time by one box in the first frame. Retraining per video is far too slow.

Siamese trackers solve this by learning a generic similarity function offline. At test time they compare the fixed template from frame one against candidate patches in each new frame with single-object tracking logic, using the same matching idea as siamese networks. The later proposal refinement in SiamRPN builds directly on this base.

Think of It Like This

A fabric swatch held against coats

You cut a small swatch from a coat in a shop and walk the racks holding it against every coat you pass. You are not judging coats in general, only asking which one matches this exact swatch under this light.

It works until the coat turns inside out. The swatch still matches the outer fabric, but the visible side changed completely. That is where the analogy stops: trackers also fail when appearance changes more than similarity can bridge.

How It Actually Works

Two identical branches embed the template $z$ and each search patch $x$ into vectors, where $z$ is the first-frame target and $x$ is one candidate. The score is cross-correlation, written $s = z \star x$ , which for flat vectors is the dot product $s = z \cdot x$ . The patch with the highest $s$ becomes the new box.

Worked example

Take a tiny template $z = [2, 1]$ . Two candidates appear in the next frame: patch A $x_A = [2, 1]$ and patch B $x_B = [0, 1]$ .

Scores:

$s_A = 2 \times 2 + 1 \times 1 = 5$
$s_B = 2 \times 0 + 1 \times 1 = 1$

Patch A wins 5.0 to 1.0, so the box moves to A. No classifier ran. Only two dot products decided it.

Watch Out For

Never updating the template during occlusion

A fixed template goes stale when the target rotates or lights change. The symptom is slow drift followed by a jump to the background. Fix it with a conservative update: blend in the new patch with a small rate such as 0.01, and freeze the update while the peak score stays low.

The Quick Version

A siamese tracker follows any target given by one first-frame box, with no per-video training.
Template and search patches share one embedding network, and similarity is cross-correlation.
The worked fixture scores 5.0 on the true patch against 1.0 on the distractor.
Speed comes from fully convolutional search, but fixed templates drift under appearance change.
Update conservatively and freeze on low scores to survive occlusion.