Core ML

Contrastive Learning

Two crops of one photo should land close together, crops of other photos far apart. Whatever you augment away is exactly what the model learns to ignore.

Two views of one photo get pulled together in the embedding space while every other photo in the batch gets pushed away, so identity organises the space rather than appearance

Why Does This Exist?

Four hundred thousand frames off a forest reserve's camera traps. Most are empty — a branch moved and the sensor fired. The rest hold deer, badgers, foxes and one persistent squirrel. You want a model that tells them apart, and a volunteer has identified 600 frames.

Self-supervised learning says invent the labels, so you mask patches and reconstruct them. It half-works. Reconstruction spends capacity wherever the pixels are, and in a camera-trap frame the pixels are bark, bracken and grass. Your model gets excellent at grass. Telling a deer from a fallow deer lives in ear shape and leg proportion — a few hundred pixels out of two million, weighted by a pixel loss no higher than the leaves behind them.

So drop reconstruction. Take one frame, make two views by cropping different corners and shifting the colours, then train the model so those two land at nearly the same point in the embedding space while views of other frames land elsewhere. No pixel gets predicted. One question: which candidate here is the other view of me?

That buys a space organised by identity instead of appearance. It charges three design choices, plus a failure mode that reads as success.

Think of It Like This

Pairing socks without knowing what a sock is

Tip a laundry basket out and pair it up. You never define "sock type". You make one judgement only: these two go together, that one doesn't.

Notice what you quietly agreed to overlook. One is inside out, one has a bobble, one came out stretched. You ignored all of it, and that decision is your definition of a pair. Decide colour doesn't count and you'll cheerfully pair black with navy — fine until someone asks you to sort the drawer by colour.

There's also a degenerate way to finish: declare everything matches everything. Basket paired, drawer useless. The only thing stopping you is having to reject the socks that genuinely don't belong.

How It Actually Works

The loss is classification in disguise

Hand the model an anchor — one view of one frame — and a set of candidates: the anchor's partner view, the positive, plus views of other frames, the negatives. Score each candidate by cosine similarity to the anchor, divide by a constant, softmax the scores, and penalise the model for not putting its mass on the positive.

\mathcal{L} = -\log \frac{\exp\!\left(\operatorname{sim}(z, z^{+})/\tau\right)}{\sum_{j=1}^{K}\exp\!\left(\operatorname{sim}(z, z_{j})/\tau\right)}

That's InfoNCE. $z$ is the anchor's embedding, $z^{+}$ the positive's, $z_{j}$ runs over all $K$ candidates including the positive, $\operatorname{sim}$ is cosine similarity, and $\tau$ is the temperature, a positive number you pick. Look at the shape: cross-entropy over $K$ classes, where the correct class is "candidate three is my partner". Labels came from which crop came out of which frame.

Positives, negatives, temperature

In order of how much they matter.

Positives come from augmentation, and this is the lever people underweight. Whatever transformation makes the second view, you're telling the model that transformation does not change identity. Random cropping says position and scale are irrelevant; colour jitter says hue is. Both are right for species ID, because a badger is a badger in any light — and both are wrong the instant your task depends on what you threw away. The augmentation is the inductive bias, and it's usually chosen by copying a config file.

Negatives need to be plentiful and ideally hard. If every negative sits far away the softmax is already confident, the gradient is near zero, and you're burning compute. That's why early image methods needed batches in the thousands: the batch was the negative set. Memory banks and momentum encoders break that link.

Temperature divides every similarity before the softmax, setting how peaked the result is. A small $\tau$ concentrates nearly all the push on the closest negative; a large one spreads it thin. Genuinely sensitive, as the code below shows.

Collapse, and the fixes that skip negatives

One solution satisfies every positive pair perfectly: map every input to the same vector. Distance zero, loss content, model useless. Negatives are the only thing ruling it out.

Can you drop negatives and block collapse another way? Two approaches hold up. Put a predictor head on one branch and stop the gradient on the other, so the branches can't agree on a constant. Or penalise the embedding dimensions for correlating, which spreads the representation out by construction. Both work, and both beat shopping for a bigger batch size.

Show Me the Code

Temperature, made concrete. Six candidates, one of them a hard negative — the same clearing, no deer in it.

1import numpy as np2
3
4def weights(anchor: np.ndarray, cands: np.ndarray, tau: float) -> np.ndarray:5    u = anchor / np.linalg.norm(anchor)6    c = cands / np.linalg.norm(cands, axis=1, keepdims=True)7    z = (c @ u) / tau                    # cosine similarity, then divide by temperature8    z = z - z.max()                      # shift before exp, or a small tau overflows9    p = np.exp(z)10    return p / p.sum()                   # InfoNCE is cross-entropy over exactly this11
12
13rng = np.random.default_rng(5)14anchor = rng.normal(size=32)15cands = rng.normal(size=(6, 32))                 # six candidates from one batch16cands[0] = anchor + 0.4 * rng.normal(size=32)    # the positive: another crop of this photo17cands[1] = anchor + 1.1 * rng.normal(size=32)    # a hard negative: same clearing, no deer18for tau in (0.05, 0.2, 1.0):19    p = weights(anchor, cands, tau)20    print(f"tau {tau:.2f}  positive {p[0]:.3f}  hardest negative takes {p[1] / p[1:].sum():.0%} of the push")21# -> tau 0.05  positive 0.989  hardest negative takes 100% of the push22# -> tau 0.20  positive 0.722  hardest negative takes 85% of the push23# -> tau 1.00  positive 0.294  hardest negative takes 33% of the push

At $\tau = 0.05$ the other four negatives receive nothing. One number, and the loss went from spreading its attention to obsessing over one frame.

Watch Out For

An augmentation that deletes the signal you needed

You pretrain on camera-trap frames with the standard recipe: random resized crop, horizontal flip, strong colour jitter, grayscale sometimes. Species ID transfers beautifully. Then someone asks the same encoder to grade berry ripeness from the same cameras, and it's barely better than guessing. Not capacity — you spent 400,000 frames teaching the model that colour carries no information.

The tell is a representation that transfers well to one task and badly to another it should have suited. List your augmentations, name the property each declares irrelevant, and check that list against every downstream task you have in mind. Where two tasks disagree you need two encoders or a weaker augmentation, not a longer run.

Collapse hiding behind a healthy loss curve

Training looks fine. Loss falls, no spikes, nothing in the logs. Then a linear probe scores at chance and you find every frame mapping to nearly the same 128 numbers. Of course the loss fell — when all embeddings coincide every positive pair is close, and a weak negative term never bites hard enough to matter.

Loss cannot see this, so watch the geometry. Compute the per-dimension standard deviation of the embeddings across a batch and log its mean every few hundred steps: healthy training holds it roughly steady, collapse walks it toward zero. Set that alert before you launch — collapse is cheap to spot early and expensive to discover at the probe.

The Quick Version

Contrastive methods never reconstruct. They rank candidates: which of these is the other view of me?
InfoNCE is cross-entropy over similarities, so the model is classifying which crop matches which.
Positives come from augmentation, and the augmentation defines what the representation ignores. A design decision, not preprocessing.
Negatives should be many and hard. Batch size mattered early on because the batch was the negative set.
Temperature decides how much the loss fixates on the nearest negative. Not a knob to set by feel.
Collapse hides behind a falling loss. Stop-gradient or decorrelation prevent it without negatives.