Contrastive Learning

Two crops of one photo should land close together, crops of other photos far apart. Whatever you augment away is exactly what the model learns to ignore.

Two crops scoring 0.85 pull together while three batch mates scoring 0.30, 0.10 and 0.05 push apart, giving loss 0.56 at temperature 0.5

Why Does This Exist?

Labels cost time from people who know the answer. Unlabelled photos and sentences sit everywhere for free, so you want a way to turn that raw pile into features you will reuse later.

Rebuilding is one answer. Hide part of an input and ask the model to repaint it. That teaches texture and syntax, but it spends capacity wherever the pixels are. A model that repaints leaves has not proved it knows which animal walked past.

Contrastive learning skips the repaint. Take one photo, make two altered views, and teach the model those two belong together while views from other photos do not. No pixel gets predicted. The task is ranking: which candidate here is my partner?

This page covers that ranking rule, called InfoNCE, and the three choices that decide what it learns: how you build positives, how you pick negatives, and how temperature sharpens the scores. The family it belongs to is self-supervised learning. The full image recipe with strong crops and a projection head is SimCLR. A text reader trained the same way is sentence embeddings.

You need one idea first. An embedding is a short vector that stands for one view. Cosine similarity says how alike two embeddings point, from minus 1 for opposite to 1 for identical.

Think of It Like This

Pairing socks from a laundry pile

You tip a laundry basket out and pair socks. You never define sock type. You judge only this goes with this, that does not.

You quietly agree to overlook differences. One is inside out, one is stretched, one has a bobble. You ignore all of it, and that choice is your definition of a pair. Decide colour does not count and you will pair black with navy without blinking.

Two failure shapes show up here too. Declare everything matches and you finish fast with a useless drawer. And your colour blindness is fine until someone asks you to sort by colour. Then the very thing you taught yourself to ignore is the task.

Where it stops: socks arrive in true pairs, photos do not. Your two views are invented by cropping, and the stop rule is yours to pick. Pick it to suit the job you will ask for next.

How It Actually Works

Hand the model an anchor, meaning one view. Offer a small lineup: its partner view, called the positive, plus views from other inputs, called the negatives. Score each candidate by cosine similarity to the anchor, divide by temperature, run softmax, and charge cross-entropy for not picking the partner.

\mathcal{L} = -\log \frac{\exp\!\left(\operatorname{sim}(z, z^{+})/\tau\right)}{\sum_{j=1}^{K}\exp\!\left(\operatorname{sim}(z, z_{j})/\tau\right)}

Here $z$ is the anchor embedding, meaning the vector for the view you start from. $z^{+}$ is the positive embedding, meaning the vector for its partner. $z_{j}$ runs over all $K$ candidates including the positive, where $K$ is the lineup size. $\operatorname{sim}$ is cosine similarity, meaning direction agreement ignoring length. $\tau$ is temperature, meaning one positive number you pick that sharpens the scores. $\mathcal{L}$ is the loss, meaning the number you minimise.

Positives decide what to ignore

Positives come from data augmentation. Whatever change makes the second view, you tell the model that change does not alter identity. Random cropping says position and scale do not matter. Colour jitter says hue does not matter. Both help when the job is species ID, and both hurt the day colour is the answer. The augmentation is the bias. Copying a recipe without asking what it discards is how good pretraining goes wrong.

Negatives and temperature set the pressure

Negatives need to be many and hard. Easy strangers sit far away, softmax turns confident, gradients go quiet, and you burn compute for nothing. Hard negatives close to the anchor keep the pressure on. That is why large batches helped early image work: the batch was the negative set.

Temperature divides every similarity before softmax and sets how peaked the result is. Small tau piles almost all push on the closest rival. Large tau spreads the push thin. You will feel this in the numbers next.

Worked example: one mate against three rivals

Take one anchor with four candidates. Similarities are 0.85 for the mate and 0.30, 0.10, 0.05 for three rivals. Temperature is 0.5.

Divide first: 1.70, 0.60, 0.20, 0.10. Exponentiate: exp(1.70) is 5.47, exp(0.60) is 1.82, exp(0.20) is 1.22, exp(0.10) is 1.11. Add for 9.62. The mate takes 5.47 divided by 9.62, which is 0.57. Loss is minus log of 0.57, which is 0.56.

\text{logits} = [1.70, 0.60, 0.20, 0.10], \quad \text{exps} = [5.47, 1.82, 1.22, 1.11]

p^{+} = 5.47 / 9.62 \approx 0.57, \quad \mathcal{L} = -\log(0.57) \approx 0.56

The diagram draws these same numbers: two crops scoring 0.85 linked by one pull, three batch mates scoring 0.30, 0.10 and 0.05 pushed outward, loss 0.56 at temperature 0.5. Lift the 0.30 rival and the loss climbs. Drop temperature and the closest rival eats more of the push.

Code

The point is the check: same similarities, same temperature, same 0.57 and 0.56.

1import numpy as np2
3sims: np.ndarray = np.array([0.85, 0.30, 0.10, 0.05])4tau: float = 0.55logits: np.ndarray = sims / tau6logits = logits - logits.max()  # shift keeps exp safe, leaves ratios unchanged7probs: np.ndarray = np.exp(logits) / np.exp(logits).sum()8loss: float = float(-np.log(probs[0]))9
10print(f"{probs[0]:.2f} {loss:.2f}")  # -> 0.57 0.56

The printout matches the hand run above: mate probability 0.57, loss 0.56. Change tau to 1.0 and watch the mate share fall.

Watch Out For

An augmentation that erases the signal you need

You pretrain with strong colour jitter because the recipe says so. Species ID transfers well. Then the same encoder grades ripeness and sits near chance. You taught it colour means nothing.

Symptom: one downstream task sings while another it should suit flops, and longer training does not close the gap. Fix: list each augmentation, name the property it declares irrelevant, and test that list against every job you plan. Where two jobs disagree, train two encoders or weaken the augmentation. Do not run longer.

Collapse that looks like training

Loss falls smoothly. No spikes, no warnings. A linear probe then scores near chance and every input maps to nearly the same point. Of course positives sit close: everything sits close.

Symptom: falling loss with flat downstream accuracy, and per-dimension spread of embeddings drifting toward zero across a batch. Fix: log that spread every few hundred steps and stop on geometry, not loss. Keep hard negatives in the batch, and if you drop negatives entirely, use a stop-gradient branch or a decorrelation penalty so constant output cannot win.

The Quick Version

Contrastive learning ranks candidates instead of rebuilding inputs: pick the partner view from the lineup.
InfoNCE is cross-entropy over cosine similarities divided by temperature, with the partner as the correct class.
Positives come from augmentation, and each augmentation declares one change irrelevant to identity.
Negatives should be many and hard, and temperature decides how much the closest rival dominates the push.
One worked batch scores 0.85 for the mate against 0.30, 0.10 and 0.05, giving mate probability 0.57 and loss 0.56 at temperature 0.5.
Collapse maps every input to the same point behind a falling loss, so track embedding spread during training.