SimCLR Contrastive Pretraining

Show the net two random crops of one photo and teach it they match, while crops from other photos do not. Matching without labels builds strong features.

SimCLR pulls two crops of one photo together and pushes other photos apart

Why Does This Exist?

Labeling millions of photos costs too much, yet unlabeled photos are free. SimCLR turns each photo into its own lesson: two views of photo $i$ must agree more than views of different photos. A siamese network gives the shared encoder, and contrastive loss gives the push-apart rule. Together they learn features that later classify with few labels.

This page covers the SimCLR recipe only: strong crops, projection head, and NT-Xent over a large batch.

The one idea you need first.

An embedding $z$ is a short vector that stands for a view. Cosine similarity $\mathrm{sim}(u,v) = u \cdot v / (\lVert u \rVert \lVert v \rVert)$ says how alike two embeddings are, from $-1$ to $1$ .

Think of It Like This

Two blurred photos of one dog in a lineup

You hold two blurry snapshots of one dog and face a lineup of many dogs. Your job is to point at the pair that came from the same animal. Ears, spots, and posture survive the blur, so you match them and ignore the rest.

SimCLR is that game played millions of times. Crops are the blur, the lineup is the batch, and the prize is a feature reader that knows dogs from cats before it ever sees a label. Where it stops: a real lineup has one match, while SimCLR treats every other photo as a non-match even when two dogs look alike.

How It Actually Works

From each photo make two random views with crop, color jitter, and blur. Encode both with shared backbone $f$ plus small projection head $g$ to get $z_i$ and $z_j$ . For a batch of $N$ photos there are $2N$ views. The NT-Xent loss for pair $(i,j)$ is

L_{i,j} = -\log \frac{\exp(\mathrm{sim}(z_i,z_j)/\tau)}{\sum_{k \ne i} \exp(\mathrm{sim}(z_i,z_k)/\tau)}

In plain words: $\tau$ is the temperature that sharpens scores, the top term rewards the true mate, and the bottom sum runs over all $2N-1$ other views. Small $\tau = 0.5$ makes hard negatives count more.

Worked example

Say $N = 4$ , so $8$ views. For one anchor let the mate score $0.92$ and seven negatives score $0.10, 0.20, 0.05, 0.30, 0.12, 0.18, 0.25$ . With $\tau = 0.5$ , mate term is $\exp(1.84) = 6.30$ . Negative terms are $\exp(0.2)=1.22$ , $\exp(0.4)=1.49$ , $\exp(0.1)=1.11$ , $\exp(0.6)=1.82$ , $\exp(0.24)=1.27$ , $\exp(0.36)=1.43$ , $\exp(0.5)=1.65$ , summing to $9.99$ . Denominator is $6.30 + 9.99 = 16.29$ . Loss is $-\log(6.30/16.29) = -\log(0.387) \approx 0.95$ . Only a higher mate similarity or weaker negatives lowers it.

Watch Out For

Weak crops that make the task trivial

Small crops with no color change let the net match by average brightness. Accuracy looks fine yet transfer flops. The symptom is high pair accuracy with poor image classification later. Use strong random crops plus color jitter and Gaussian blur together.

Judging the projection head features as final

SimCLR trains $g(z)$ for the loss but downstream uses $f$ before $g$ . Reading off $g$ throws away about two accuracy points. Always freeze $f$ , drop $g$ , and probe or fine-tune from $h = f(x)$ .

The Quick Version

SimCLR learns from two augmented views of one photo treated as a positive pair.
NT-Xent with temperature 0.5 contrasts the mate against all other views in the batch.
Strong crops and a nonlinear projection head carry most of the gain.
Large batches supply the negatives, so small batches need memory tricks.
Use the backbone output, not the projection output, for later tasks.