Contrastive Learning

Generative self-supervised models such as autoencoders force networks to reconstruct raw pixel arrays, spending up to $80\%$ of their parameter capacity predicting high-frequency background foliage, camera noise, and lighting shifts. Contrastive learning shifts the objective from low-level pixel synthesis to high-level semantic representation: instead of repainting the scene, the network maps distinct views of the same object to invariant coordinates in an embedding space while pushing unrelated images apart.

Crop One Dog Twice

In self-supervised frameworks like SimCLR, positive pairs originate from a single unlabelled image $x$. Applying two stochastic data augmentation transforms $t, t' \sim \mathcal{T}$ (random resized cropping, color jitter, Gaussian blur, and horizontal flipping) produces two correlated views $\tilde{x}_i$ and $\tilde{x}_j$. A base convolutional or transformer encoder $f(\cdot)$ extracts feature representations $h$, which a non-linear projection head $g(\cdot)$ maps to unit-normalized latent vectors $z_i, z_j$. Because both views depict the same underlying entity, the objective applies an attractive gradient that pulls $z_i$ and $z_j$ together.

Score Beats With Numbers

The normalized temperature-scaled cross-entropy loss (NT-Xent / InfoNCE) measures alignment using cosine similarity $\text{sim}(u, v) = \frac{u^T v}{\|u\| \|v\|}$:

$\mathcal{L}_{i,j} = -\log \frac{\exp(\text{sim}(z_i, z_j) / \tau)}{\sum_{k=1}^{2N} \mathbb{1}_{[k \neq i]} \exp(\text{sim}(z_i, z_k) / \tau)}$

When a positive pair achieves cosine similarity $0.85$ against three negative batch strangers scoring $0.15, 0.10,$ and $0.05$, scaling by temperature $\tau = 0.5$ dramatically sharpens contrast:

• Positive: $\exp(0.85 / 0.5) = \exp(1.70) \approx 5.474$
• Negative 1: $\exp(0.15 / 0.5) = \exp(0.30) \approx 1.350$
• Negative 2: $\exp(0.10 / 0.5) = \exp(0.20) \approx 1.221$
• Negative 3: $\exp(0.05 / 0.5) = \exp(0.10) \approx 1.105$

The positive pair captures $\frac{5.474}{5.474 + 1.350 + 1.221 + 1.105} \approx 0.598$ ($60\%$) of the total softmax probability mass, heavily penalizing any drift away from alignment.

Push Strangers Apart

Optimization operates as a continuous push-pull physical system in latent space. For an anchor vector $z_i$, the attractive gradient pulls its augmented partner $z_j$ inward while repulsive gradients concurrently push all other batch instances (unrelated photos of cats, cars, or trees) outward across the unit hypersphere. Denominator competition forces the encoder to distribute negative representations uniformly across the spherical surface.

Clusters Form Without Labels

Iterating this push-pull dynamic over millions of unlabelled images produces emergent semantic structure. Augmented crops of different dog photographs repeatedly encounter similar feature compositions (snouts, fur textures, ears), while consistently being repelled from vehicles and landscapes. Consequently, dog representations spontaneously aggregate into a cohesive cluster, distinct from cats and cars, despite zero human category annotations.

Size Your Batch Right

Contrastive representation fidelity depends heavily on the volume of negative contrast:

• Large Mini-Batches (SimCLR): Scaling batch size to $N = 4,096$ provides $8,190$ negative candidates per step, yielding exceptionally sharp gradient signals at the expense of extreme multi-GPU memory requirements.
• Momentum Queues (MoCo): Decouples dictionary size from mini-batch size by caching embeddings in a continuous dynamic FIFO queue ($65,536$ keys) updated via a slowly moving momentum encoder, enabling contrastive pre-training on standard workstations.

Where It Breaks

Unsupervised contrastive learning operates under the naive assumption that every sample in the mini-batch other than the augmented partner is a true negative. When two independent photographs of dogs (such as a Golden Retriever and a Beagle) coincide in the same mini-batch, the loss function erroneously treats them as strangers. Strong repulsive gradients forcibly push both dogs toward opposite poles of the embedding space, tearing apart the emerging semantic cluster.

The Quick Version

• Pixel reconstruction wastes capacity on backgrounds; contrastive learning targets invariant meaning.
• Two augmented crops of a single image form a positive anchor pair.
• High similarity (0.85) scaled by temperature 0.5 dominates negative competitors (0.15, 0.10, 0.05).
• Unrelated images act as simultaneous negative strangers pushed outward.
• Semantic clusters form spontaneously on the unit hypersphere without supervision.
• Larger batches provide sharper contrast; memory banks (MoCo) decouple queue size from GPU limits.
• Co-occurring same-class instances trigger false negative collisions, repelling related concepts.