Barlow Twins Identity Target

Line up two views of one photo feature by feature, keep matching slots near one, and push different slots toward zero. Redundancy falls and useful factors stay.

Barlow Twins drives the cross-correlation of two views toward identity

Why Does This Exist?

Contrast can keep redundant features because any correlated pair still matches. Barlow Twins asks for more: each feature slot should match itself across views and ignore other slots. That cuts redundancy directly. Readers who know siamese networks get the twin shape; the new part is the matrix target, not a pair list.

This page covers the correlation loss only. Pair losses live in contrastive loss.

The one idea you need first.

Cross-correlation $C$ is a $d \times d$ matrix over a batch: $C_{ij}$ is the correlation of slot $i$ in view A with slot $j$ in view B. Identity means ones on the diagonal and zeros elsewhere.

Think of It Like This

Two choirs singing the same score in tune

Two choirs sing one score in separate rooms. Matching voice parts should agree note for note, while soprano should not copy bass. A judge scores tuning within parts and leakage across parts.

Diagonal ones are in-tune parts, off-diagonal zeros are no leakage, and weight $0.005$ sets how harshly leakage counts. Where it stops: choirs hear each other, while the two views train from separate crops.

How It Actually Works

Normalize projector outputs $z^A$ and $z^B$ over the batch to zero mean and unit variance per slot. Form $C = (Z^A)^T Z^B / N$ for batch size $N$ . Loss is $L = \sum_i (1-C_{ii})^2 + \lambda \sum_{i \ne j} C_{ij}^2$ with $\lambda = 0.005$ and width $d = 8192$ .

In plain words: $N$ averages over the batch, $d$ is the wide projector, and $\lambda$ trades invariance against decorrelation.

Worked example

Say $d = 2$ and measured $C = [[0.9, 0.2],[0.1, 0.85]]$ . Diagonal terms give $(0.1)^2 + (0.15)^2 = 0.01 + 0.0225 = 0.0325$ . Off-diagonal give $0.04 + 0.01 = 0.05$ , times $0.005$ is $0.00025$ . Total $0.03275$ . Training pushes diagonals to $1$ and wipes the $0.2$ leak, which is exactly redundancy leaving the code.

Watch Out For

Small projector that hides the gain

A 256-dim head cannot hold 8192 decorrelated slots, so off-diagonal pressure does little. The symptom is weak transfer despite low loss. Use a wide projector near 8192 and batch-norm on its output beforeStats.

The Quick Version

Barlow Twins matches two views by driving cross-correlation toward identity.
Diagonal ones demand view invariance per slot.
Off-diagonal zeros with weight 0.005 cut redundancy.
Wide projectors near 8192 dims matter.
No negatives, queues, or cluster passes are used.