BYOL Without Negatives

Train one net to predict what a slow copy sees in another crop of the same photo. No strangers needed, so collapse must be stopped another way.

BYOL predicts the slow target view without any negative pairs

Why Does This Exist?

Negative pairs cost memory and can wrongly push two dogs apart. BYOL asks whether positives alone suffice. It keeps an online net plus a slow target, predicts across views, and blocks gradient into the target. The siamese network shape stays, but the contrastive loss goes away.

This page covers predictor plus stop-gradient only. Cluster variants live in pages like SwAV, linked from image retrieval.

The one idea you need first.

Stop-gradient means the target branch gives a number but takes no update from this loss. Only the online weights move by gradient; the target follows by slow momentum $m = 0.996$ .

Think of It Like This

Tracing a teacher who walks very slowly

A student traces drawings taped on a slow walker. The walker moves a little each day, so yesterday tracing still helps today. The student never sees other walkers, yet her hand gets steadier because the target keeps drifting just out of reach.

The online net is the student, the target is the walker, and the predictor is her pencil grip. Where it stops: a real student could freeze her hand and claim success, while BYOL asymmetry keeps that trivial answer unstable.

How It Actually Works

Encode view $v_1$ with online encoder $f$ then predictor $q$ to get $p = q(f(v_1))$ . Encode view $v_2$ with target encoder $f_t$ to get $z = f_t(v_2)$ , with no gradient into $z$ . Minimize $L = 2 - 2 \cdot \mathrm{sim}(p,z)$ , which is squared error on unit vectors. Update $f_t = m f_t + (1-m) f$ with $m = 0.996$ .

In plain words: $q$ adds the asymmetry, $m$ sets target slowness, and $L$ is zero only when prediction matches the slow view.

Worked example

Let unit vectors $p = [0.8, 0.6]$ and $z = [0.85, 0.53]$ . Norms are $1.0$ and $\sqrt{0.7225+0.2809} = \sqrt{1.0034} \approx 1.002$ . Dot is $0.68 + 0.318 = 0.998$ . Cosine is $0.998/1.002 \approx 0.996$ . Loss is $2 - 2 \times 0.996 = 0.008$ . A collapsed constant gives high error once the target drifts, so only tracking views keeps loss this low.

Watch Out For

Removing the predictor and collapsing quietly

Without $q$ , online and target can match by outputting one constant. Loss hits zero while features carry nothing. The symptom is zero loss plus chance-level probes. Keep the predictor and check linear probes, not just the loss.

The Quick Version

BYOL matches an online prediction to a slow momentum target with no negatives.
Predictor plus stop-gradient supply the asymmetry that stops collapse.
Momentum 0.996 keeps the target stable across steps.
Loss is cosine distance on unit vectors.
Probe the encoder, since the loss alone can hide collapse.