Self-Supervised Learning

Hide part of your own data and train the model to predict what you hid. The answer was already there, so you get supervision nobody had to pay for.

Masking one word in an unlabelled sentence turns the hidden word into the training target, so the data labels itself

Why Does This Exist?

Picture a wind farm operator with 4.1 million service reports going back fourteen years. You want a model that reads a report and says whether it describes a gearbox failure. Exactly 900 reports carry that label. The other 4,099,100 carry nothing.

That gap is the whole problem. Supervised learning needs one target per example, and labelling the pile by hand would cost millions, then cost it again for the next question you want answered. Meanwhile the models that read text well were trained on hundreds of billions of words. Nobody labels at that scale. It cannot be bought.

So stop asking humans and start asking the data. Reports already carry structure: word order, the vocabulary of bearings versus blades, the fact that "replaced the" is followed by a noun. Cover part of a report, make the model predict what you covered, and grade it against the piece you hid. You knew the answer because you hid it. That exercise is called a pretext task: genuine supervised training on targets no person produced.

The one idea this page stands on is the supervised bargain: a model learns from pairs of inputs and targets. Self-supervision keeps the bargain and manufactures the targets. This page covers how the targets get made and what the model absorbs along the way. Its sibling semi-supervised learning spends a small labelled set instead, and contrastive learning covers the family that never reconstructs anything.

Think of It Like This

Covering one square of a street map

Take a street map of a city you have never visited. Cover one square with your thumb, guess what sits underneath from the streets running into it, then lift your thumb and check. No teacher and no answer sheet. The map was the answer sheet the entire time.

Repeat that a few thousand times and something besides guessing improves. You start knowing the city: the river bends south, the grid tilts near the docks, wide roads meet at roundabouts and narrow ones do not. None of that was the exercise. It fell out of the exercise, because you cannot guess a covered square without it.

The analogy stops in one place. A city sits still, so re-covering the same map teaches less each visit. Language never sits still, and a model can also memorise its way to good guesses without learning anything reusable. That is why the masking pattern changes every pass and why a falling pretext loss is never the measure. The exercise is disposable. What you built to pass it is what you keep.

How It Actually Works

Turning hiding into labels

Take one sentence, converted to integer token ids. Replace some positions with a mask id, feed the damaged version in, and ask for a probability distribution over the vocabulary at each hidden slot. Score it against the tokens you removed:

\mathcal{L}(\theta) = -\sum_{i \in M} \log p_\theta(x_i \mid x_{\setminus M})

In plain words: $M$ is the set of positions you covered, $x_i$ is the token that sat there, $x_{\setminus M}$ is everything left visible, and $p_\theta$ is the model's predicted probability under its current weights $\theta$ . The loss is ordinary cross-entropy, the same loss you would use for any classifier over a vocabulary. Nothing in it refers to a person.

Two families, and what each is good at

Masked prediction hides pieces and rebuilds them. Language models do it with word tokens, masked autoencoders do it with image patches, and it shines on local structure: which word, which patch, which nearby detail. It is dense too, since every hidden slot is a separate training signal.

Contrastive methods rebuild nothing. They make two altered views of one example and train the model so those two land close together in vector space while other examples spread apart. That tunes whole-example identity rather than fine detail, which is why it took over image work first. The full treatment is in contrastive learning.

Both are genuinely supervised. Neither uses a label anyone wrote down.

Worked example

Take a 100 word report and mask 15 positions, the standard 15 percent recipe. At one masked slot, the model puts probability 0.70 on the true word. Using the natural log, that slot contributes:

-\ln(0.70) = 0.357

Fifteen slots means fifteen graded predictions from a single report nobody annotated. Change which 15 you mask and the same report becomes a fresh exercise, which is why the training set is far bigger than the document count suggests. The ceiling is maskings, not documents.

Watch Out For

A pretext task with a lazy shortcut

Predicting image rotation sounds reasonable until you notice outdoor photos put bright sky at the top, so the model learns to find the bright edge and never learns about objects. Masked-pixel rebuilding has the same hole: neighbouring pixels look nearly identical, so copying them scores well and teaches nothing. The symptom is unmistakable: pretext loss falls smoothly while a downstream probe sits flat.

Make the shortcut unavailable. Mask contiguous blocks instead of scattered pixels, mask at a high rate so local copying cannot cover it, and strip colour cues the model could exploit. Ask one question of any pretext task before spending compute on it: what is the laziest way to score well, and does that route require the structure you want?

Picking checkpoints by pretext loss

You pretrain for 90,000 steps, notice step 40,000 had lower masked-token loss, and ship that checkpoint. Later it underperforms the one you threw away. Pretext loss measures skill at the fake task, and past a point the two curves stop moving together.

Judge checkpoints downstream instead. Freeze the encoder, train a small head on your labelled reports, and report that number for every checkpoint you compare. It costs a rounding error next to pretraining. With no labelled set at all, assemble a few hundred examples first: flying on pretext loss alone is guessing.

The Quick Version

Labels were the binding constraint, so self-supervision manufactures targets out of the data's own structure.
A pretext task hides part of an example and predicts it. The hidden part is the label, and training on it is exactly as supervised as any classifier.
Masked prediction handles local detail with dense signal per example. Contrastive methods handle whole-example identity.
It works because prediction has prerequisites. Guessing a hidden word forces the weights to encode syntax, association, and context.
You traded human labels for compute. Usually a good trade, still a trade.
The pretext task is a proxy. Good rebuilding does not imply good features for your task, so always probe downstream.