Core ML

Self-Supervised Learning

Hide part of your own data and train the model to predict what you hid. The answer was already sitting there, so you get supervision that nobody paid for.

Masking part of an unlabelled example turns the hidden piece into the training target, so the data supplies its own label and no human writes any of it

Why Does This Exist?

A wind farm operator hands you 4.1 million service reports, typed by technicians standing in the nacelle with cold hands, going back fourteen years. You want a model that reads one and says whether it describes a gearbox failure.

Exactly 900 of those reports have been labelled for that. The other 4,099,100 have nothing.

Now the wall, and it's arithmetic rather than clever. Supervised learning needs one target per example. A technician who actually knows gearboxes can tag a report for maybe two dollars once you count review, so tagging the pile runs past eight million dollars — and you'd pay it again for the next question you want answered. Meanwhile the models that read text well at all were trained on hundreds of billions of words. Nobody labels that.

So flip it. The reports already carry structure: word order, the vocabulary for bearings versus blades, the fact that "replaced the" is followed by a noun. Hide a piece of a report and make the model predict what you hid. You know the answer, because you covered it up. That's a pretext task — real supervised training, on targets no human produced.

Think of It Like This

Covering up squares on a city map

Take a street map of a city you don't know. Cover one square with your thumb, guess what's underneath from the streets running into it, then lift your thumb and check. No teacher, no answer sheet. The map was the answer sheet the whole time.

Do that a few thousand times and something else happens. You stop getting better at guessing squares and start knowing the city: the river bends south, the grid tilts near the docks, big roads meet at roundabouts and small ones don't. None of that was the exercise. It fell out of the exercise, because you can't guess a covered square without it.

That's the bet. The exercise is disposable, and what you built to pass it is what you keep.

How It Actually Works

Inventing the label

Take one report, tokenised into integer ids. Replace a quarter of the positions with a [mask] id, feed the damaged version in, and ask for a distribution over the vocabulary at each hidden slot. Score it against what you removed.

\mathcal{L}(\theta) = -\sum_{i \in M} \log p_\theta\!\left(x_i \mid x_{\setminus M}\right)

$M$ is the set of positions you covered, $x_i$ is the token that was really there, $x_{\setminus M}$ is everything you left visible, and $p_\theta$ is the model's probability under its current weights. Ordinary cross-entropy, the same loss you'd reach for with any classifier over a vocabulary. Nothing in it refers to a person.

Two families, and what each is good at

Masked prediction hides pieces and reconstructs them. BERT does it with word tokens, masked autoencoders do it with image patches, and it shines when what you care about is local: which word, which patch, which bit of nearby structure. It's dense, too — every hidden slot is a separate signal.

Contrastive methods reconstruct nothing. They build two views of one example and train the model so those two land close in the embedding space while other examples spread out. That tunes the representation to whole-example identity rather than fine detail, which is why it took over image work first. Contrastive learning gets its own page.

Both are genuinely supervised. Neither uses ground truth anyone wrote down.

Why hiding a word teaches anything at all

Take the masked slot in "replaced the [mask] after a vibration alarm on unit 12". To put probability on gearbox instead of anemometer, the model needs three separate things encoded in its weights: that the slot wants a noun, that vibration alarms travel with gearboxes and not with anemometers, and that "unit 12" narrows down which turbine model is in play.

None of that is optional. The prediction depends on it, so the gradient drives those structures into the weights whether you asked for them or not. That's the mechanism — not "the model learns to understand text", but the model learns whatever the prediction happened to require.

Show Me the Code

The label generator consults nobody. Four lines.

1import numpy as np2
3
4def mask_batch(tokens: np.ndarray, rate: float, seed: int) -> tuple[np.ndarray, np.ndarray]:5    """Hide part of the report. What you hid becomes the target."""6    rng = np.random.default_rng(seed)7    hidden = rng.random(tokens.shape) < rate       # no human is consulted anywhere here8    inputs = np.where(hidden, -1, tokens)          # -1 is the [mask] id9    return inputs, np.where(hidden, tokens, -100)  # -100 means "no loss on this slot"10
11
12log = np.array([7, 12, 3, 44, 9, 21, 5, 60, 18, 2])  # one tokenised service report13inputs, targets = mask_batch(log, rate=0.25, seed=2)14print(inputs)15print(targets)16print(f"{int((targets != -100).sum())} supervised predictions, 0 human labels")17# -> [ 7 12  3 -1  9 21 -1 -1 18  2]18# -> [-100 -100 -100   44 -100 -100    5   60 -100 -100]19# -> 3 supervised predictions, 0 human labels

Three graded predictions out of one ten-token report. Change the seed and the same report becomes a different exercise, which is why 4.1 million reports is a far bigger training set than it looks: the ceiling is maskings, not documents.

Watch Out For

A pretext task the model can shortcut

Predicting image rotation sounds reasonable until you notice outdoor photos have a bright sky at the top, so the model learns "find the bright edge" and never learns about objects. Masked-pixel reconstruction has the same hole: neighbouring pixels are nearly identical, so copying them scores well and teaches nothing. The tell is pretext loss dropping smoothly while downstream accuracy sits flat.

Make the shortcut unavailable. Mask contiguous blocks instead of scattered pixels, mask at a high rate so local copying can't cover it, and drop colour cues the model could exploit. Ask one question of any pretext task before you spend compute on it: what's the laziest way to score well, and does that route require the structure I want?

Choosing checkpoints by pretext loss

You pretrain for 90,000 steps, notice step 40,000 had a lower masked-token loss, and ship that checkpoint. Later it underperforms the one you threw away. Pretext loss measures skill at the fake task, and past a point the two curves stop moving together.

The only honest read is a downstream probe. Freeze the encoder, train a linear head on your 900 labelled reports, and report that number for every checkpoint you compare. It costs a rounding error next to pretraining. With no labelled set at all, get one — a few hundred examples of weak supervision beats flying on pretext loss.

The Quick Version

Labels were the binding constraint, so self-supervision manufactures targets out of the data's own structure.
A pretext task hides part of an example and predicts it. The hidden part is the label, and it's exactly as supervised as any other classifier.
Masked prediction handles local detail with dense signal per example. Contrastive methods handle whole-example identity.
It works because the prediction has prerequisites. Guess a hidden word and you're forced to encode syntax, association and context.
You traded human labels for GPU hours. Usually a good trade, still a trade.
The pretext task is a proxy. Good reconstruction does not imply good features for your task, and closing that gap is open research.