Self-Supervised Learning
Labelled data is the bottleneck of machine learning. See how pretext tasks let models generate their own labels from unlabelled data to learn useful representations.
Labelled data is the bottleneck of machine learning. Getting humans to tag millions of examples is slow and expensive. But we have access to near-infinite unlabelled data—if only we could use it to train our models.
Pretext Tasks
Self-supervision solves this by inventing a pretext task. You hide or distort part of the input, and force the model to reconstruct it. The data itself provides the answer, so you get a supervised training signal without paying any humans for labels.
Masked Prediction
In language, we mask out words and ask the model to predict them. To guess "cat" successfully, the model is forced to learn syntax, context, and semantics. The prediction task is disposable, but the representations it learns are incredibly valuable.
Next-Frame Prediction
For video and robotics, the pretext task might be predicting the next frame. To predict what happens next, the model has to implicitly learn physics, object permanence, and motion—all without a single manual label.
Where It Breaks
Self-supervised learning only works if the pretext task is hard enough to force deep understanding. If the model can cheat—like predicting the mean pixel color—it will learn a weak representation. A trivial pretext task fails to transfer to downstream applications.
The Quick Version
- Human-labelled data is expensive; unlabelled data is virtually infinite.
- Pretext tasks create labels directly from the input data.
- Masking parts of the input forces the model to learn deep structure.
- Predicting the future teaches physics and logic.
- A trivial pretext task produces representations that fail to transfer.