Semi-Supervised Learning
Train on a handful of labelled examples, let that model guess labels for the rest, and keep only the guesses it is confident about. Confidence is the whole method.
Why Does This Exist?
Suppose a radiology department wants a model that flags chest scans needing urgent review. A radiologist has fully labelled 500 scans, urgent or not, and that took weeks. Forty thousand more scans sit in the archive, unlabelled, because nobody has weeks to spare on the rest.
Pure supervised learning trains on the 500 and ignores the other 40,000 entirely, which throws away nearly all the data you hold. Unsupervised learning could chew through all 40,500 scans, but it has no notion of urgent: nothing in the raw pixels says which grouping matters. Neither method uses what you actually have, which is a little supervision plus a lot of raw signal.
Semi-supervised learning is the method that uses both. Train on the 500 labelled scans, let that model predict on the 40,000 unlabelled ones, and treat its most confident predictions as if they were real labels. The archive stops being dead weight and starts being a teacher, one careful round at a time.
The one idea this page stands on is the supervised bargain: targets teach, and targets are expensive. This page covers spending 500 of them to put 40,000 to work. Its mirror image, self-supervised learning, manufactures targets from the data itself, and active learning picks which examples deserve labelling next.
Think of It Like This
A teaching assistant with one marked stack and a huge unmarked box
A teaching assistant grades 20 essays properly, checking every argument line by line. Then a box of 2,000 more essays arrives with no time to grade them the same way. Ignoring the box wastes it. So the assistant skims each essay fast, guided by what the 20 marked ones taught about strong and weak arguments.
Some essays in the box are obviously strong or obviously weak, and those calls get marked without a second look. The middle ones, vague thesis and uneven argument, get set aside rather than guessed at. The confident calls join the marked pile and sharpen the assistant's judgment for the next skim through what remains.
The analogy stops where grading stops being reversible. A wrongly marked essay can be regraded next term, but a wrong pseudo-label retrains the very judge that accepted it, so each mistake makes the next one easier to accept. That is why the bar stays high and the middle pile stays set aside. Only trust the calls that were never close.
How It Actually Works
The pseudo-labelling loop
Train a model on the 500 labelled scans. Run it over the 40,000 unlabelled ones and read off a predicted probability per scan. Keep the predictions above a confidence bar, say 0.95 urgent or below 0.05, and add those scans to the training set with their predicted label attached. That attached label is a pseudo-label. Retrain on the combined set and repeat, each round pulling in whatever now clears the bar.
The threshold is the whole method
Set the bar high and you accept few new pseudo-labels, but nearly all are correct: safe, slow progress. Set it low and you pull in far more of the archive, but a growing share of those pseudo-labels are wrong, and the model now trains on its own mistakes as if they were ground truth. No setting is simply right. The threshold is a dial between how much of the archive you use and how much noise you invite into training.
A softer lever: consistency
Pseudo-labelling commits to a hard label the moment confidence clears the bar. Consistency regularization never commits to one. Take an unlabelled scan, apply two different small changes such as a slight rotation and a brightness shift, and penalise the model when its two predictions on the same underlying scan disagree. No label is ever assigned. The model is just pushed toward answering the same way regardless of the perturbation. The two techniques combine well rather than competing.
Worked example
Suppose round one pushes 12,000 of the 40,000 scans past the 0.95 bar. At 95 percent precision on those accepted guesses, the new training data carries:
wrong labels. Now drop the bar so 30,000 scans clear it at 88 percent precision:
wrong labels. Accepting two and a half times more of the archive multiplied the injected errors sixfold, from 600 to 3,600. That multiplication is the dial from the previous section, and it is why the bar moves only with a held-out labelled check watching every round.
Watch Out For
Letting wrong pseudo-labels compound across rounds
A wrong pseudo-label accepted in round one does not just sit there. The model retrains on it, and if that mistake lands where the model was already unsure, the region looks more settled than it is, so the next wrong prediction nearby clears the bar more easily. Each bad label makes the next one easier to accept, and the loop can finish worse than the 500 labelled scans alone.
Track pseudo-label accuracy against a small held-out labelled set every round, not just at the end. A quietly rising error rate there is the first sign the loop is teaching itself something false. When you see it, raise the bar or stop the loop.
Confidently rubber-stamping an imbalanced pool
If 95 percent of the archive is routine and 5 percent urgent, even mild bias toward the majority class labels almost everything routine with high confidence, and pseudo-labelling agrees with itself round after round. The imbalance entrenches instead of correcting.
Compare the class balance of accepted pseudo-labels against the labelled set each round. A widening gap means the loop is amplifying a skew, not learning from data. Rebalance what you accept before the majority drowns the minority entirely.
The Quick Version
- Semi-supervised learning trains on a small labelled set plus a much larger unlabelled pool, using the model's own confident predictions as extra training signal.
- Pseudo-labelling commits to a hard label above a confidence bar. Consistency regularization never commits and penalises disagreement across altered views instead.
- The threshold trades accepted volume against pseudo-label error. No setting is universally correct.
- Errors compound: each wrong pseudo-label makes the next wrong prediction in that region easier to accept.
- Without a held-out labelled check each round, the loop can finish worse than the labelled set alone.