Core ML

Semi-Supervised Learning

Train on a handful of labeled examples, let that model guess labels for the rest, and keep only the guesses it's confident about. Confidence is the whole method.

A small labeled set and a large unlabeled pool both train a model; the model then labels the unlabeled pool itself, and only its most confident guesses rejoin the training set for another round

Why Does This Exist?

A radiology department wants a model that flags chest X-rays needing urgent review. A radiologist has fully labeled 500 scans — genuinely urgent, or not — and that took weeks. Forty thousand more scans sit in the archive, unlabeled, because nobody has weeks to spare on the rest.

Full supervised learning trains on the 500 and ignores the other 40,000 entirely, which throws away almost all the data you actually have. Unsupervised learning could chew through all 40,500 scans, but it has no notion of "urgent" — nothing in the data says which cluster matters. Neither method uses what you actually have: a little supervision and a lot of raw signal.

Semi-supervised learning is the method that uses both. Train on the 500 labeled scans, then let that model make predictions on the 40,000 unlabeled ones, and treat its most confident predictions as if they were real labels. The archive stops being dead weight.

Think of It Like This

Grading a small marked stack, then skimming a much bigger one

A teaching assistant fully grades 20 essays, checking every argument line by line. Then a box of 2,000 more essays arrives with no time to grade them properly. Instead of ignoring the box, the assistant skims each one fast, using what the 20 marked essays taught about what a strong or weak argument looks like.

Some essays in the box are obviously strong or obviously weak — the assistant is confident and marks them without a second look. Others sit in the middle: vague thesis, uneven argument, hard to call. Those get set aside rather than guessed at. The confident calls join the marked pile, sharpening the assistant's sense of what "strong" looks like for the next skim through what's left.

That's the whole method: use a small amount of real judgment to make a fast call on the rest, and only trust the calls that weren't close.

How It Actually Works

The pseudo-labelling loop

Train a model on the 500 labeled scans. Run it over the 40,000 unlabeled ones and get a predicted probability for each. Keep the predictions above a confidence threshold — say, 95% urgent or 95% not — and add those scans to the training set with their predicted label attached, called a pseudo-label. Retrain on the combined set. Repeat, each round pulling in whatever is now confident enough to clear the bar.

The threshold is the whole method

Set the threshold high and you accept few new pseudo-labels, but nearly all of them are correct — safe, slow progress. Set it low and you pull in far more of the archive, but a growing share of those pseudo-labels are wrong, and the model now trains on its own mistakes as if they were ground truth. There's no threshold that's simply "right"; it's a dial between how much of the archive you use and how much noise you accept into training.

Consistency regularisation, the other lever

Pseudo-labelling commits to a hard label the moment confidence clears the bar. Consistency regularisation never commits to one. Take an unlabeled scan, apply two different augmentations — a slight rotation, a brightness shift — and penalize the model when its two predictions disagree on the same underlying scan. No label is ever assigned; the model is just pushed toward giving the same answer regardless of how the input was perturbed. It's a softer signal than pseudo-labelling, and the two techniques are often combined rather than chosen between.

Confirmation bias: why a wrong pseudo-label is worse than a missing one

Accept a wrong pseudo-label and the next round's model trains partly on that mistake. If the mistake sits in a region the model was already unsure about, that region now looks more settled than it is, and the model's confidence there rises — which means the next wrong prediction in that same region is more likely to clear the threshold too. Each bad label makes the next one easier to accept. This compounding is why semi-supervised learning can end up worse than training on the 500 labeled scans alone: not a small loss, but a model actively convincing itself of something false, one confident-looking round at a time.

Show Me the Code

A tiny labeled set, a large unlabeled pool, and two confidence thresholds — one strict, one loose.

1import numpy as np2
3
4def fit(x: np.ndarray, y: np.ndarray, steps: int = 500) -> tuple[float, float]:5    """1D logistic regression: p(y=1|x) = sigmoid(w*x + b)."""6    w, b = 0.0, 0.07    for _ in range(steps):8        p = 1 / (1 + np.exp(-(w * x + b)))9        w -= 0.3 * float(np.mean((p - y) * x))10        b -= 0.3 * float(np.mean(p - y))11    return w, b12
13
14rng = np.random.default_rng(0)15x_lab = np.array([0.05, 0.15, 0.25, -0.9, -1.1, -1.0])  # 6 labeled scans16y_lab = np.array([1.0, 1.0, 1.0, 0.0, 0.0, 0.0])17pool = np.concatenate([rng.normal(-1, 1, 800), rng.normal(1, 1, 800)])  # 1,600 unlabeled18truth = np.concatenate([np.zeros(800), np.ones(800)])  # for checking pseudo-label errors only19
20w, b = fit(x_lab, y_lab)21p = 1 / (1 + np.exp(-(w * pool + b)))22for tau in (0.99, 0.6):23    keep = np.maximum(p, 1 - p) >= tau24    wrong = int(((p[keep] > 0.5) != truth[keep].astype(bool)).sum())25    print(f"threshold {tau}: accepts {int(keep.sum())} of {len(pool)}, {wrong} wrong")26# -> threshold 0.99: accepts 1082 of 1600, 86 wrong27# -> threshold 0.6: accepts 1548 of 1600, 249 wrong

The strict threshold accepts fewer pseudo-labels but keeps the error rate near 8%. Dropping the bar to 0.6 nearly doubles the accepted count and triples the error rate — a direct look at the dial the previous section described.

Watch Out For

Confirmation bias compounding across rounds

A pseudo-label accepted in round one that happens to be wrong doesn't just sit there — the model retrains on it, and if that mistake reinforces an existing blind spot, the next wrong prediction nearby clears the confidence threshold more easily than it should. Track pseudo-label accuracy against a small held-out labeled set every round, not just at the end. A quietly rising error rate in that check is the first sign the loop is teaching itself something false.

Confidence rubber-stamping an imbalanced pool

If 95% of the unlabeled archive is routine scans and 5% are urgent, a model with even mild bias toward the majority class will confidently label almost everything "routine" — and pseudo-labelling will happily agree with itself, entrenching the imbalance rather than correcting it. Check the class balance of accepted pseudo-labels each round against the labeled set's balance; a growing gap between them means the loop is amplifying a skew, not learning from data.

The Quick Version

Semi-supervised learning trains on a small labeled set and a much larger unlabeled pool, using the model's own confident predictions as extra training signal.
Pseudo-labelling commits to a hard label above a confidence threshold; consistency regularisation never commits, penalizing disagreement across augmented views instead.
The threshold trades accepted volume against pseudo-label error — there's no universally correct setting.
Confirmation bias compounds: a wrong pseudo-label makes the next wrong prediction in that region easier to accept, round after round.
It can end up worse than training on the labeled set alone if the loop isn't checked against held-out ground truth.