Self-Training and Pseudo-Labeling

Self-training teaches a new model from the old model's most confident predictions, growing labeled data without growing the labeling bill.

One thousand labels train a 0.74 model whose 5000 predictions keep 1800 confident ones for retraining to 0.79

Why Does This Exist?

Labeled images run out fast while unlabeled ones pile up. Training only on the labeled slice wastes the pile, but trusting every prediction poisons training with confident mistakes.

Self-training takes the middle path. Train a teacher on labeled data, predict the unlabeled pool, keep the confident slice as pseudo-labels, and retrain a student on the union. This page covers that outer loop specifically. The umbrella mechanisms live in semi-supervised learning, the threshold specialist is FixMatch, and the averaging teacher is mean teacher.

Think of It Like This

A senior who grades practice papers

A senior solves a thousand problems with answer keys, then attempts five thousand more without keys and marks only the solutions they feel sure about. A junior studies the solved thousand plus the confident attempts together.

Confidence is the marking, and retraining rounds are school terms. The analogy stops at calibration: seniors know their limits better than neural nets, whose confidence needs its own checking.

How It Actually Works

Round one trains on labeled images only. Round two runs the teacher over unlabeled images and keeps predictions above $\tau = 0.9$ or the top- $k$ per class to protect the tail. Round three retrains from scratch or fine-tunes on labeled plus pseudo-labeled data, sometimes with stronger augmentation on the pseudo slice. Repeat until gains flatten, refreshing pseudo-labels each round so fixed mistakes do not fossilise.

Worked example

Start: $1{,}000$ labeled street images, model at $0.74$ accuracy. Teacher predicts $5{,}000$ unlabeled frames; $1{,}800$ clear $0.9$ confidence with roughly $0.88$ precision on a probe, adding about $1{,}584$ correct and $216$ wrong labels. Retrain on $2{,}800$ mixed images reaches $0.79$ . The $216$ wrong ones are the drift tax, paid for $1{,}584$ free teachers.

Code

Confident-slice selection with class balance guard.

1def select_pseudo(max_probs: list[float], tau: float = 0.9) -> tuple[int, float]:2    """Count and share of predictions clearing the confidence gate."""3    kept = sum(1 for p in max_probs if p >= tau)  # hard gate per image4    return kept, kept / len(max_probs)5
6
7print(select_pseudo([0.95] * 1800 + [0.6] * 3200))  # -> (1800, 0.36)

Watch Out For

Retraining on the same mistakes each round

Freezing pseudo-labels once and retraining for many rounds bakes early errors in deeper each time. The symptom is accuracy rising one round then sliding as wrong labels compound.

Refresh predictions every round and cap pseudo weight below labeled weight. Track probe precision of the kept slice, not just its size.

The Quick Version

Self-training loops teacher prediction, confident selection, and student retraining.
Gate at $0.9$ or top- $k$ per class keeps the tail represented.
Worked numbers: $1{,}800$ of $5{,}000$ kept at $0.36$ share, $0.74$ to $0.79$ accuracy.
About $216$ wrong labels ride along as the drift tax.
Refresh pseudo-labels each round and weigh them below true labels.