Core ML

Transfer Learning

A pretrained model already knows edges, textures and syntax. So freeze all of that, train a new head, and unfreeze deeper only when your labels support it.

Feature extraction and fine-tuning are the same network with the freeze boundary in a different place, so choosing where to draw that line is the whole decision

Why Does This Exist?

Three thousand contract clauses, each tagged by a paralegal as containing an indemnity or not. That's your entire labelled set, and you want a classifier that catches the ones a human missed.

Train a 14-million-weight text encoder from scratch on 3,000 examples and you know how it ends. By the third pass the training set is memorised, validation sits near the majority-class baseline, and every fix moves the number by noise. That's overfitting in its purest form: about 4,700 free weights per labelled clause.

A network trained on a very large corpus doesn't spread task knowledge evenly across its depth. Early layers settle into general structure — edges and textures in vision, subword shapes and local syntax in text — because that structure is what every example shares. Later layers combine those into whatever the pretraining objective wanted. The early part isn't about that task. It's about the data.

So don't start from random. Start from those weights and train only what you have data for. One decision is left: where's the line between the layers you keep and the layers you'll move?

Think of It Like This

Altering a suit off the rack

A rack suit was cut for someone else's body, from a pattern that already knows what a shoulder is and how a lapel rolls. None of that knowledge is about the previous customer. It's about bodies.

So a tailor takes it in at the waist, shortens the sleeves, and hands it back. Small changes, near the surface, cheap and reversible. Nobody unpicks the shoulder seam: it's the most expensively right part, and the hardest to put back.

Sometimes the shoulder genuinely has to move. Fine — but that needs fabric in the seam allowance. Try it on a suit with none and you don't get a better fit, you get a ruined suit. Labelled examples are the fabric.

How It Actually Works

Two strategies, one boundary

Feature extraction freezes every pretrained weight and trains only a new output layer. The encoder becomes a fixed function from clause to vector, and that vector feeds a classifier with a few thousand weights. Fast, since the frozen part stores no gradients, and it can barely overfit because there's almost nothing to fit. Often called a linear probe.

Fine-tuning unfreezes some or all of the pretrained weights and keeps training them at a much lower rate. Stronger when it works, because the features reshape around your task instead of staying whatever pretraining left behind. It needs more labelled data and more care, in that order.

The decision rule, on two axes

Two things decide it: how much labelled data you have, and how far your domain sits from the pretraining one.

Little data, near domain — English contract text against a model pretrained on general English. Feature extraction, and be pleased about it. The features already fit; you need the readout.

Lots of data, far domain — tens of thousands of labels for something the corpus never contained, like radar returns. Unfreeze broadly: the early layers learned structure that's wrong for your inputs, and you have the data to replace it.

Little data, far domain. The hard corner, and no rule saves you. Unfreeze the last two or three blocks, hold everything below fixed, and lean on augmentation to stretch what you have.

The mechanics that decide whether it works

The rate matters more than the boundary. Fine-tuning wants a learning rate one to two orders of magnitude below pretraining's, because you're nudging a good solution rather than searching for one. To be careful, decay it by depth:

\eta_{\ell} = \eta_{\text{top}} \cdot \gamma^{\,L - \ell}

$\ell$ is the layer index counting up from the input, $L$ the layer count, $\eta_{\text{top}}$ the rate at the topmost block, and $\gamma$ sits around 0.8 to 0.95. Layers nearer the input move least, which matches where the general structure lives. Add a warmup so the randomly initialised head doesn't fire a wild first gradient through weights you care about. And on a small set, freeze the normalisation statistics rather than letting a few hundred examples redefine them.

With a modern large pretrained model, head-only often lands within a point or two of full fine-tuning. Measure it first, then decide whether unfreezing earns its complexity.

Show Me the Code

Count the weights you made trainable against the labels you own.

1def budget(blocks: list[int], head: int, unfrozen: int, labelled: int) -> tuple[int, float]:2    """unfrozen = how many top blocks keep their gradients. Zero is feature extraction."""3    live = head + sum(blocks[len(blocks) - unfrozen:])4    return live, live / labelled            # trainable weights per labelled clause5
6
7blocks: list[int] = [1_180_000] * 12        # the pretrained encoder, twelve blocks8head: int = 1_538                           # one linear layer: indemnity clause or not9for unfrozen in (0, 2, 12):10    live, per_example = budget(blocks, head, unfrozen, labelled=3_000)11    print(f"top {unfrozen:2d} blocks live -> {live:>10,} weights, {per_example:>8,.1f} per example")12# -> top  0 blocks live ->      1,538 weights,      0.5 per example13# -> top  2 blocks live ->  2,361,538 weights,    787.2 per example14# -> top 12 blocks live -> 14,161,538 weights,  4,720.5 per example

Half a weight per clause against 4,720. Not a hard threshold — pretrained initialisation and a low rate both buy slack a from-scratch model wouldn't get — but it's the number to look at before assuming unfreezing everything wins.

Watch Out For

Fine-tuning at the pretraining learning rate

You copy the optimiser config out of the pretraining script and start training. Within a few hundred steps, gradients from 3,000 clauses have overwritten the pretrained weights, and what's left is a randomly initialised network with an odd starting point. The tell: validation opens below your frozen-feature baseline and never climbs back.

Run the frozen baseline first and treat it as the floor to beat. Then fine-tune at roughly a hundredth of the pretraining rate, warm up over the first few hundred steps, and watch the first evaluation rather than the last. If step 200 is already worse than the probe, stop. No amount of patience fixes a rate.

Unfreezing everything on a few hundred examples

The frozen probe gets you to 0.81. You unfreeze the whole encoder expecting more, and training loss drops to nearly zero within an epoch while validation goes backwards. That's roughly four thousand free weights per labelled example, used exactly as you'd expect: one configuration per training clause, nothing worth generalising.

The instinct is to train longer or add dropout. Neither helps — the problem is capacity you handed the optimiser. Freeze more. Walk the boundary down one block at a time, keep the frozen baseline in every comparison, and stop at the first depth where validation stops improving. When you're out of data rather than depth, the answers are more labels, stronger augmentation, or accepting the probe.

The Quick Version

Pretrained networks divide by depth: early layers hold general structure, later layers task-specific combinations.
Feature extraction freezes everything and trains a head. Cheap, hard to overfit, always worth measuring.
Fine-tuning unfreezes some depth at a far lower rate. Stronger, hungrier, easier to break.
Choose on two axes: labelled data and domain distance. Little data, near domain, freeze; lots of data, far domain, unfreeze broadly.
Little data, far domain is the hard corner. Unfreeze the top few blocks and augment hard.
Mechanics that matter: a much lower rate, optional layer-wise decay, a warmup, frozen normalisation statistics.