Transfer Learning in Computer Vision

Transfer learning starts vision models from weights already trained on millions of images, then adapts them to a small task instead of learning edges from zero.

A frozen base holding about 25.6M weights feeds one new trainable head of 20,490 weights, so a small dataset trains thousands of values, not millions.

Why Does This Exist?

Training a strong CNN from random weights demands millions of labelled images and weeks of multi-GPU time: the network must discover edges, textures and object parts before it learns anything about the actual task. Most teams have neither the data nor the budget, yet their problem still needs vision. Researchers noticed the early layers of trained networks converge to the same edge and texture detectors regardless of the dataset, which raised the obvious question: why should every project re-learn what an edge looks like? Transfer learning reuses a pretrained vision model as the starting point and adapts it to the new task. Small clinics, farms and factories get strong accuracy from hundreds of images instead of millions.

This page covers the freeze-then-fine-tune procedure with a parameter count worked by hand. The model zoo itself lives on the pretrained-models page, and head-only training on the feature extraction page.

Think of It Like This

Hiring a trained chef for one recipe

Teaching a complete beginner your pastry recipe means first teaching heat, knives, dough and timing: years before the first pastry. Hiring an experienced chef skips all of that; one week on your specific recipe and they produce. The chef is the pretrained base, their general kitchen skill is the early-layer features, and your recipe is the new classification head.

The analogy stops at forgetting. A chef keeps old skills while learning yours, but a network fine-tuned carelessly overwrites its base weights, which is why the learning rate stays tiny once layers unfreeze.

How It Actually Works

Freeze, then fine-tune

Split parameters into base weights $\theta_{\text{base}}$ and head weights $\theta_{\text{head}}$ . Step one removes the original 1,000-way ImageNet head and attaches a randomly initialized head sized to the new class count. Step two freezes $\theta_{\text{base}}$ (no gradient updates) and trains only $\theta_{\text{head}}$ on the target data, forcing the new head to interpret proven features. Step three unfreezes the top blocks and trains everything with a learning rate around ten times smaller than normal, letting features adapt without destroying them. Fine-tuning schedules and layer choices decide how far step three goes.

Worked parameter count

Take a ResNet-50 base whose final features have 2,048 channels, adapted to 10 classes. The new head is one linear layer: $2{,}048 \times 10$ weights plus 10 biases, giving $20{,}480 + 10 = 20{,}490$ trainable values. The frozen base holds about 25.6M weights, so the frozen share is $20{,}490 / 25{,}600{,}000 \approx 0.0008$ , barely 0.08% of the network. That is the whole economy of the method: hundreds of training images fit 20,490 values comfortably, while 25.6M values from scratch would memorize them.

When the source helps

Transfer works when source and target share visual structure: natural photos to products, animals, or street scenes. ImageNet features carry edges, textures and part shapes that survive the jump. Unrelated domains such as microscopic cells or radar scans share little, and the head-only phase stalls at poor accuracy no matter how long it trains.

Code

1in_features, num_classes = 2048, 102head_params = in_features * num_classes + num_classes3print(head_params)4# -> 204905print(round(head_params / 25_600_000 * 100, 3))6# -> 0.08

Watch Out For

Negative transfer across unrelated domains

Pretrained features only help when the domains share visual structure. A model raised on natural photos transfers poorly to microscopic cells or radar scans, and forcing it can score worse than a small network trained from scratch. The symptom is head-only training stalled far below expectations. Check domain similarity first, and budget for deep fine-tuning or from-scratch training when the gap is wide.

Catastrophic forgetting from a hot learning rate

Unfreezing the base with a full-size learning rate rewrites the pretrained weights in a few batches, destroying the transferred knowledge and overfitting the small dataset. The symptom is training loss collapsing while validation collapses too. Unfreeze only the top blocks first, with a learning rate around ten times smaller than the head used.

The Quick Version

Transfer learning reuses a pretrained base and trains a new head for the target classes.
Freeze the base first, then fine-tune top blocks with a learning rate near one-tenth.
A 2,048-to-10 head trains 20,490 values, about 0.08% of a ResNet-50.
Shared visual structure between source and target decides how well it works.
Unrelated domains and hot learning rates cause negative transfer and forgetting.