Linear Probing Frozen Nets

Freeze the backbone and train one linear layer on top. Good features score high with nothing else to hide behind.

Linear probing trains one layer on frozen backbone features

Why Does This Exist?

Fine-tuning mixes feature quality with adaptation skill. Probing isolates features: freeze $f$ , train $W, b$ with $y = W f(x) + b$ , and compare backs fairly. Gaps to full tunes in fine-tuning show what adaptation adds.

This page covers probing only. Tuning moves are sibling pages.

The one idea you need first.

Probe is logistic regression on frozen $f(x)$ . No gradient enters $f$ . Standardize features or tune weight decay for fair numbers.

Think of It Like This

Tasting stock before seasoning a soup

A chef tastes plain stock to judge bones and time, then seasons for service. Good stock needs little salt; weak stock needs rescue. Tasting plain separates base from fix.

Frozen features are stock, linear heads are salt, and fine-tuning is full seasoning. Where it stops: soups finish seasoned, while papers report both numbers.

How It Actually Works

Freeze $f$ . Extract train plus test features once. Train linear $h$ with cross entropy and tuned decay. Report top- $1$ on image classification sets. Compare probe versus full tune per backbone.

In plain words: one layer, frozen eyes, and decay as the dial.

Worked example

Backbone A probe $76.3$ percent, full $84.1$ , gap $7.8$ . Backbone B probe $79.5$ , full $83.2$ , gap $3.7$ . B features transfer better despite lower full peak. The gap, not the peak, ranks frozen quality.

Watch Out For

Weak decay hiding good features

Default decay underfits probes and ranks flip. Good backs look flat. Sweep decay over $0.01$ to $10$ on validation and report the best per backbone.

The Quick Version

Linear probing trains one layer on frozen features.
It grades representation quality alone.
Probe versus tune gaps show adaptation value.
Tune decay per backbone for fairness.
Cheap features-once, heads-many workflow.