Domain Adaptation for Vision

A model trained on sunny streets stumbles in the rain. Domain adaptation reuses its labeled sunny images plus unlabeled rainy ones to close that gap.

Labeled sunny images score 0.91 while unlabeled rainy ones score 0.63, until a shared encoder aligns both into one feature space and recovers 0.84 on rain

Why Does This Exist?

Collecting labeled images is expensive, so teams train where labels already exist: sunny-day streets, synthetic renders, one hospital's scanner. Then they deploy where labels do not exist: night rain, real streets, the next hospital's scanner. Accuracy falls off the same cliff every time. In the running example on this page, a street classifier scores 0.91 on its sunny source domain and 0.63 on the rainy target domain, a gap of 0.28 with no change to the labels themselves.

The naive fix is labeling the new domain, which costs exactly what the team was avoiding. The lazy fix is hoping the gap is small, which it rarely is. Domain adaptation is the middle path: keep the labeled source images, add unlabeled target images, and force the model to represent both in one shared feature space where a single classifier works for both.

This page teaches the adaptation methods that read target data. Its companion, domain generalization and adaptation, is the map of the whole territory, including the branch where no target data exists at all. The cousin that fine-tunes on labeled target images is transfer learning.

The one idea you need first

A domain is the joint distribution images and labels are drawn from: the camera, the lighting, the rendering engine, the hospital. Domain shift means the source and target domains differ while the task stays the same. A classifier is a feature extractor $f$ that maps an image $x$ to a vector $f(x)$ , plus a head that maps that vector to a label. Adaptation keeps the head and repairs $f$ so source and target vectors land in the same region.

Think of It Like This

A driver retraining for the other side of the road

You learned to drive on the right side of the road. Moving to a country that drives on the left, you do not relearn steering, braking, or mirrors. You keep every skill and adjust exactly one thing: which side of the road your habits assume. A few supervised hours in the new traffic, and the old skills transfer.

The source domain is right-side driving, the target domain is left-side traffic, and the shared encoder is everything you kept: steering, pedals, mirrors. The domain critic is the instructor in the passenger seat who notices whenever you drift to the familiar side. It stops holding where labels change meaning rather than appearance: if the new country also flips every road sign, no amount of lane adjustment recovers the signs.

How It Actually Works

Three parts train together. A shared encoder $f$ maps both source and target images to vectors. A task head $h$ predicts labels from those vectors using the labeled source images, with task loss $L_{task}$ . A domain critic $d$ looks at a vector and guesses source or target, with domain loss $L_{domain}$ . The encoder is trained to fool the critic while the critic is trained to catch it, through a gradient reversal layer that flips the sign of $L_{domain}$ on its way into $f$ :

L = L_{task} + \lambda L_{domain}

Here $\lambda$ (lambda) sets the strength of the alignment pressure, and the reversal means $f$ climbs $L_{domain}$ while $d$ descends it. At equilibrium the critic guesses at chance, which means the two domains overlap in feature space, and the head trained on source labels works on target vectors too. A quieter alternative skips the critic and directly shrinks a distance between the two feature clouds, such as the distance between their means.

Worked example

Take a street classifier evaluated on 100 sunny images and 100 rainy images. It gets 91 sunny ones right and 63 rainy ones right:

\text{source} = 91 / 100 = 0.91, \qquad \text{target before} = 63 / 100 = 0.63

The shift costs $0.91 - 0.63 = 0.28$ . Now measure the feature gap. Suppose source vectors average $(0.20, 0.30)$ and target vectors average $(0.60, 0.50)$ . The centroid distance is:

\sqrt{(0.60 - 0.20)^2 + (0.50 - 0.30)^2} = \sqrt{0.16 + 0.04} = \sqrt{0.20} \approx 0.45

After adversarial training with $\lambda = 0.5$ , the target centroid moves to $(0.24, 0.32)$ :

\sqrt{(0.24 - 0.20)^2 + (0.32 - 0.30)^2} = \sqrt{0.0016 + 0.0004} = \sqrt{0.002} \approx 0.04

The gap shrinks from 0.45 to 0.04, and the rainy accuracy climbs to 84 out of 100, or 0.84. That recovers $0.84 - 0.63 = 0.21$ of the 0.28 lost, which is the typical outcome: most of the gap back, not all of it.

Code

The snippet below computes the centroid gap before and after alignment for the worked example. It is the check to run on real features: if this distance does not fall, the critic is losing and nothing else on the page can work.

1import math2
3
4def centroid_gap(src: tuple[float, float], tgt: tuple[float, float]) -> float:5    dx = tgt[0] - src[0]6    dy = tgt[1] - src[1]7    return math.sqrt(dx * dx + dy * dy)8
9
10src = (0.20, 0.30)11print(round(centroid_gap(src, (0.60, 0.50)), 2))12# -> 0.4513print(round(centroid_gap(src, (0.24, 0.32)), 2))14# -> 0.0415print(round(0.91 - 0.63, 2), round(0.84 - 0.63, 2))16# -> 0.28 0.21

The first number is the 0.45 gap the diagram shows as two separated clouds. The second is the 0.04 gap after alignment, the two clouds sitting on each other. The third line is the accounting: 0.28 lost to shift, 0.21 recovered.

Watch Out For

Forcing alignment between domains that share nothing

When the source and target are too far apart, say synthetic game frames adapted to X-ray scans, a strong $\lambda$ erases the class structure along with the domain gap. The symptom is distinctive: the domain critic falls to chance, exactly as hoped, while the source accuracy collapses too. Both heads sit on mush.

Fix it by ramping $\lambda$ from 0 upward instead of starting high, and by watching source accuracy as the guardrail. If source accuracy drops more than a point or two while the critic is being fooled, the domains share too little and you need labeled target data, which is transfer learning, not adaptation.

Aligning so hard the classes collapse

A critic that only sees domain labels cannot tell class separation from domain separation, so an over-weighted $L_{domain}$ merges different classes that happen to sit near each other. The symptom is per-class accuracy that equalizes downward: every class scores mediocre instead of some scoring well. The overall number can even look stable while the confusion matrix fills off-diagonal.

Fix it by keeping $\lambda$ modest and checking per-class accuracy on a labeled target validation split, never just the mean. Class-conditional alignment, matching each class cloud separately, is the upgrade when plain alignment blurs boundaries.

The Quick Version

Domain adaptation repairs a model for a new domain using labeled source images plus unlabeled target images.
A shared encoder feeds a task head and a domain critic trained against each other through gradient reversal.
The worked example loses 0.28 to shift, shrinks the feature gap from 0.45 to 0.04, and recovers 0.21 to reach 0.84.
Ramping the alignment strength from zero protects class structure that a strong critic would erase.
Adaptation needs target images at training time; without any, the answer is generalization, not adaptation.