Adapters for Vision Tuning

Slide small down-up bottlenecks into frozen layers and train only them. A few percent of weights adapts big vision.

Adapters add tiny bottlenecks inside frozen vision layers

Why Does This Exist?

Full copies per task bloat storage. Adapters keep one frozen backbone and store $3$ percent task weights as bottlenecks after attention and MLP. Fine-tuning becomes swapping small files.

This page covers vision adapters only. Prompt tokens are sibling pages.

The one idea you need first.

Adapter is down $W_d$ to $64$ , nonlinearity, up $W_u$ back, plus skip: $y = x + W_u(\sigma(W_d x))$ . Only $W_d, W_u$ learn. Place after attention and MLP per block.

Think of It Like This

Plug-in pedals on a shared guitar rig

A guitarist keeps one amp rig and clips pedals for rock versus jazz. Pedals shape tone without rewiring the amp, and each gig bag holds its own set.

Backbone is the rig, adapters are pedals, and skips keep clean tone underneath. Where it stops: pedals cannot fix a broken amp, so huge shifts still retune.

How It Actually Works

Freeze ViT $f$ . Insert adapters in each block. Train adapters plus head $h$ on task data. Bottleneck $64$ sets size; larger $256$ helps hard shifts. Store adapters per task and compose by swapping.

In plain words: narrow, bend, widen, then add back to the stream.

Worked example

ViT-B $86$ M: adapters $2.6$ M, about $3.0$ percent. Pets accuracy $94.2$ versus $94.6$ full, gap $0.4$ . Ten tasks store $26$ M versus $860$ M full copies, a $33$ x saving. The math favors adapters past three tasks.

Watch Out For

Adapters too narrow for large shifts

Rank $8$ starves sketch-to-photo jumps and gaps hit $5$ points. Tuning stalls while loss looks fine. Raise to $64$ or $256$ for far shifts before judging adapters.

The Quick Version

Adapters are down-up bottlenecks with skips in frozen layers.
Rank 64 with 3 percent weights suits most tasks.
Place after attention and MLP blocks.
Swap small files per task.
Widen rank for far domain shifts.