Prompt Tuning Soft Tokens

Learn twenty vectors that sit before frozen model inputs. Words stay fixed while the soft prefix steers answers.

Prompt tuning learns soft tokens before frozen image-text models

Why Does This Exist?

Hand prompts plateau and full tuning costs much. Twenty learned vectors steer frozen joint models per task with kilobytes stored. Zero-shot classification prompts become trained vectors here.

This page covers text-side soft tokens only. Vision tokens are the prior page.

The one idea you need first.

Soft prompt $P$ of $20$ vectors prepends to inputs $[P, X]$ into frozen $f$ . Only $P$ learns by task loss. Init from word embeddings beats random.

Think of It Like This

A custom stamp before a fixed form letter

A office letter stays fixed, but a custom stamp at top routes it to the right desk. Each department cuts its own stamp and shares the letter stock.

Frozen model is the letter, soft tokens are stamps, and training cuts the shape. Where it stops: stamps cannot rewrite paragraphs, so hard shifts need more.

How It Actually Works

Freeze VLM $f$ . Init $20$ vectors from class words. Prepend to text inputs, forward, and train $P$ on task loss for many steps. Sweep length $8$ to $32$ and learning rate high, since only $P$ moves.

In plain words: $20$ vectors, long schedule, and word init matter most.

Worked example

Pets $37$ set: hand prompt $85.1$ percent, tuned $20$ tokens $91.4$ , gain $6.3$ after $2000$ steps. Random init trails word init $2.1$ points at same steps. Length $20$ beats $8$ by $1.4$ points here.

Watch Out For

Short schedules judging soft prompts weak

Few hundred steps leave vectors raw and scores lag hand prompts. Work looks failed early. Train thousands of steps with a strong schedule before comparing to adapters or full tunes.

The Quick Version

Prompt tuning learns 20 vectors before frozen inputs.
No backbone weights move per task.
Word init beats random starts.
Long schedules are required.
Kilobytes per task versus megabytes.