Visual Prompt Tuning Tokens

Freeze the whole backbone and learn fifty input tokens plus a head. Tiny tuning steers big vision cheaply.

Visual prompt tuning prepends learned tokens to a frozen backbone

Why Does This Exist?

Full fine-tuning copies billions of weights per task. VPT learns $50$ tokens plus a head with the backbone frozen, nearing full scores on small sets. Fine-tuning cost falls to megabytes per task.

This page covers vision tokens only. Text soft prompts are the next page.

The one idea you need first.

Prompt tokens $P$ join patch tokens $X$ as $[P, X]$ into frozen ViT $f$ . Shallow adds $P$ once; deep adds per layer. Only $P$ plus head $h$ learn.

Think of It Like This

Colored tabs in a fixed reference book

A library book cannot be marked, but fifty colored tabs steer readers to key pages with margin notes on tabs only. Each course uses its own tab set on one shared book.

Backbone is the book, tokens are tabs, and heads are course quizzes. Where it stops: tabs cannot rewrite pages, so huge shifts still need tuning.

How It Actually Works

Init $50$ tokens random, prepend each layer for deep VPT, forward frozen $f$ , train head $h$ plus $P$ with task loss. Sweep token count $10$ to $100$ and pick by validation. Store $P$ plus $h$ per task.

In plain words: $50$ vectors steer attention, $f$ stays fixed, and $h$ reads the answer.

Worked example

Flowers $102$ set: full tune $99.1$ percent, VPT-deep $98.8$ with $0.5$ percent weights. Gap $0.3$ points for $200$ x fewer weights. Shallow trails at $97.2$ , so depth buys $1.6$ points on fine petals.

Watch Out For

Too few tokens for fine splits

Ten tokens starve bird-species tasks and accuracy sags $4$ points. Validation flatlines early. Raise toward $50$ to $100$ and check per-task curves before judging VPT in fine-tuning trials.

The Quick Version

VPT prepends learned tokens to frozen ViT inputs.
Deep variants add tokens per layer.
Fifty tokens plus a head near full tuning.
Half a percent of weights per task.
Sweep token counts per dataset.