Textual Inversion

Textual inversion teaches a frozen diffusion model one new word, a learned embedding for your object, using three photos and zero weight updates.

Why Does This Exist?

Every personalization method before it moved millions of weights. Fine-tuning a UNet for one teapot risks the model, the storage, and the setup time, all for a concept expressible as one word the model never learned. The heavy machinery exists for hard cases; most users need a light word, not a rebuilt brain.

Textual inversion learns exactly one embedding vector, $768$ numbers, for a placeholder token like S*. Everything else freezes. Three photos and a few thousand steps later, an oil painting of S* renders your teapot in any style the base model knows. The full-model alternative is DreamBooth, which trades weight updates for stronger identity.

Think of It Like This

Adding one entry to a dictionary

A library owns every book ever printed but its dictionary lacks the word for your family's stew. You do not rewrite the library; you slip one new dictionary card into the drawer defining your stew. Every reader, every recipe, every menu in the building can use the word immediately because the meaning plugs into existing language.

The embedding is the dictionary card and the frozen model is the library. It fails when one card must carry a whole cuisine: complex multi-object concepts overflow a single vector, which is why multi-embedding variants exist.

How It Actually Works

Introduce placeholder string S* mapped to a learnable vector $v$ . Freeze the UNet, the VAE, and the text encoder body; only $v$ receives gradients. Training minimises the standard diffusion denoising loss over $3$ – $5$ concept photos with neutral prompts like a photo of S*:

v^* = \arg\min_v \; \mathbb{E} \lVert \epsilon - \epsilon_\theta(x_t, t, c(v)) \rVert^2

$c(v)$ is the conditioning built from the learned vector. Because the loss is the model's native objective, optimisation needs no new machinery, just $3{,}000$ – $5{,}000$ steps on the tiny photo set. At inference, S* composes freely: S* on the moon, a statue of S*, each rendered with the base model's full stylistic range.

Worked example

Vector $v$ holds $768$ numbers versus $860$ million frozen UNet weights: the learned fraction is under one-millionth. Initialise $v$ from the embedding of the word teapot; initial loss on your teapot photos sits $15$ percent above the model's average photo loss, and converges to within $2$ percent after $3{,}000$ steps. The file on disk is $3$ KB, emailable, swappable, and stackable with other learned words.

Code

Learnable-token training loop in miniature, showing what freezes and what moves.

1def train_step(v: list[float], grad: list[float], lr: float = 0.005) -> list[float]:2    """Update only the placeholder embedding; the model never moves."""3    return [a - lr * g for a, g in zip(v, grad)]  # UNet, VAE, text encoder frozen4
5
6v = [0.0] * 768  # placeholder embedding S*7print(len(v) * 8)  # -> 6144 bytes in float64, about 3 KB in float32

Watch Out For

Underfitting complex subjects with one vector

A single $768$ -number vector carries limited identity. Ornate subjects, patterned fabrics, multi-part objects reconstruct approximately: right category, wrong details. Users blame the photo count and add more images that cannot fix a capacity ceiling.

Match capacity to subject: one vector for simple objects, two or three learned tokens for detailed ones. If likeness plateaus while loss keeps falling, capacity, not data, is the wall.

Neutral prompts that smuggle in style

Train with a painting of S* and the embedding absorbs painterliness along with identity; later a photo of S* still renders painterly. The placeholder soaks up everything constant across training captions.

Keep training prompts neutral and varied in background: a photo of S*, a cropped photo of S*. Reserve style words for inference.

The Quick Version

Textual inversion learns one $768$ -number embedding for a placeholder like S*.
The UNet, VAE, and text encoder stay completely frozen throughout training.
Three photos and about $3{,}000$ steps close the loss gap from $15$ percent to $2$ percent.
The result is a $3$ KB file composing freely with any prompt the base model handles.
Single vectors underfit ornate subjects, and styled training captions leak style into identity.