IP-Adapter

IP-Adapter adds a separate image attention path beside text conditioning, so a reference photo steers generation without rewriting the text prompt.

Why Does This Exist?

Text prompts describe poorly. A paragraph cannot pin down a face, a logo, or a pattern closely enough for consistent regeneration, and stuffing image features into the text embedding corrupts the language the model relies on. Creators needed image prompts that behave like first-class citizens.

IP-Adapter adds a parallel cross-attention branch for image features beside the existing text branch. The base model freezes; small projection layers map CLIP image embeddings into keys and values the UNet already knows how to attend to. Text sets the scene, image sets the subject, and one weight balances them. The layout-control sibling is owned by ControlNet and spatial conditioning, which steers geometry rather than identity.

Think of It Like This

A casting director with a headshot

A director briefs a scene in words, then slides one headshot across the table: her, in this scene. The words control everything except the face; the photo controls the face. Turn the photo face-down and casting goes generic; staple it over the script and the scene descriptions stop mattering.

The adapter weight is how firmly the headshot is pinned. It fails when the headshot and the brief contradict, a spacesuit scene with a beach photo, and the blend picks an awkward middle neither side wanted.

How It Actually Works

Standard cross-attention computes $Z_t = \mathrm{Attn}(Q, K_t, V_t)$ from text keys and values. IP-Adapter adds a decoupled branch $Z_i = \mathrm{Attn}(Q, K_i, V_i)$ from projected image features, sharing the query $Q$ so both attentions read the same spatial positions:

Z = Z_t + \lambda \, Z_i, \qquad K_i = W_k f_{\text{img}}, \; V_i = W_v f_{\text{img}}

Only the small projection matrices $W_k, W_v$ train, about $22$ million parameters against a frozen billion-plus base. The scale $\lambda$ defaults near $0.6$ : higher values lock identity harder, lower values let text dominate. Image features come from a frozen CLIP image encoder, so no new vision backbone is needed.

Worked example

Queries at one spatial block attend text keys giving $Z_t = [0.5, -0.2]$ and image keys giving $Z_i = [0.3, 0.6]$ . At $\lambda = 0.6$ the output is $[0.5 + 0.18, -0.2 + 0.36] = [0.68, 0.16]$ , text-led with a firm identity pull. Push $\lambda$ to $1.5$ and the output becomes $[0.95, 0.70]$ , where the reference dominates and the prompt's scene barely registers. The usable band, roughly $0.4$ – $0.8$ , is where both signals stay audible.

Code

Decoupled attention blending, the operation behind the strength slider.

1def blend(z_text: list[float], z_img: list[float], lam: float = 0.6) -> list[float]:2    """Add the image branch to the text branch at strength lam."""3    return [round(a + lam * b, 2) for a, b in zip(z_text, z_img)]4
5
6zt = [0.5, -0.2]7zi = [0.3, 0.6]8print(blend(zt, zi))  # -> [0.68, 0.16] balanced9print(blend(zt, zi, lam=1.5))  # -> [0.95, 0.7] reference dominates

Watch Out For

Reference leakage into the background

Strong image branches copy more than the subject: the reference's background, lighting, and framing bleed into outputs even when the prompt asks elsewhere. Users blame the text encoder for ignored prompts while the image branch overrules it.

Lower $\lambda$ first, then crop references tight around the subject. Test with a prompt that contradicts the reference background to measure leakage directly.

Expecting fine detail the encoder never kept

CLIP image embeddings summarise semantics, not pixels. Tiny text, exact patterns, and precise counts in the reference wash out through the encoder before the adapter ever sees them.

Treat IP-Adapter as identity and style control, not pixel fax. For exact reproduction use inpainting or reference-latent methods instead.

The Quick Version

IP-Adapter adds a parallel image cross-attention branch beside frozen text attention.
Output blends as $Z_t + \lambda Z_i$ with only small projection layers trained.
Our block gives $[0.68, 0.16]$ at $\lambda = 0.6$ and $[0.95, 0.70]$ at $\lambda = 1.5$ .
The usable strength band sits roughly $0.4$ – $0.8$ for balanced control.
Tighten crops to fight background leakage, and never expect pixel-exact copying.