Image Editing

Instruction-driven editing rewrites photos from plain commands like make it sunset, changing pixels through language instead of masks and brushes.

Instruction editing blends photo conditioning with command conditioning to rewrite images from words

Why Does This Exist?

Masks are tedious and imprecise for global edits. Asking for sunset over a lake should not require painting the sky pixel by pixel, yet inpainting demands exactly that. Language already describes the change; the editor should accept the sentence.

Instruction editing trains on triplets, original photo, written command, edited result, generated at scale from paired captioning plus image models. At inference the photo and the command condition the denoiser jointly, and two guidance strengths balance obedience to each. The mask-based foundation is owned by image editing and inpainting, which teaches the clamping machinery this page assumes.

Think of It Like This

A retoucher who takes dictation

A photographer dictates to a retoucher: same portrait, golden-hour light. The retoucher holds the original under one hand and the instruction in mind, changing only what the sentence requires. No masks, no selections, just understanding of both inputs at once.

The two guidance strengths are how loudly each input speaks. The sitting fails on contradictory dictation, change nothing but everything, and the retoucher must pick a middle the photographer never wanted.

How It Actually Works

The denoiser sees the noisy latent plus the encoded original photo concatenated as extra channels, and the command text through cross-attention. Sampling uses dual classifier-free guidance with separate scales for image and text:

\hat{\epsilon} = \epsilon_\varnothing + s_I (\epsilon_I - \epsilon_\varnothing) + s_T (\epsilon_T - \epsilon_\varnothing)

$\epsilon_\varnothing$ is unconditional, $\epsilon_I$ photo-conditioned, $\epsilon_T$ fully conditioned. Typical values $s_I = 1.5$ holding the photo and $s_T = 7.5$ pushing the command. Training triplets come from prompting a language model for edit commands plus a text-to-image model for before-after pairs, filtered for consistency.

Worked example

Daylight lake, command make it sunset. At $s_T = 7.5$ skies burn orange and water mirrors it while the dock's geometry holds under $s_I = 1.5$ . Drop $s_I$ to $1.0$ and the dock drifts into a generic pier; raise $s_T$ past $10$ and the scene oversaturates into kitsch. The two-knob balance is the entire craft of the method.

Code

Dual guidance blending, the two-knob update inside each step.

1import numpy as np2
3
4def dual_guidance(e_uncond: float, e_img: float, e_full: float, s_i: float = 1.5, s_t: float = 7.5) -> float:5    """Combine photo obedience with command obedience."""6    return e_uncond + s_i * (e_img - e_uncond) + s_t * (e_full - e_uncond)7
8
9print(round(dual_guidance(0.1, 0.2, 0.4), 3))  # -> 2.5 strong command pull

Watch Out For

Identity drift on faces at high text strength

Commands like make him smile at $s_T = 7.5$ reshape identity along with expression: jawlines shift, moles vanish. Face edits need lower text strength plus identity checks, because likeness lives in details the command never mentions.

Pin faces with $s_T$ near $4$ and verify with face similarity, not eyeballing. What reads as a good smile may be a different person.

Training-triplet artefacts inherited wholesale

Synthetic triplets carry their generator's biases: same skies, same skin smoothing. The editor learns the triplet factory's house style on top of real edits.

Filter triplets for diversity and mix real edit pairs where available. Audit outputs for repeated stylistic tics across unrelated commands.

The Quick Version

Instruction editing rewrites photos from commands, no masks required.
The denoiser conditions on the photo plus the command with dual guidance scales.
Typical balance is $s_I = 1.5$ for the photo and $s_T = 7.5$ for the command.
Our step pulls $0.1 \to 2.5$ under strong command pressure.
Faces drift at high text strength, and synthetic triplets inject house styles.