Skip to content
AI360XpertBeta

Image Editing and Inpainting

You mask the part of a photo you want changed and keep the rest. The model redraws only inside the mask, so the sofa turns red while your cat stays exactly the same.

Inpainting blends one denoising step by keeping noisy original pixels where the mask is 0 and taking the network guess where the mask is 1, giving 1.02 kept and 0.3 painted
Inpainting blends one denoising step by keeping noisy original pixels where the mask is 0 and taking the network guess where the mask is 1, giving 1.02 kept and 0.3 painted

Why Does This Exist?

You generate the perfect product shot, except the mug is blue and the client asked for red. You fix the prompt and reroll. Now the mug is red, but it sits in a different kitchen, under different light, next to a different cat. Text-to-image gives you no way to point at one thing and say "change that, nothing else."

Inpainting is that pointing finger. You paint a mask over the pixels you want replaced, keep your prompt, and the model redraws only inside the mask. The rest of the photo is restored pixel for pixel, so the red mug lands in the original kitchen with the original cat.

This page covers the masked-blend trick that makes it work, which runs on top of ordinary sampling. If denoising loops are new to you, start with diffusion models, then the schedule math in forward and reverse diffusion. Pinning down layout with edges or poses instead of masks is ControlNet, and rewriting a photo from a written order with no mask at all is instruction-driven editing.

Think of It Like This

The restorer and the taped-off painting

You own a landscape painting with a water stain across the sky. You hire a restorer, and the first thing they do is tape over every good inch. The tape is the mask: paint under tape cannot be touched, and only the stain gets repainted. As they work, they keep glancing at the real clouds at the stain's edge, mixing each brushstroke to continue the surrounding color and light.

Point for point: taped canvas is mask value 0, pixels the model must restore. Bare stain is mask value 1, pixels the model may invent. Each glance at the neighboring clouds is one denoising step conditioned on the true surroundings.

Where it stops: your restorer can see the true painting under the varnish and match its pigments. The model has no original inside the mask, only your prompt and the edges around the hole. So it can paint the wrong object with beautifully matched lighting, and no amount of blending will fix a bad prompt.

How It Actually Works

A standard diffusion sampler already walks a chain from pure noise xTx_T down to a clean image x0x_0, where the subscript tt counts how much noise remains. Inpainting hijacks that walk. At every step it lets the network guess the whole frame, then throws away the guess wherever the mask says keep and pastes the true photo back in, with matching noise added.

The three inputs

You supply the original image x0x_0 (the clean photo, values here scaled so 1.0 is full brightness), a binary mask MM (a per-pixel switch: 1 means repaint this pixel, 0 means keep it), and a text prompt cc (a conditioning vector that steers the guess, describing what should appear inside the hole).

The one-line trick, repeated every step

At step tt, two versions of the slightly cleaner frame xt−1x_{t-1} exist side by side. The network predicts one: x^t−1\hat{x}_{t-1}, its usual denoised guess for the whole frame given the current noisy frame xtx_t, the step tt, and your prompt cc. The photo supplies the other: q(xt−1∣x0)q(x_{t-1} \mid x_0), which is the original image with exactly the noise level of step t−1t-1 added back by fixed math, so it sits at the right noise level for this point in the chain.

That noisy-photo value follows the same forward recipe the model trained on. Write αˉt−1\bar{\alpha}_{t-1} for the cumulative signal kept through step t−1t-1 (a number between 0 and 1, the fraction of the original still alive), and ϵ\epsilon for fresh Gaussian noise (independent static with mean zero, resampled every step). Then:

xt−1known=αˉt−1 x0+1−αˉt−1 ϵx_{t-1}^{\text{known}} = \sqrt{\bar{\alpha}_{t-1}}\, x_0 + \sqrt{1 - \bar{\alpha}_{t-1}}\, \epsilon

In plain words: shrink the photo toward zero, then top it up with static. The blend that forms the actual next frame is one line per pixel, where ⊙\odot means multiply pixel by pixel:

xt−1=M⊙x^t−1+(1−M)⊙xt−1knownx_{t-1} = M \odot \hat{x}_{t-1} + (1 - M) \odot x_{t-1}^{\text{known}}

Where the mask is 1, the network guess survives. Where it is 0, the guess is discarded and the noisy photo returns. Nothing is learned here and no weights change. That single line, applied from t=Tt = T down to t=1t = 1, is the whole method.

Worked example: two pixels, one step

Take αˉt−1=0.64\bar{\alpha}_{t-1} = 0.64. Then 0.64=0.8\sqrt{0.64} = 0.8 of the photo survives and 1−0.64=0.36=0.6\sqrt{1 - 0.64} = \sqrt{0.36} = 0.6 weights the fresh noise. Pixel B is kept (M=0M = 0) with photo value x0=0.9x_0 = 0.9 and drawn noise ϵ=0.5\epsilon = 0.5:

0.8×0.9+0.6×0.5=0.72+0.30=1.020.8 \times 0.9 + 0.6 \times 0.5 = 0.72 + 0.30 = 1.02

So the known branch for B is 1.02. (Above 1.0 is fine mid-chain; later steps pull values back into range.) The network also guessed B as 0.7, but M=0M = 0 discards it:

B:0×0.7+1×1.02=1.02, keptB:\quad 0 \times 0.7 + 1 \times 1.02 = 1.02,\ \text{kept}

Pixel A is masked (M=1M = 1). The network guessed 0.3 for it, and the known branch is thrown away there:

A:1×0.3+0×1.02=0.3, paintedA:\quad 1 \times 0.3 + 0 \times 1.02 = 0.3,\ \text{painted}

One step, two fates: A takes the model's 0.3, B keeps the photo's 1.02. The diagram above draws exactly these numbers.

Why the edges line up

The override happens at every step, not once at the end. So while the hole is still noisy, the guess inside it is denoised next to the true surroundings at the same noise level, and each pass pulls the new texture a little closer to the old light and geometry. Paste once at the end and you get a sticker. Blend at every step and the boundary has dozens of chances to agree with itself.

Most real editors run this loop in compressed space rather than on raw pixels, which is latent diffusion: the mask is shrunk to latent size and the same blend runs there. Same line, smaller grid.

The fine-tuned upgrade

The trick above works with any pretrained model, but the network never asked for a hole. It guesses the kept pixels too, gets overruled, and the seam can show. Dedicated inpainting models fix that in training: they feed the network extra input channels carrying the mask and the masked photo, and train on randomly cut holes, so the model learns to read edges instead of fighting them. The price is a full fine-tuning run. The gain is cleaner borders on hard holes like straight architecture or text.

Know the trade before you reach for the tool. Inpainting buys local control and costs global coherence: the bigger the hole, the less context surrounds it, and a mask covering most of the frame is just slow text-to-image with extra steps. When the whole photo must change mood or style, skip the mask and use instruction editing. When layout must hold exact positions, use spatial conditioning. The method assumes enough true surroundings to steer by. Starve it of context and it fills the hole with plausible nonsense wearing your lighting.

Code

The point of this snippet is the blend line itself. Two pixels, one step, the same numbers as the worked example: pixel A masked, pixel B kept.

import math

def noisy_known(x0: float, eps: float, alpha_bar: float) -> float:    """Original photo pushed to the noise level of this step."""    return math.sqrt(alpha_bar) * x0 + math.sqrt(1.0 - alpha_bar) * eps

def blend(pred: float, known: float, mask: int) -> float:    """Mask 1 takes the network guess, mask 0 restores the photo."""    return mask * pred + (1 - mask) * known

alpha_bar = 0.64  # weights 0.8 photo, 0.6 noiseknown_b = noisy_known(0.9, 0.5, alpha_bar)print(round(known_b, 2))                # -> 1.02print(round(blend(0.3, known_b, 1), 2))  # -> 0.3print(round(blend(0.7, known_b, 0), 2))  # -> 1.02

Pixel A keeps the 0.3 guess, pixel B keeps the 1.02 noisy photo, and the 0.7 guess for B is discarded exactly as the math says.

Watch Out For

Outpainting paints a second copy of your subject

Outpainting is inpainting turned outward: you extend the canvas and mask the new border. Prompt it with the original caption, say a man in a field, and you will often get a second man standing in the new grass. The model cannot tell the prompt described the old pixels, so it obeys it again on the new ones.

The symptom is duplication at the seam: repeated people, repeated houses, repeated text. The fix is to describe only what belongs in the new area, like empty grassy field under the same sky, and to grow the canvas in narrow strips so each extension has the maximum true context to lean on.

A hard mask edge prints a visible seam

A mask cut with a razor edge on a flat region, like sky or a white wall, often leaves a faint outline even when the fill looks right. The blend changes value abruptly across one pixel, and smooth gradients forgive nothing.

The symptom is a ghost contour tracing your brushwork. The fix is to feather the mask, softening its edge over a few pixels so MM ramps from 1 to 0 instead of jumping, and to keep the mask snug around the target. A tight, soft hole beats a big, sharp one every time.

The Quick Version

  • Inpainting redraws only inside a mask while the rest of the photo is restored untouched, so one detail can change without rerolling the scene.
  • Each reverse step blends per pixel as MM times the network guess plus (1−M)(1 - M) times the noisy original, with 1 meaning repaint and 0 meaning keep.
  • The kept branch re-adds matching noise via αˉ\sqrt{\bar{\alpha}} times the photo plus 1−αˉ\sqrt{1 - \bar{\alpha}} times fresh noise, so old and new pixels sit at the same noise level.
  • Because the paste repeats at every step, the new region is denoised next to the true surroundings dozens of times, which is what lines up the edges.
  • Big holes starve the method of context, duplicated subjects mean your prompt described the old pixels, and ghost outlines mean your mask edge needs feathering.