ControlNet and Spatial Conditioning
A text prompt says what to draw but not where. ControlNet adds a spatial map, like edges or depth, that pins down the layout while the frozen image model keeps doing the drawing.
Why Does This Exist?
Text prompts are vague about space. "A cottage on a hill" never says where the roofline runs, how deep the porch sits, or which way a dancer faces. You can reroll the seed all afternoon and still not land the exact pose an animator needs or the exact footprint an architect drew.
ControlNet fixes the layout without retraining the image model. You hand the sampler a spatial map, an edge sketch, a depth field, or a stick figure pose, and the output has to respect it. The prompt still picks content and style. The map picks positions.
The one idea you need first: modern image models denoise a compressed latent a little at a time, and each block of that denoiser shapes the picture. Latent diffusion covers that loop, so read it first if the denoising step is new to you. This page adds the side path that steers it. Strength control without layout lives in classifier-free guidance, and editing finished pixels lives in image editing and inpainting.
Think of It Like This
A stencil taped over the painter's canvas
You hire a fast airbrush artist who paints anything you describe, but never twice in the same layout. So you cut a stencil from cardboard, the outline of the house you want, tape it over the canvas, and say: paint a cottage at sunset, but only inside these lines. The artist still picks color, texture, and light. The stencil picks position.
The artist is the frozen diffusion model. The stencil is your edge or depth map. The hand that holds the stencil, and learns exactly how hard to press, is the trainable copy.
Where it stops: a real stencil never learns. It blocks paint the same way on day one and day one hundred. ControlNet's copy starts that stiff, then learns when to press harder and when to let the painter improvise, which is why the conditioning scale matters.
How It Actually Works
Conditioning means extra input beyond the prompt. Text conditioning picks content. Spatial conditioning picks layout, pixel by pixel, because the map lines up with the image grid. ControlNet is the adapter that lets a pretrained denoiser accept that second input without forgetting what it already knows.
The frozen base and the trainable copy
Name the pieces. Let be the noisy latent entering one denoiser block, and let be the spatial map, resized to the same grid as . Let be the pretrained block, with its weights locked and gradients off. Let be a trainable copy of that block's encoder path, with weights that update. Let be a 1x1 convolution joining the two. Let be the conditioning scale, a number you set at sample time. Let be the block output.
Training locks and feeds into . The copy's features pass through and get added back onto the frozen path:
In plain words: the frozen path draws as before, and the copy contributes a correction shaped by the map. At sample time you'll set . At you get the base model. At you get full guidance.
Zero convolutions start silent
A zero convolution is an ordinary 1x1 convolution with an unusual start: its weight and bias are both set to . So on step one maps everything to zero and . The model behaves exactly like the base model, which protects all that pretrained quality.
Gradients still flow into , , and . Training slowly opens the tap, and the copy has to earn its influence instead of scrambling the base model with random features on day one. That slow open is the whole trick.
Worked example: 3.0 plus 0.8 makes 3.8
A toy fixture with one feature channel, so you can check the arithmetic. Say the frozen block outputs and the copy outputs for one edge patch.
At init, : , so . Text alone decides, and the picture matches the base model.
Late in training, with bias still : , so with , . The layout visibly pulls the feature. Turn the knob to and . Same map, gentler hand. The diagram draws exactly these numbers.
One copy learns one kind of map, so you pick the map for the job:
- Canny edges hold outlines and graphic structure.
- Depth holds volume and room layout, with near objects bright and far ones dark.
- Pose holds a body's joint positions from a stick figure.
- Segmentation holds region labels, such as sky versus building versus grass.
The trade is direct. More conditioning buys layout fidelity and costs variety: push up and every sample obeys the map but starts to look traced. When you want variation, or only a style change with no layout demand, skip ControlNet and reach for prompt work or LoRA for image models, which retunes style cheaply without pinning geometry. The method assumes lines up with the picture you want. A shifted, blurry, or conflicting map fights the prompt, and both lose.
Code
This snippet is the addition rule from the equation above, not the full UNet. It shows the silent start and the trained state on two channels. Channel one repeats the worked example.
import numpy as np
def zero_inject(base: np.ndarray, control: np.ndarray, w: float, scale: float = 1.0) -> np.ndarray: return base + scale * (w * control) # silent at init while w is 0, additive once trained
base = np.array([3.0, 1.0])control = np.array([2.0, -1.0])print(zero_inject(base, control, 0.0)) # -> [3. 1.]print(zero_inject(base, control, 0.4)) # -> [3.8 0.6]print(zero_inject(base, control, 0.4, scale=0.5)) # -> [3.4 0.8]The first line equals the base model exactly. The second line matches the worked example on channel one: . The third shows the scale knob softening the pull to and .
Watch Out For
A dense edge map that overrules your prompt
Symptom: you ask for an oil painting of a dog and get one, but the background keeps the straight panels and hard corners of the car in your reference photo. Every line in the map survived, including the ones you never wanted.
Fix: simplify the map before it reaches the model. Blur it lightly, raise the edge threshold so faint lines drop out, or crop to the region you actually care about. If the prompt still loses, turn the conditioning scale down until the layout holds and the texture lets go.
A condition that disagrees with the prompt
Symptom: the pose stick figure leaps while your prompt says sitting, and you get smeared limbs or an extra arm. Two authorities gave two orders and the sampler split the difference.
Fix: give each region one authority. Redraw or re-pose the map to match the words, or rewrite the prompt to match the map. When layout truly doesn't matter for a pass, switch ControlNet off for that pass instead of fighting it.
The Quick Version
- ControlNet steers layout with a spatial map while the prompt steers content, because text alone can't pin positions.
- The base model stays frozen and a trainable copy reads the noisy latent plus the map.
- Zero convolutions start at zero output, so training begins as the base model and earns control gradually.
- Each block computes . In the fixture, 3.0 plus 0.8 makes 3.8, or 3.4 at half scale.
- More conditioning buys fidelity and costs variety, so mismatched maps and heavy scales give rigid or smeared images.