DALL-E Family

The DALL-E lineage moved from token-predicting transformers to CLIP-guided diffusion, teaching the field to separate understanding text from painting it.

DALL-E 1 predicts discrete tokens directly while DALL-E 2 splits into a CLIP prior plus a diffusion decoder

Why Does This Exist?

Early text-to-image pasted language onto GANs and got surreal collages. The missing piece was a joint representation where words and pictures share meaning before any pixel renders. CLIP supplied exactly that, and the DALL-E lineage turned it into a product arc the whole field followed.

DALL-E 1 proved transformers could paint from tokens. DALL-E 2's unCLIP split understanding from rendering: a prior maps text to CLIP image embeddings, a decoder paints from them. That modular split, plus native inpainting and variations, set the template every later system refines. The autoregressive ideas underneath are covered in autoregressive image generation, and the family map is text-to-image and text-to-video.

Think of It Like This

A studio that hires an interpreter, then painters

The first studio employs bilingual painters who read briefs and paint directly: capable, but every painter must master both languages. The second studio hires one brilliant interpreter who converts every brief into a universal visual sketch, then rooms of monolingual painters who only read sketches.

The CLIP embedding is the universal sketch. Specialisation beats bilingualism at scale: the interpreter perfects meaning while painters perfect rendering. The studio fails when the sketch drops a detail no painter can recover, the information bottleneck below.

How It Actually Works

DALL-E 1 trains a $12$ B decoder-only transformer on $256 \times 256$ images compressed by a dVAE to $32 \times 32$ tokens ( $1{,}024$ positions) concatenated after $256$ text tokens. DALL-E 2 replaces this with unCLIP: a prior, autoregressive or diffusion, maps CLIP text embeddings to CLIP image embeddings, and a diffusion decoder renders pixels conditioned on the image embedding plus optional text:

\text{text} \xrightarrow{\text{prior}} z_{\text{CLIP-img}} \xrightarrow{\text{decoder}} \text{pixels}

Variations come free: feed a real image's CLIP embedding to the decoder for same-meaning alternatives. Inpainting works by clamping known regions during decoding, and DALL-E 2's outpainting popularised frame extension as a product feature.

Worked example

DALL-E 1 sequence: $256$ text tokens plus $1{,}024$ image tokens $= 1{,}280$ positions through $12$ billion parameters. The dVAE compresses $8\times$ per side ( $256 \to 32$ ), so each token carries an $8 \times 8$ patch. DALL-E 2's prior instead outputs one $512$ -number CLIP vector per prompt, and the decoder expands it to $1{,}024$ -pixel images, trading token-by-token control for semantic reliability.

Code

Sequence budgeting across the two designs, showing where length goes.

1def dalle_budgets() -> tuple[int, int]:2    """Total modelled positions for v1 tokens versus v2 embedding."""3    v1 = 256 + 32 * 32  # text tokens plus dVAE image tokens4    v2 = 1  # one CLIP image embedding through the prior5    return v1, v26
7
8print(dalle_budgets())  # -> (1280, 1) tokens versus one vector

Watch Out For

CLIP bottleneck dropping fine detail

The single-vector sketch cannot carry exact counts, small text, or precise spatial relations. Painters render beautiful scenes with wrong details, and no decoder tuning recovers dropped meaning.

Enrich conditioning with token-level text features alongside the pooled vector. Audit compositional prompts specifically, not just beauty.

Photorealistic faces and the consent gap

High-fidelity strangers enable impersonation and misinformation at trivial cost. Capability demos that celebrate likeness without consent guardrails ship harm with the feature.

Gate photorealistic human generation behind consent and provenance tooling. Publish the policy with the model, not after the incident.

The Quick Version

DALL-E 1 predicts $1{,}024$ dVAE tokens after $256$ text tokens with a $12$ B transformer.
DALL-E 2 splits into a CLIP prior plus a diffusion decoder, the unCLIP design.
One $512$ -number embedding replaces $1{,}280$ modelled positions as the interface.
Variations and inpainting fall out of the embedding interface naturally.
Bottleneck detail loss and face-consent risks are the standing concerns.