Text-to-Image and Text-to-Video

Text-to-image and text-to-video turn written prompts into pictures and clips by aligning language embeddings with visual generation, then sampling.

Text encodings steer image generators, and video adds temporal layers to keep all frames consistent

Why Does This Exist?

Words are cheap and pictures are hard. Billions of people can describe a scene they cannot draw, photograph, or render. Bridging that gap turns language, humanity's most accessible interface, into a universal art department, and extending it to video turns storyboards into footage.

The bridge has two spans. First, align frozen language embeddings with a visual generator through cross-attention or prefix conditioning. Second, for video, add temporal layers so frames agree, because per-frame generation flickers. Model portraits in this gallery include the DALL-E family, Imagen, Midjourney-style models, Sora-style models, and the architecture survey in video generation architectures.

Think of It Like This

An architect's office with a film unit

An architect's office takes written briefs and returns building renders: the brief sets the program, draftsmen supply the visuals. Years later the office opens a film unit that animates the buildings, and every hallway walkthrough must show the same building from every angle.

The text encoder is the brief reader, the image generator the draftsmen, temporal layers the continuity supervisor. The office fails on briefs it cannot parse, contradictory programs, and on walks through buildings the draftsmen never designed from all sides.

How It Actually Works

A frozen text encoder, CLIP, T5, or similar, converts prompts to embeddings once. The visual generator, diffusion UNet, DiT, or autoregressive predictor, reads them through cross-attention or prefix tokens at every scale. Alignment quality is scored by CLIP similarity between prompt and output, plus human preference on composition and counting.

Video adds the time axis: temporal attention across frames, motion modules, and consistency losses. A $4$ -second clip at $24$ fps holds $96$ frames that must share identity, lighting, and camera logic. Training pairs captioned video with text at internet scale, far noisier supervision than image alt-text.

Worked example

Prompt a red bicycle leaning on a blue wall: the text encoder emits $77$ tokens, the generator attends them while denoising. CLIP score between output and prompt lands near $0.32$ for a good match versus $0.20$ for a mismatched colour. For video, $4 \times 24 = 96$ frames denoise jointly; a per-frame model scores $0.31$ per frame yet flickers the bicycle's frame shape, while the joint model holds geometry across all $96$ .

Watch Out For

Attribute binding failures hidden by high scores

CLIP scores forgive swapped attributes, where a red bicycle plus blue wall scores like a blue bicycle plus red wall when both colours appear somewhere. Compositional prompts fail silently under global similarity.

Probe binding explicitly with colour-shape swaps. Human checks on object-attribute pairs catch what the score forgives.

Counting and text rendering illusions

Models render three apples as four and garble short words into glyph-like marks. Demos cherry-pick successes while products need reliability, and neither counting nor lettering is truly solved.

Test counts one through seven and short quoted words directly. Scope product claims to what the tests show, not the gallery.

The Quick Version

Text encoders convert prompts to embeddings that steer visual generators.
CLIP similarity near $0.32$ marks good prompt following versus $0.20$ mismatches.
Video adds temporal layers so $96$ frames share identity and camera logic.
Attribute swaps fool global scores; probe binding with deliberate swaps.
Counting and rendered text remain unreliable despite gallery successes.