Stable Diffusion 3
The 2024 architecture introducing Multimodal Diffusion Transformers (MM-DiT) and Rectified Flow matching to vastly improve prompt adherence and text generation.
Paper: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Authors: Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Glaser, Amelia Wattenberger, Robin Rombach · 2024
Read the paper
The Problem
Previous iterations of Stable Diffusion (like SD 1.5 and SDXL) struggled heavily with prompt adherence, especially for complex compositional requests and generating legible text. These earlier models used a unified U-Net architecture and relied on standard diffusion formulations, which created complex, curved trajectories from noise to data that were computationally expensive to navigate during inference.
The Idea
Stability AI fundamentally re-architected the model on two fronts. First, they discarded the U-Net in favor of a Multimodal Diffusion Transformer (MM-DiT). Crucially, instead of concatenating text and image tokens early, they maintained separate streams for text and image data. Second, they replaced standard diffusion with Rectified Flow, a formulation that connects data and noise distributions on straight paths, yielding better generation at fewer sampling steps.
How It Works
Multimodal Diffusion Transformer (MM-DiT) Instead of forcing image and text representations into a single shared backbone, MM-DiT uses two separate sets of weights for the two modalities. Text tokens and image patches are processed through their own independent self-attention and MLP layers. They only exchange information during a joint cross-attention phase before splitting back apart. This dual-stream design allows the model to learn modality-specific representations while still aligning them tightly for generation.
Rectified Flow Matching Rather than simulating a complex thermodynamic diffusion process, Rectified Flow maps the prior (pure noise) to the target data distribution via an Ordinary Differential Equation (ODE). Specifically, it aims to create straight-line trajectories between noise and data. This straightening means the model can take larger step sizes during inference without veering off course, dramatically reducing the number of steps required to generate a high-quality image.
Text Encoders To improve text adherence, SD3 leverages three different text encoders simultaneously: two CLIP models (ViT-L and ViT-bigG) and a massive T5-XXL model. The T5-XXL model provides the deep language understanding necessary for spelling words correctly and following intricate positional instructions.
Why It Mattered
SD3 represented a leap forward in open-weight image generation. It essentially solved the long-standing "text in images" problem, allowing users to generate signs, labels, and typography with high reliability. The shift to a Transformer backbone also meant the model followed predictable scaling laws, enabling a family of models ranging from 800M to 8B parameters.
What Came After
The open release of the SD3 Medium weights sparked a new generation of community fine-tunes and control tools. The architectural choices (dual-stream Transformers and flow matching) cemented a shift in the broader generative AI landscape, with competitors like Flux largely adopting similar paradigms for high-end text-to-image models.