Text-to-Image Generation Service
AdvancedOverview
A Text-to-Image Generation Service (like Midjourney or DALL·E) uses diffusion models to convert natural language prompts into high-resolution images. Unlike text generation which streams instantly, image generation is computationally heavy and takes 5 to 60 seconds per request. The core system design challenge is managing asynchronous GPU workloads, strict queuing, dynamic batching, and handling massive, bursty traffic without dropping requests.
Functional Requirements
- Users submit a text prompt (and optional parameters like aspect ratio or style).
- The system processes the prompt and returns 1-4 high-resolution images.
- Users receive real-time progress updates (e.g., 20%, 50%, 100%) during generation.
- Prompts and resulting images must be scanned for NSFW or policy-violating content.
Non-Functional Requirements
- Asynchronous Processing: the system must not block HTTP requests for 60 seconds.
- High GPU Utilization: batch requests dynamically to maximize inference efficiency.
- Resilience: if a GPU node crashes mid-generation, the job must be requeued.
- Autoscaling: scale expensive GPU clusters up and down based on queue depth.
Capacity Estimation
Assume 1 Million users per day generating 10 images each (10 Million images/day).
- Compute: Generating one image takes ~10 seconds on an A10G GPU. 10M images * 10s = 100M GPU-seconds/day. That requires at least 1,150 GPUs running at 100% utilization 24/7. Due to peak traffic bursts, the actual cluster size must be 2-3x larger.
- Storage: A generated 1024x1024 image (JPEG/WebP) is ~1MB. 10M images = 10 TB/day of blob storage (S3). Over a year, this requires ~3.6 Petabytes.
- Network: Serving 10 TB/day out to clients requires a CDN handling ~1 Gbps of sustained egress.
High-Level Architecture
The architecture heavily relies on an Asynchronous Job Queue pattern. A user sends a prompt to the API Gateway, which immediately writes a "Job" to a database, pushes the Job ID to a distributed Message Queue (e.g., RabbitMQ or SQS), and returns a 202 Accepted status to the client. A massive fleet of GPU Worker nodes pulls jobs from the queue. The worker runs prompt safety checks, generates the image using a Diffusion Model, streams progress updates back to the client via WebSockets/Redis PubSub, uploads the final image to S3, and updates the Job status in the DB.
Data Model
| Entity | Fields / Schema | Storage Choice |
|---|---|---|
| generation_job | job_id (PK), user_id, prompt, status (queued, processing, completed, failed), image_url, created_at | Relational DB (PostgreSQL) or Document DB (MongoDB) |
| generated_assets | asset_id, bytes (image file) | Blob Storage (S3) + CDN (CloudFront) |
| job_queue | job_id, priority, payload | Message Broker (RabbitMQ / Redis / Kafka) |
Detailed Design
Dynamic Batching
Unlike standard web servers that process one request per thread, GPU workers must process multiple requests simultaneously to be cost-effective. A specialized Inference Router pulls multiple jobs from the queue, waits a few milliseconds to form a batch of identical shapes (e.g., four 1024x1024 requests), and pushes them through the Diffusion Model in a single forward pass.
Progress Streaming
Because generation takes time, UX requires a progress bar. The GPU worker publishes progress events (e.g., "step 10/50") to a fast pub/sub system like Redis. The API Gateway (or a dedicated WebSocket service) subscribes to these events and pushes them to the connected client. This decouples the heavy GPU worker from managing thousands of persistent WebSocket connections.
Safety and Moderation
Safety is a two-step process. First, the text prompt is classified using a fast NLP model to block obvious violations (e.g., violence, CSAM). Second, because diffusion models can accidentally generate inappropriate content even from safe prompts, the generated image is run through an Image NSFW classifier before being uploaded to S3 and shown to the user.
Bottlenecks & Solutions
The main bottleneck is GPU Cold Starts and Autoscaling. While stateless CPU microservices can autoscale in seconds, spinning up a new GPU instance, loading OS drivers, and pulling a 10GB+ model weight file into VRAM can take 5-10 minutes. If traffic spikes suddenly, the queue will back up. Solutions include maintaining a "warm pool" of idle GPUs, optimizing container image sizes, and using weights-caching on the host OS.