BigGAN

BigGAN showed that class-conditional generation scales brutally well: bigger batches, wider layers, and a truncation knob trading variety for fidelity.

Why Does This Exist?

Before BigGAN, scaling GANs up mostly broke them. Bigger models collapsed more spectacularly, and ImageNet's thousand classes smeared together into generic textured blobs. The field suspected adversarial training simply could not absorb modern compute budgets.

BigGAN proved the opposite with engineering discipline: batch size $2048$ , shared class embeddings, orthogonal regularisation, and self-attention at high resolution. Samples jumped from blurry collages to recognisable objects in scene context. The progressive alternative to this scale-first approach is ProGAN, and the style-separation alternative is StyleGAN.

Think of It Like This

A census versus a street poll

A pollster asking ten people about a thousand neighbourhoods gets noise. A census asking two thousand people per neighbourhood gets a sharp picture of each one, including rare streets the small poll never samples twice.

Batch size is the census budget. With $2048$ samples per step, gradient estimates stabilise, rare classes appear every step, and the critic's feedback stops whiplashing. The census costs far more clipboards, which is the compute bill below, and counting everyone tempts you to only report typical residents, which is the truncation tradeoff.

How It Actually Works

The generator uses conditional batch normalisation: each class $c$ gets an embedding that supplies the scale and bias of every normalisation layer, so one network carries a thousand class behaviours. Self-attention blocks at $64 \times 64$ let distant regions coordinate, keeping multi-object scenes coherent. Orthogonal regularisation keeps weight matrices near-orthogonal, preserving gradient norms through very deep stacks.

The truncation trick operates at sample time. Instead of drawing $z$ from the full Gaussian, draw from a truncated Gaussian limited to a threshold $\psi$ : values beyond $\psi$ standard deviations are resampled. At $\psi = 1.0$ samples match training diversity; near $\psi = 0.5$ and below, samples concentrate near the mode, crisper but more typical, and variety visibly shrinks.

Worked example

With $z$ standard normal and $\psi = 0.5$ , roughly $62$ percent of raw draws fall inside $[-0.5, 0.5]$ and survive, since $\Phi(0.5) - \Phi(-0.5) \approx 0.383$ . The rest resample until they land inside. A golden retriever at $\psi = 0.4$ is tack-sharp but one of a few canonical poses; at $\psi = 1.0$ poses diversify and the occasional three-legged dog slips through. Fidelity and variety share one knob.

Code

Truncated sampling, the few lines behind the fidelity knob.

1import random2
3
4def truncated_noise(n: int, psi: float = 0.5, seed: int = 0) -> list[float]:5    """Draw standard normals, resampling anything outside [-psi, psi]."""6    rng = random.Random(seed)7    z = [rng.gauss(0, 1) for _ in range(n)]  # start from the full Gaussian8    for i, v in enumerate(z):9        while abs(v) > psi:  # resample the tails back inside10            v = rng.gauss(0, 1)11        z[i] = v12    return z13
14
15z = truncated_noise(10000, psi=0.5)16print(all(abs(v) <= 0.5 for v in z))  # -> True every draw inside bounds

Watch Out For

Judging the model at one truncation level

Papers report BigGAN quality near $\psi = 0.5$ , where samples flatter the model. Deploy at $\psi = 1.0$ for coverage and the same model produces noticeably weaker images, including class leakage and broken geometry.

Always evaluate at the truncation you will ship. Report quality and diversity as a curve over $\psi$ , never as a single cherry-picked point.

Assuming big batches fit small hardware

The $2048$ batch ran across hundreds of TPU cores with synchronised statistics. Naive small-batch replication changes batchnorm behaviour and gradient noise together, and the recipe's stability does not transfer.

Scale batch and learning rate together, accumulate gradients to emulate size, and expect retuning. BigGAN is a systems achievement as much as an architecture.

The Quick Version

BigGAN scales conditional generation with batch $2048$ , class embeddings, and orthogonal weights.
Conditional batch norm gives each of a thousand classes its own normalisation behaviour.
Truncation near $\psi = 0.5$ sharpens samples by resampling the Gaussian tails.
About $38$ percent of raw draws survive inside $[-0.5, 0.5]$ before resampling.
Evaluate at shipping truncation, and do not expect the recipe to transfer to small batches unchanged.