ImageGPT

ImageGPT trains a plain GPT transformer on flattened pixel sequences, proving the language modelling recipe learns visual features with no 2D machinery.

ImageGPT flattens pixels into a token sequence and trains an unmodified GPT on next-pixel prediction

Why Does This Exist?

Vision and language ran on separate machinery: convolutions here, transformers there, with endless debate about which bias vision truly needs. Settling it required the purest experiment, the exact language recipe with nothing visual added, to see how far sequence modelling alone reaches.

ImageGPT is that experiment. Flatten the image, cluster colours to $512$ codes, train GPT-2 architectures as-is. The models generated coherent images and, more importantly, learned features rivalling supervised pretraining on classification probes. The result licensed the transformer takeover of vision. The pixel-level predecessor is PixelCNN, and the scaled-up descendant is Parti.

Think of It Like This

A novelist asked to paint by numbers

A novelist who has never held a brush is handed paint-by-number canvases with $512$ numbered colours and told to fill them left to right, top to bottom, guessing each cell from the story so far. No sketching lessons, no colour theory, just sequence instinct.

Surprisingly the novelist produces recognisable scenes and develops opinions about composition, because narrative prediction transfers to visual prediction. The paintings stay small, the novelist works one cell at a time, but the lesson lands: the machinery matters less than the prediction game.

How It Actually Works

Preprocess crops to $32 \times 32$ and k-means colours to $512$ clusters ( $9$ -bit colour), turning each image into $1{,}024$ tokens. Train a decoder-only transformer, GPT-2 small through XL sizes, on next-token cross-entropy with a single learned position embedding per index. No 2D position grids, no convolution stems, no image-specific changes whatsoever:

L = -\sum_{i=1}^{1024} \log p_\theta(t_i \mid t_{<i}), \qquad t_i \in \{0, \dots, 511\}

After generative training, freeze the transformer and probe it: linear classifiers on middle-layer activations match supervised ResNet features on several benchmarks, showing the next-pixel game taught genuine visual understanding, not just texture statistics.

Worked example

One $32 \times 32$ image is $1{,}024$ positions times a $512$ -way choice. At $4.5$ nats per position the image costs $4{,}608$ nats. The XL model's $6.8$ billion parameters process sequences of length $1{,}024$ with full attention, about a million pairwise interactions per layer, which is why experiments stayed at $32$ pixels: the sequence recipe works, but its compute bill scales with the square of resolution.

Code

Sequence shaping for the GPT treatment, the only vision code in the recipe.

1def gpt_sequence(px: int = 32) -> tuple[int, int]:2    """Positions and vocabulary for ImageGPT preprocessing."""3    positions = px * px  # flattened raster order4    vocab = 512  # k-means colour clusters, 9 bit colour5    return positions, vocab6
7
8print(gpt_sequence())  # -> (1024, 512) positions and colour codes

Watch Out For

Flattening away resolution and calling it general

Results at $32$ pixels do not transfer to megapixel generation. Full attention over $65{,}536$ positions is computationally hopeless, which is why practical descendants quantise patches first instead of pixels.

Read ImageGPT as a representation-learning result, not a generator blueprint. Scaling the recipe needs token compression, the VQ route.

Colour clustering artefacts mistaken for model failures

K-means $512$ colours band smooth gradients and shift hues slightly. Blame the model for banded skies and you will tune transformers to fix a preprocessing artifact.

Inspect reconstructions through the clustering alone first. Anything already broken there is data pipeline, not architecture.

The Quick Version

ImageGPT trains unmodified GPT-2 transformers on flattened, colour-clustered pixels.
$32 \times 32$ images become $1{,}024$ positions over a $512$ -code vocabulary.
Frozen features rival supervised pretraining on classification probes.
Quadratic attention confines the pure recipe to tiny images.
Colour banding comes from k-means preprocessing, not the transformer.