VAR Visual Autoregressive Modeling

VAR predicts images scale by scale instead of pixel by pixel, generating a thumbnail first and then each finer resolution in one parallel step.

VAR predicts each resolution of the image as one parallel step, from a single token to the full map

Why Does This Exist?

Pixel-by-pixel autoregression is correct and unusable: $1{,}024$ sequential steps per image makes high-resolution sampling crawl. The ordering itself is the inefficiency, since it ignores how images actually compose, globally first, details later.

VAR redefines the autoregressive unit as the scale. Predict the $1 \times 1$ thumbnail, then the $2 \times 2$ map given it, doubling up to full resolution, with all tokens inside one scale predicted in parallel. Ten steps replace a thousand. The hierarchical token idea it builds on comes from VQ-VAE-2, and the pixel-order baseline is PixelCNN.

Think of It Like This

A painter working thumbnail to canvas

A painter starts with a one-inch thumbnail: composition only. Then a small study, then a half-size canvas, then the full mural, each stage copying the previous one's layout and adding one level of detail everywhere at once. Nobody paints murals one bristle-dot at a time in reading order.

Scales are the study sizes and parallelism is painting each whole study at once. The method fails when a detail needed at the end was never hinted in the thumbnail, so coarse maps must carry enough layout to anchor everything above.

How It Actually Works

The quantiser produces token maps $r_1, \dots, r_K$ at doubling resolutions, $1 \times 1$ through $16 \times 16$ for $256$ -pixel images with $K = 10$ scales. The transformer models:

p(r_1, \dots, r_K) = \prod_{k=1}^{K} p(r_k \mid r_{<k})

Conditioning uses all previous (upsampled) maps, and every token within $r_k$ generates in parallel. Training is next-scale cross-entropy; sampling runs $K$ forward passes total. Block-causal attention masks enforce the scale ordering while staying parallel inside each scale.

Worked example

Scales $1, 2, 4, 8, 16$ (five shown of ten): token counts $1, 4, 16, 64, 256$ , summing to $341$ tokens for the visible ladder and $680$ across all ten scales to $16 \times 16$ . Sampling needs $10$ passes versus $1{,}024$ pixel steps, a hundredfold cut, while likelihood stays competitive because the factorization respects image structure better than raster order does.

Code

Scale ladder token counts, showing where the parallelism comes from.

1def var_ladder(scales: int = 10) -> list[int]:2    """Token counts per scale from 1x1 doubling upward."""3    return [(2 ** k) ** 2 for k in range(scales)]4
5
6ladder = var_ladder(5)7print(ladder)  # -> [1, 4, 16, 64, 256] tokens per scale8print(sum(ladder))  # -> 341 tokens across the five shown scales

Watch Out For

Coarse errors compounding upward

Every scale conditions on all previous ones, so a bad $2 \times 2$ layout poisons all eight scales above it. There is no revision pass; the ladder only climbs.

Invest capacity in early scales and verify thumbnails before trusting details. Rejection-sample at coarse levels where it is cheap, not at full resolution.

Quantiser mismatch across scales

One shared codebook must serve thumbnails and fine textures alike. Starve either end and its scales degrade; the ladder is only as strong as its weakest resolution.

Check per-scale reconstruction separately during tokeniser training. Scale-specific codebooks cost parameters but buy uniform quality.

The Quick Version

VAR factorises generation over scales, not pixels: $p(r_k \mid r_{<k})$ .
All tokens within one scale generate in parallel; only scales run sequentially.
Ten passes replace $1{,}024$ pixel steps for $16 \times 16$ token maps.
Our five-scale ladder holds $1 + 4 + 16 + 64 + 256 = 341$ tokens.
Coarse mistakes compound upward, and the codebook must serve every scale evenly.