VQ-VAE-2

VQ-VAE-2 stacks two codebooks, a coarse one for layout and a fine one for texture, so large sharp images get discrete tokens at both scales.

Why Does This Exist?

Single-scale VQ-VAE hits a wall on big images. One codebook must describe both the arrangement of a scene and the weave of every fabric in it, so either the grid is coarse and details smear, or the grid is fine and the token sequence grows too long for any transformer to model.

Hierarchy breaks the dilemma. A top-level map with heavy downsampling captures where things are, and a bottom-level map conditioned on it fills in what they look like up close. Each level gets its own codebook and its own generator, matched to its job. The single-scale version it extends is VQ-VAE, and the generators that consume both maps are covered in autoregressive image generation.

Think of It Like This

A city map plus a set of street photos

A tourist guide gives you two things: a small city map showing districts and avenues, and a thick photo book of individual streets. The map alone cannot show you a doorway; the photo book alone cannot show you how districts connect. Together they describe the whole city at the right resolution for each question.

The top codebook is the city map and the bottom codebook is the photo book. The failure mode is a mismatch: photos conditioned on the wrong district describe streets that lead nowhere, which is why the bottom level must always see the top level during training.

How It Actually Works

The encoder produces two quantised maps. The top map downsamples aggressively, typically $8\times$ , and its tokens model global structure with a large autoregressive prior. The bottom map downsamples modestly, typically $4\times$ , and its tokens model local texture conditioned on the top map, so it never has to rediscover the layout:

x \to (z_{\text{top}}, z_{\text{bottom}}), \qquad p(z_{\text{bottom}} \mid z_{\text{top}}), \quad p(z_{\text{top}})

Each map has its own codebook and its own commitment loss, with the same nearest-lookup rule as single-scale VQ-VAE. Generation runs top-first: sample the layout tokens, then sample detail tokens given the layout, then decode once.

Worked example

Take a $64 \times 64$ image. The top encoder downsamples $8\times$ to an $8 \times 8$ map: $64$ layout tokens. The bottom encoder downsamples $4\times$ to a $16 \times 16$ map: $256$ detail tokens. The top prior models a sequence of length $64$ , short enough for a big transformer, while the bottom prior handles $256$ tokens but with the layout already given, a far easier prediction task. Total $320$ tokens describe $4{,}096$ pixels, roughly $13$ pixels per token, with structure and texture cleanly separated.

Code

Map sizes from image size and downsampling factor, the arithmetic that sizes both priors.

1def token_maps(image_px: int, top_down: int = 8, bottom_down: int = 4) -> tuple[int, int]:2    """Count top (layout) and bottom (detail) tokens for a square image."""3    top = (image_px // top_down) ** 24    bottom = (image_px // bottom_down) ** 25    return top, bottom6
7
8print(token_maps(64))  # -> (64, 256)

Watch Out For

Training the bottom level without top conditioning

Drop the conditioning of bottom on top and both levels learn the same global structure redundantly. Samples then show texture that disagrees with layout, fur flowing across a face boundary, because nothing told the detail level what the layout level decided.

Always feed upsampled top features into the bottom encoder and decoder. Verify by scrambling top tokens at sample time: bottom textures should visibly follow the scramble.

Starving one level of codebook capacity

Give both levels $512$ codes and the top level, which needs far fewer distinct layouts than the bottom needs textures, wastes capacity while the bottom starves. Detail quality plateaus no matter how long you train.

Size codebooks to the job: smaller on top, larger at the bottom, and check per-level usage separately. A level using under a tenth of its codes is oversized.

The Quick Version

VQ-VAE-2 quantises images at two scales: a coarse layout map and a fine detail map.
The top map downsamples about $8\times$ and models global structure with its own prior.
The bottom map downsamples about $4\times$ and models texture conditioned on the layout.
A $64$ pixel image becomes $64$ layout tokens plus $256$ detail tokens.
Bottom must always see top, and codebook sizes should match each level's job.