Neural Audio Codecs
Raw audio is tens of thousands of numbers per second, far too long for a Transformer to write out. A neural audio codec squeezes each slice of sound into a few codebook tokens, so generating audio becomes predicting the next token.
Why Does This Exist?
Count what raw audio asks of you. One second sampled 16,000 times is a list of 16,000 numbers. Ten seconds is 160,000 numbers in a row, and an autoregressive model would have to predict them one at a time while its cost grows with the square of the sequence length. That is not a big model problem. It is a hopeless sequence length problem.
So you want what text already has: a short string of discrete symbols. Classical codecs do not give you that. MP3 is a hand-built perceptual pipeline tuned for ears and file size, and its bytes were never meant to be next-token targets. A neural audio codec is built for the opposite goal. It learns to squeeze sound into a short integer alphabet that a Transformer already knows how to model, then rebuild the wave from those integers afterward.
Two sentences of grounding, since everything below assumes them. An autoencoder is an encoder that squeezes its input into a compact code plus a decoder that rebuilds the input from that code, trained to make the rebuild match the original. A neural codec is that idea with one hard bottleneck in the middle: the code must be discrete tokens, not free decimals.
This page covers the compressor in the middle: squeeze time, snap to a codebook, refine the leftover, decode. The text version of the same trick lives in tokenization. The wave-to-text trip is automatic speech recognition, the architecture that hears a whole clip before writing lives in encoder-decoder ASR, and the text-to-wave trip that increasingly consumes codec tokens is text to speech.
Think of It Like This
The paint counter's swatch books
You bring a chipped paint flake to a hardware store and ask for that exact color. The clerk does not sell you liquid light. They open a swatch book of 1,024 printed colors, hold your flake against the pages, pick the closest swatch, and write down its number. That number is your first token.
But the match is slightly off, so they mix a correction for the difference between your flake and the swatch. Then they open a second, finer book, match the correction itself to its closest swatch, and write down that number too. Two small numbers now describe your color better than either book could alone, and the clerk can remix both later.
Where it stops: paint corrections are continuous and order-free, while a codec's stages are strictly ordered, coarse first and detail later. Shuffle the two numbers and the remix breaks, because the decoder reads stage two as a correction to stage one, never the other way around.
How It Actually Works
Three moves in order: squeeze time with an encoder, snap each frame to codebook entries with residual refinement, rebuild with a decoder. The symbols below, each defined where it first appears: is one frame's continuous vector, is how long that vector is, is a codebook (a list of example vectors), is how many entries the codebook holds, is entry number , is a chosen entry's integer id, is a leftover vector, and is the number of refinement stages.
Squeeze time first
The encoder is a stack of convolutions that collapses chunks of waveform into one vector per chunk. Its stride says how many waveform samples fold into a single frame. With frames per second and a sample rate :
Say the input is mono audio at samples per second and the encoder's total stride is . Then frames per second, and each frame is one vector of length . Ten seconds is no longer 160,000 numbers to predict. It is 500 vectors, and the vectors are about to become integers.
Snap each frame to the nearest swatch
A codebook is a fixed list of example vectors that the model learns during training. Quantization means replacing the continuous with the id of the closest entry:
In plain words: measure the squared distance from to every entry, keep the winner's integer id , throw the decimals away. The decoder holds the same book, so looking up from restores an approximation of the frame. A book with entries costs 10 bits per frame, because .
One book alone forces an ugly choice. A small book cannot cover the variety of real sound, so reconstructions come out crude. A giant book gives the downstream language model a gigantic vocabulary to learn, which is exactly the problem you were escaping. The fix is several small books in sequence.
Quantize the leftover, then the leftover's leftover
Residual vector quantization spends one small codebook per stage, where each stage cleans up the previous stage's error. With stages:
In plain words: stage one picks the closest entry to and keeps its id . The leftover is what stage one missed. Stage two quantizes that leftover with a second book, keeping id . The rebuilt frame (read "z-hat", the estimate of ) is the sum of the two chosen entries. Early stages carry the broad sound, later stages carry texture, and the ids stay small because no single book has to cover everything.
Worked example: one frame, two stages
Take one frame . Book one holds and . Squared distances:
- To : .
- To : .
Stage one keeps id (entry ). The leftover is . Book two holds and . Distances: to is , to is . Stage two keeps id . The second leftover is , and the rebuild matches the frame exactly. Real audio rarely lands this cleanly; leftovers shrink but seldom vanish.
Now the bitrate with these settings, both books at entries (10 bits each): 50 frames per second 2 stages 10 bits bits per second. That is the whole compression story in one line.
What the language model actually predicts
Each frame becomes integers in coarse-to-fine order, so the model predicts a flat stream like for frame one, then for frame two. At 50 frames per second with 2 stages that is 100 tokens per second against 16,000 raw samples per second, about 160 times shorter. Because the stages are ordered, good generators predict or condition on the coarse ids before the fine ones rather than treating all ids as equals.
Name the trade-off before you use it. More stages buy fidelity and cost sequence length plus modeling effort; fewer stages are cheap to generate and sound it. And when your job is moving audio between people, calls, streaming, archives, reach for a plain perceptual codec instead. A neural codec earns its keep when its tokens feed generation or understanding. It assumes new audio resembles its training diet, and unfamiliar voices, instruments, or noise come back metallic.
Code
The snippet below replays the worked example: nearest-entry search, then the leftover pass. Pure Python, no dependencies, so you can run it anywhere.
Vector = list[float]
def dist2(a: Vector, b: Vector) -> float: return sum((x - y) ** 2 for x, y in zip(a, b))
def nearest_id(z: Vector, book: list[Vector]) -> int: return min(range(len(book)), key=lambda k: dist2(z, book[k]))
def rvq(z: Vector, books: list[list[Vector]]) -> tuple[list[int], Vector, Vector]: ids: list[int] = [] leftover: Vector = list(z) for book in books: i = nearest_id(leftover, book) ids.append(i) entry = book[i] leftover = [x - e for x, e in zip(leftover, entry)] recon: Vector = [0.0 for _ in z] for i, book in zip(ids, books): recon = [x + e for x, e in zip(recon, book[i])] return ids, recon, leftover
z: Vector = [0.75, 0.25]books: list[list[Vector]] = [[[1.0, 0.0], [0.0, 1.0]], [[0.0, 0.0], [-0.25, 0.25]]]ids, recon, leftover = rvq(z, books)print(ids) # -> [0, 1]print(recon) # -> [0.75, 0.25]print(leftover) # -> [0.0, 0.0]print(50 * 2 * 10) # -> 1000The ids pick then , the rebuild matches the frame, the leftover hits zero, and the last line replays the bitrate: 50 frames, 2 stages, 10 bits each.
Watch Out For
Most of the codebook goes silent
Training sometimes lets a few popular entries do all the work while hundreds of codes sit unused. This is codebook collapse. The symptom is unmistakable: reconstructions turn metallic and bubbly, and a usage histogram shows a handful of ids carrying nearly every frame.
Do not fix it by training longer. Track code usage or perplexity per book during training, and revive dead entries by resetting them near active ones or by updating books with slow averages of the vectors assigned to them. If usage is still skewed at the end, the book is too big for the data, so shrink before blaming the decoder.
Fine tokens predicted without their coarse parents
The ids per frame are not interchangeable slots. Stage two is a correction to stage one, so a generator that predicts all stages flat, or shuffles their order, produces audio that sounds clean while saying the wrong thing: stable texture over scrambled content.
The symptom is eerie, good acoustics with broken words or wandering pitch. Condition each stage on the earlier ones, through staged heads or a coarse-first delay pattern, and keep the on-disk order coarse to fine. If you subsample stages to save length, drop from the fine end only.
The Quick Version
- A neural audio codec turns a waveform into a short string of integer tokens so a Transformer can generate sound with next-token prediction.
- The encoder folds thousands of samples per second into tens of frame vectors per second; with stride 320, 16,000 samples become 50 frames.
- Each frame is replaced by the id of its nearest codebook entry, and residual stages quantize the leftover error coarse to fine.
- A 1,024-entry codebook costs 10 bits per pick, so 50 frames times 2 stages times 10 bits is 1,000 bits per second.
- Generation must respect stage order: fine detail means nothing without the coarse sound it corrects.
- For transport between people use a plain perceptual codec; neural tokens earn their keep when they feed generation or understanding.