Skip to content
AI360XpertBeta

Neural Audio Codecs

Raw audio is tens of thousands of numbers per second, far too long for a Transformer to write out. A neural audio codec squeezes each slice of sound into a few codebook tokens, so generating audio becomes predicting the next token.

A neural audio codec squeezes each slice of waveform into a couple of codebook tokens with residual vector quantization, so a Transformer can predict sound the way it predicts text
A neural audio codec squeezes each slice of waveform into a couple of codebook tokens with residual vector quantization, so a Transformer can predict sound the way it predicts text

Why Does This Exist?

Count what raw audio asks of you. One second sampled 16,000 times is a list of 16,000 numbers. Ten seconds is 160,000 numbers in a row, and an autoregressive model would have to predict them one at a time while its cost grows with the square of the sequence length. That is not a big model problem. It is a hopeless sequence length problem.

So you want what text already has: a short string of discrete symbols. Classical codecs do not give you that. MP3 is a hand-built perceptual pipeline tuned for ears and file size, and its bytes were never meant to be next-token targets. A neural audio codec is built for the opposite goal. It learns to squeeze sound into a short integer alphabet that a Transformer already knows how to model, then rebuild the wave from those integers afterward.

Two sentences of grounding, since everything below assumes them. An autoencoder is an encoder that squeezes its input into a compact code plus a decoder that rebuilds the input from that code, trained to make the rebuild match the original. A neural codec is that idea with one hard bottleneck in the middle: the code must be discrete tokens, not free decimals.

This page covers the compressor in the middle: squeeze time, snap to a codebook, refine the leftover, decode. The text version of the same trick lives in tokenization. The wave-to-text trip is automatic speech recognition, the architecture that hears a whole clip before writing lives in encoder-decoder ASR, and the text-to-wave trip that increasingly consumes codec tokens is text to speech.

Think of It Like This

The paint counter's swatch books

You bring a chipped paint flake to a hardware store and ask for that exact color. The clerk does not sell you liquid light. They open a swatch book of 1,024 printed colors, hold your flake against the pages, pick the closest swatch, and write down its number. That number is your first token.

But the match is slightly off, so they mix a correction for the difference between your flake and the swatch. Then they open a second, finer book, match the correction itself to its closest swatch, and write down that number too. Two small numbers now describe your color better than either book could alone, and the clerk can remix both later.

Where it stops: paint corrections are continuous and order-free, while a codec's stages are strictly ordered, coarse first and detail later. Shuffle the two numbers and the remix breaks, because the decoder reads stage two as a correction to stage one, never the other way around.

How It Actually Works

Three moves in order: squeeze time with an encoder, snap each frame to codebook entries with residual refinement, rebuild with a decoder. The symbols below, each defined where it first appears: zz is one frame's continuous vector, dd is how long that vector is, CC is a codebook (a list of example vectors), KK is how many entries the codebook holds, eke_k is entry number kk, ii is a chosen entry's integer id, rr is a leftover vector, and QQ is the number of refinement stages.

Squeeze time first

The encoder is a stack of convolutions that collapses chunks of waveform into one vector per chunk. Its stride SS says how many waveform samples fold into a single frame. With FF frames per second and a sample rate RR:

F=R/SF = R / S

Say the input is mono audio at R=16,000R = 16{,}000 samples per second and the encoder's total stride is S=320S = 320. Then F=16,000/320=50F = 16{,}000 / 320 = 50 frames per second, and each frame is one vector zz of length dd. Ten seconds is no longer 160,000 numbers to predict. It is 500 vectors, and the vectors are about to become integers.

Snap each frame to the nearest swatch

A codebook CC is a fixed list of KK example vectors e1,…,eKe_1, \ldots, e_K that the model learns during training. Quantization means replacing the continuous zz with the id of the closest entry:

i=arg⁡min⁡k∥z−ek∥2i = \arg\min_k \lVert z - e_k \rVert^2

In plain words: measure the squared distance from zz to every entry, keep the winner's integer id ii, throw the decimals away. The decoder holds the same book, so looking up eie_i from ii restores an approximation of the frame. A book with K=1,024K = 1{,}024 entries costs 10 bits per frame, because 210=1,0242^{10} = 1{,}024.

One book alone forces an ugly choice. A small book cannot cover the variety of real sound, so reconstructions come out crude. A giant book gives the downstream language model a gigantic vocabulary to learn, which is exactly the problem you were escaping. The fix is several small books in sequence.

Quantize the leftover, then the leftover's leftover

Residual vector quantization spends one small codebook per stage, where each stage cleans up the previous stage's error. With Q=2Q = 2 stages:

r1=z−ei1,z^=ei1+ei2r_1 = z - e_{i_1}, \qquad \hat{z} = e_{i_1} + e_{i_2}

In plain words: stage one picks the closest entry ei1e_{i_1} to zz and keeps its id i1i_1. The leftover r1r_1 is what stage one missed. Stage two quantizes that leftover with a second book, keeping id i2i_2. The rebuilt frame z^\hat{z} (read "z-hat", the estimate of zz) is the sum of the two chosen entries. Early stages carry the broad sound, later stages carry texture, and the ids stay small because no single book has to cover everything.

Worked example: one frame, two stages

Take one frame z=[0.75,0.25]z = [0.75, 0.25]. Book one holds A=[1.0,0.0]A = [1.0, 0.0] and B=[0.0,1.0]B = [0.0, 1.0]. Squared distances:

  • To AA: (0.75−1.0)2+(0.25−0.0)2=0.0625+0.0625=0.125(0.75 - 1.0)^2 + (0.25 - 0.0)^2 = 0.0625 + 0.0625 = 0.125.
  • To BB: (0.75−0.0)2+(0.25−1.0)2=0.5625+0.5625=1.125(0.75 - 0.0)^2 + (0.25 - 1.0)^2 = 0.5625 + 0.5625 = 1.125.

Stage one keeps id 00 (entry AA). The leftover is r1=[0.75−1.0, 0.25−0.0]=[−0.25,0.25]r_1 = [0.75 - 1.0,\ 0.25 - 0.0] = [-0.25, 0.25]. Book two holds C=[0.0,0.0]C = [0.0, 0.0] and D=[−0.25,0.25]D = [-0.25, 0.25]. Distances: to CC is 0.0625+0.0625=0.1250.0625 + 0.0625 = 0.125, to DD is 00. Stage two keeps id 11. The second leftover is r2=[0,0]r_2 = [0, 0], and the rebuild A+D=[0.75,0.25]A + D = [0.75, 0.25] matches the frame exactly. Real audio rarely lands this cleanly; leftovers shrink but seldom vanish.

Now the bitrate with these settings, both books at 1,0241{,}024 entries (10 bits each): 50 frames per second ×\times 2 stages ×\times 10 bits =1,000= 1{,}000 bits per second. That is the whole compression story in one line.

What the language model actually predicts

Each frame becomes QQ integers in coarse-to-fine order, so the model predicts a flat stream like i1,i2i_1, i_2 for frame one, then i1,i2i_1, i_2 for frame two. At 50 frames per second with 2 stages that is 100 tokens per second against 16,000 raw samples per second, about 160 times shorter. Because the stages are ordered, good generators predict or condition on the coarse ids before the fine ones rather than treating all ids as equals.

Name the trade-off before you use it. More stages buy fidelity and cost sequence length plus modeling effort; fewer stages are cheap to generate and sound it. And when your job is moving audio between people, calls, streaming, archives, reach for a plain perceptual codec instead. A neural codec earns its keep when its tokens feed generation or understanding. It assumes new audio resembles its training diet, and unfamiliar voices, instruments, or noise come back metallic.

Code

The snippet below replays the worked example: nearest-entry search, then the leftover pass. Pure Python, no dependencies, so you can run it anywhere.

Vector = list[float]

def dist2(a: Vector, b: Vector) -> float:    return sum((x - y) ** 2 for x, y in zip(a, b))

def nearest_id(z: Vector, book: list[Vector]) -> int:    return min(range(len(book)), key=lambda k: dist2(z, book[k]))

def rvq(z: Vector, books: list[list[Vector]]) -> tuple[list[int], Vector, Vector]:    ids: list[int] = []    leftover: Vector = list(z)    for book in books:        i = nearest_id(leftover, book)        ids.append(i)        entry = book[i]        leftover = [x - e for x, e in zip(leftover, entry)]    recon: Vector = [0.0 for _ in z]    for i, book in zip(ids, books):        recon = [x + e for x, e in zip(recon, book[i])]    return ids, recon, leftover

z: Vector = [0.75, 0.25]books: list[list[Vector]] = [[[1.0, 0.0], [0.0, 1.0]], [[0.0, 0.0], [-0.25, 0.25]]]ids, recon, leftover = rvq(z, books)print(ids)       # -> [0, 1]print(recon)     # -> [0.75, 0.25]print(leftover)  # -> [0.0, 0.0]print(50 * 2 * 10)  # -> 1000

The ids pick AA then DD, the rebuild matches the frame, the leftover hits zero, and the last line replays the bitrate: 50 frames, 2 stages, 10 bits each.

Watch Out For

Most of the codebook goes silent

Training sometimes lets a few popular entries do all the work while hundreds of codes sit unused. This is codebook collapse. The symptom is unmistakable: reconstructions turn metallic and bubbly, and a usage histogram shows a handful of ids carrying nearly every frame.

Do not fix it by training longer. Track code usage or perplexity per book during training, and revive dead entries by resetting them near active ones or by updating books with slow averages of the vectors assigned to them. If usage is still skewed at the end, the book is too big for the data, so shrink KK before blaming the decoder.

Fine tokens predicted without their coarse parents

The QQ ids per frame are not interchangeable slots. Stage two is a correction to stage one, so a generator that predicts all stages flat, or shuffles their order, produces audio that sounds clean while saying the wrong thing: stable texture over scrambled content.

The symptom is eerie, good acoustics with broken words or wandering pitch. Condition each stage on the earlier ones, through staged heads or a coarse-first delay pattern, and keep the on-disk order coarse to fine. If you subsample stages to save length, drop from the fine end only.

The Quick Version

  • A neural audio codec turns a waveform into a short string of integer tokens so a Transformer can generate sound with next-token prediction.
  • The encoder folds thousands of samples per second into tens of frame vectors per second; with stride 320, 16,000 samples become 50 frames.
  • Each frame is replaced by the id of its nearest codebook entry, and residual stages quantize the leftover error coarse to fine.
  • A 1,024-entry codebook costs 10 bits per pick, so 50 frames times 2 stages times 10 bits is 1,000 bits per second.
  • Generation must respect stage order: fine detail means nothing without the coarse sound it corrects.
  • For transport between people use a plain perceptual codec; neural tokens earn their keep when they feed generation or understanding.