Neural Audio Codecs

Why Does This Exist?

If you want to generate high-fidelity audio (like a symphony or a human voice), you have to generate a staggering amount of data. Standard CD-quality audio requires generating 44,100 numbers (samples) for every single second of sound. If you try to use a Transformer to generate a 3-minute song, it would have to predict 8 million numbers in a row. This sequence is far too long; the Transformer would run out of memory instantly.

To fix this, we need to compress the audio into a much shorter sequence. We already have audio codecs like MP3, but MP3s are designed for human ears, not neural networks. MP3 compression is mathematically messy and hard for an AI to learn from. Neural Audio Codecs (like Meta's EnCodec or Google's SoundStream) were invented to use AI to compress audio into a clean, discrete "alphabet" of sounds. By shrinking 44,100 numbers down to just 75 "tokens" per second, they made it possible for standard Large Language Models (LLMs) to generate music and speech.

Think of It Like This

How It Actually Works

Neural Codecs are essentially specialized autoencoders. They consist of an Encoder, a Quantizer, and a Decoder.

1. The Encoder

The raw audio wave is fed into a Convolutional Neural Network (CNN). The CNN looks at a chunk of audio and compresses it down into a continuous, dense mathematical vector (an embedding).

2. Vector Quantization (The Codebook)

This is the most important step. A continuous vector is infinite; it can be any decimal number. An LLM needs a finite vocabulary (like the 26 letters of the alphabet). We use a Codebook—a predefined list of, say, 1,024 specific sound vectors. We take the continuous vector from the Encoder, find the vector in the Codebook that looks the most similar to it, and throw away the original vector. We just keep the ID number of the codebook entry (e.g., ID #42). Now, instead of 44,100 continuous numbers, one second of audio is represented by just 75 discrete integer IDs.

3. Residual Vector Quantization (RVQ)

There is a catch. If your codebook only has 1,024 sounds, you can't possibly capture all the subtle details of a symphony. It will sound robotic. But if you make the codebook have a billion sounds, the LLM won't be able to learn it. RVQ solves this by using multiple codebooks in a hierarchy.

• Codebook 1 captures the main sound (e.g., a guitar strum).
• Codebook 2 captures the error (the subtle squeak of the guitar string that Codebook 1 missed).
• Codebook 3 captures the echo of the room. This allows the audio to be perfectly reconstructed using just a few small codebooks.

4. Audio Generation via Language Models

Once the neural codec is trained, the magic happens. You take thousands of hours of music, compress them into these discrete tokens using the Encoder, and train a standard GPT model on them. You train the GPT to simply predict the next token. To generate a new song, you ask the GPT to output a sequence of tokens, and you pass those tokens to the Codec's Decoder to turn them back into a beautiful, continuous sound wave. This is the exact architecture behind models like Suno, Udio, and AudioLM.

Show Me the Code

This pseudocode demonstrates how Residual Vector Quantization uses multiple codebooks to capture the fine details of audio without requiring a massive vocabulary.

1import torch2
3def residual_vector_quantization(continuous_vector, codebooks):4    """5    Compresses a continuous audio vector into a sequence of discrete IDs using RVQ.6    """7    # continuous_vector shape: (Embedding_Dim,)8    # codebooks: A list of 4 codebooks, each with 1024 vectors9    10    residual_error = continuous_vector11    quantized_ids = []12    13    # We pass the error through multiple codebooks to capture finer and finer details14    for codebook in codebooks:15        # 1. Find the closest vector in the current codebook to the residual error16        closest_id, closest_vector = codebook.find_closest_match(residual_error)17        quantized_ids.append(closest_id)18        19        # 2. Calculate what this codebook MISSED20        # We will pass this error to the next codebook to refine21        residual_error = residual_error - closest_vector22        23    # We return a list of 4 discrete IDs (e.g., [42, 800, 15, 99])24    # The LLM will be trained to generate these exact IDs.25    return quantized_ids

Watch Out For

The Quick Version

• Generating raw audio directly requires predicting millions of samples, which is too slow and memory-intensive for standard neural networks.
• Neural Audio Codecs solve this by compressing the audio into a slow sequence of discrete, integer "tokens."
• They use Vector Quantization (a codebook) to map complex sounds to simple ID numbers, essentially creating an "alphabet" of audio.
• Residual Vector Quantization (RVQ) uses multiple layers of codebooks to capture both the broad sounds and the fine acoustic details.
• Once audio is compressed into tokens, you can train a standard text-based LLM (like GPT-4) to compose music or generate speech simply by predicting the next token.

What to Read Next

• Read Speech Synthesis to see how we combine LLMs and Neural Codecs to clone a human voice perfectly from just a 3-second audio clip.
• Read Autoregressive Image Generation to see how this exact same quantization trick (VQ-VAE) is used to generate images token-by-token.