Speech Synthesis (Text-to-Speech)
Modern Text-to-Speech uses a Language Model trained to 'speak'. By feeding the model a 3-second recording of someone's voice as a prompt, the model will 'continue' speaking the target text using that exact same voice, accent, and emotion.
Why Does This Exist?
For decades, Text-to-Speech (TTS) systems sounded like robots. Early systems literally pasted pre-recorded syllables together. Later systems (like Tacotron or WaveNet) sounded much more human, but they required hours of high-quality studio audio from a single voice actor to train. If you wanted the AI to speak in your voice, you had to spend 5 hours recording yourself reading a script in a quiet room, and then pay for hours of GPU time to train a custom model.
The current generation of TTS (popularized by models like Microsoft's VALL-E and ElevenLabs) completely changed the paradigm. They achieved Zero-Shot Voice Cloning. You only need to provide a single 3-second audio clip of someone speaking, and the AI can instantly read any text in that exact voice, complete with their accent, cadence, and even the background noise of the room.
Think of It Like This
The Master Impressionist
Imagine a world-class impressionist comedian.
- Traditional TTS: You hand the comedian a 5-hour audiobook of Morgan Freeman, lock them in a room for a week to study it, and then ask them to do a Morgan Freeman impression.
- Zero-Shot TTS: You walk up to the comedian on the street, play them a 3-second voicemail from your grandmother, and hand them a newspaper. The comedian instantly reads the entire newspaper sounding exactly like your grandmother, perfectly matching her tone of voice and her slight cough.
How It Actually Works
The breakthrough of Zero-Shot TTS was realizing that you can just treat audio generation exactly like text generation.
1. The Audio Codec (The Alphabet)
As discussed in Neural Audio Codecs, we can't train an LLM on raw audio waves. We first use a model like EnCodec to compress all audio into a vocabulary of discrete "acoustic tokens." Now, audio is just text. A dog bark is [42, 19]. The word "Hello" is [800, 31, 99].
2. The Language Model (VALL-E)
Researchers trained a massive Transformer Language Model (like GPT) on 60,000 hours of people speaking English. But they didn't just train it on the audio; they trained it jointly on the text transcript and the acoustic tokens. The training objective is simple: given a text sequence (e.g., "Hello world") and a 3-second audio prompt of someone saying "Hi there," predict the acoustic tokens for "Hello world" in that exact same acoustic style.
3. In-Context Learning
Because the Transformer is so massive, it exhibits In-Context Learning (just like ChatGPT). It doesn't need to update its weights to learn a new voice. When you feed it a 3-second audio prompt of your grandmother, the Self-Attention mechanism analyzes her acoustic tokens. When the model begins generating the new text, it constantly references your grandmother's tokens to ensure the new tokens have the same pitch, timbre, and acoustic environment.
4. Acoustic Decoding
Once the LLM finishes predicting the sequence of acoustic tokens for the target text, we simply pass those discrete tokens back through the Neural Audio Codec (the Decoder) to transform them back into a high-fidelity, continuous audio waveform.
Show Me the Code
This pseudocode shows the incredibly simple, text-like inference loop of a modern audio language model.
import torch
def generate_speech(audio_lm, codec_decoder, target_text, voice_prompt_audio): """ Clones a voice and generates new speech using an Audio Language Model. """ # 1. Convert the 3-second voice prompt into discrete acoustic tokens # E.g., [400, 12, 99, ...] prompt_tokens = codec_encoder.encode(voice_prompt_audio) # 2. Tokenize the text we want the AI to say text_tokens = text_tokenizer.encode(target_text) # 3. Create the Context Window for the LLM # We feed it the text to say, and the acoustic style to say it in context = torch.cat([text_tokens, prompt_tokens]) # 4. Autoregressively generate the new acoustic tokens generated_acoustic_tokens = [] for _ in range(MAX_AUDIO_LENGTH): # The LLM looks at the text and the voice prompt to predict the next sound logits = audio_lm(context, generated_acoustic_tokens) next_token = sample(logits) generated_acoustic_tokens.append(next_token) if next_token == END_OF_AUDIO: break # 5. Decode the discrete tokens back into a continuous WAV file final_audio_waveform = codec_decoder.decode(generated_acoustic_tokens) return final_audio_waveformWatch Out For
The Emotion Bottleneck
While Zero-Shot TTS perfectly captures the timbre (the "sound") of a voice, it struggles with intense emotional changes. Because it conditions heavily on the 3-second prompt, if you provide a prompt of someone whispering, and ask the model to read a script where the character is screaming in anger, the model will often just aggressively whisper the text. To get screaming, you usually have to provide a prompt of the person already screaming. Advanced systems are now researching ways to inject separate "emotion tokens" to control this manually.
The Quick Version
- Traditional Text-to-Speech required training a custom model on hours of clean studio audio for every new voice.
- Modern Zero-Shot TTS (like VALL-E) uses Neural Audio Codecs to treat audio exactly like text tokens.
- It uses a Large Language Model trained jointly on text transcripts and acoustic tokens.
- By providing a 3-second audio clip as a "prompt," the LLM uses in-context learning to mimic the pitch, tone, and environment of the speaker perfectly without any fine-tuning.
- This allows anyone to clone a voice instantly, posing both massive opportunities for accessibility and massive risks for audio deepfakes.
What to Read Next
- Read Neural Audio Codecs to review exactly how the continuous audio wave is compressed into the discrete tokens that the LLM predicts.
- Read Automatic Speech Recognition to see the inverse process: turning raw audio back into discrete English text.