Skip to content
AI360XpertBeta

Speech Synthesis (Text-to-Speech)

Type a sentence, attach a seconds-long clip of a voice, and the model reads your words in that voice. Underneath, the audio is just tokens, so speaking is next-token prediction.

A seconds-long voice clip gives an audio language model the voice to speak in, so it reads new text as that speaker.
A seconds-long voice clip gives an audio language model the voice to speak in, so it reads new text as that speaker.

Why Does This Exist?

Every new voice used to cost hours of clean studio recording plus its own trained model. Wanted the system to speak like you? Book studio time, read scripts aloud for an afternoon, then train a model just for your voice. One voice, one dataset, one bill. That priced out everything except a handful of professional voices.

Zero-shot speech synthesis drops that price to nearly nothing. One model serves every voice, and a seconds-long clip of someone talking tells it which voice to use. No retraining per speaker. You hand it the words and a sample, and it reads the words as that speaker, pacing and room tone included.

Know where this page sits. The classical two-stage pipeline, an acoustic model writing a spectrogram plus a vocoder turning it into sound, lives on the text-to-speech page. How raw audio becomes discrete tokens lives in neural audio codecs. The reverse trip, speech back into text, is automatic speech recognition. This page covers the language-model path: tokens in, a voice clip as prompt, speech out.

Think of It Like This

A rubber stamp and an ink pad

Your text is the shape carved into the stamp. The voice clip is the ink pad, carrying one color. The model is the hand that presses. Every print carries the stamp's words in the pad's color, and you can stamp brand-new sentences without buying new ink.

Where it stops: ink copies exactly, but the model invents. It has to produce sounds the clip never contained, in words the speaker never said. So it can mispronounce a name, slide toward a generic average voice on long passages, or carry the room noise of the pad into every print.

How It Actually Works

Three moves: turn sound into tokens, put a voice clip next to your text, then predict audio tokens one batch at a time and decode them back into sound.

Turn sound into tokens first

A waveform xx is just air pressure measured thousands of times per second, far too long for a Transformer to write out directly. A neural audio codec chops xx into short slices called frames, tens of them per second, and replaces each frame with one or more token ids drawn from learned lists called codebooks. A codebook is the codec's fixed menu of prototype sounds; a token id is the index of the closest match. Sound becomes a short integer string, and language models already know exactly what to do with those.

Speak by predicting the next token

The model reads two things side by side: text tokens tt saying what to speak, and prompt tokens pp saying how it should sound. Then it predicts the next acoustic token, appends it, and repeats. The prompt is never trained on. It sits in context and steers attention toward that speaker's pitch and pace, which is in-context learning applied to voices.

Each step ends in sampling. Greedy takes the top score, temperature spreads the bets, top-p keeps the smallest set that covers most of the probability. Those rules live in decoding strategies. When the token stream is done, the codec decoder turns it back into a waveform you can play.

Worked example with small numbers

Take a toy setup so the arithmetic stays visible. Say you have 2 seconds of audio sampled 16,000 times per second. That is 2×16,000=32,0002 \times 16{,}000 = 32{,}000 pressure numbers. Your codec runs at 50 frames per second with 2 codebooks per frame.

2×50=100 frames, and 100×2=200 prompt tokens2 \times 50 = 100 \text{ frames, and } 100 \times 2 = 200 \text{ prompt tokens}

Your sentence "hi" becomes 1 text token in this toy, so the model's context holds 1+200=2011 + 200 = 201 tokens. Speaking 2 fresh seconds needs 200 new tokens, one frame pair at a time. Notice the compression: 32,000 numbers in, 200 tokens to model.

Now one sampling step. The model scores 3 candidate next tokens with logits l=[2.0,1.0,0.0]l = [2.0, 1.0, 0.0], where each logit is a raw score for one candidate, before probabilities. Softmax turns scores into probabilities that sum to 1:

P(next=i)=eli∑jeljP(\text{next} = i) = \frac{e^{l_i}}{\sum_j e^{l_j}}

Subtract the max first (same answer, no overflow): [0,−1,−2][0, -1, -2]. Exponentiate: [1.000,0.368,0.135][1.000, 0.368, 0.135], because e−1≈0.368e^{-1} \approx 0.368 and e−2≈0.135e^{-2} \approx 0.135. Sum: 1.5031.503. Divide: [1/1.503, 0.368/1.503, 0.135/1.503]≈[0.665, 0.245, 0.090][1/1.503,\ 0.368/1.503,\ 0.135/1.503] \approx [0.665,\ 0.245,\ 0.090]. Greedy decoding picks the first candidate. And if you divide the logits by a temperature of 2 before the same steps, you get about [0.506, 0.307, 0.186][0.506,\ 0.307,\ 0.186]: flatter, riskier, more varied.

Two families, one trade-off

Left-to-right token models mimic a voice closely but speak slowly and can drift on long passages, since every token depends on the last. Parallel and flow-based models write many frames at once: faster, steadier timing, weaker mimicry of fine quirks. That is the trade-off, fidelity against speed and control.

So pick on purpose. When you need exact phoneme timing or a tiny latency budget, the classical pipeline on the text-to-speech page usually wins. When you need a new voice from seconds of audio, you pay the autoregressive price. And the whole trick rests on one assumption: the clip shows the voice you want, in the right language, the right energy, and a clean room. Break that and the output breaks with it.

Code

This snippet computes the two numbers the worked example derived by hand: the token budget for a clip, and one softmax sampling step. It uses only the standard library.

import math

def prompt_tokens(seconds: float, frames_per_second: int = 50, codebooks: int = 2) -> int:    """How many acoustic tokens a voice clip of `seconds` becomes."""    frames = int(seconds * frames_per_second)  # one frame per time slice    return frames * codebooks  # each frame carries one id per codebook

def softmax(logits: list[float]) -> list[float]:    """Turn raw scores into probabilities that sum to 1."""    shifted = [x - max(logits) for x in logits]  # subtract max: same answer, no overflow    exps = [math.exp(x) for x in shifted]    return [e / sum(exps) for e in exps]

print(prompt_tokens(2.0))  # -> 200print([round(p, 3) for p in softmax([2.0, 1.0, 0.0])])  # -> [0.665, 0.245, 0.09]

Two seconds in, 200 tokens to model, and the sampler agrees with the hand computation at [0.665,0.245,0.090][0.665, 0.245, 0.090]. Greedy takes index 0.

Watch Out For

A flat prompt reads everything flat

You hand the model a calm office-hours clip, then ask for an urgent ad read. Out comes your urgent script read calmly, and no sampling tweak fixes it. The model copies the clip's energy because energy is most of what the prompt carries.

Fix it at the source. Record the prompt in the target energy: an excited read needs an excited-sounding clip. Never expect whisper in and shouting out. If the system offers separate style or emotion controls, set those instead of fighting the prompt.

The room comes along with the voice

Generated speech carries the prompt's reverb, hiss, or background music, and listeners hear the bathroom in every sentence. That is not a rendering bug. The codec faithfully tokenized the room along with the vocal cords, so the model learned noise as part of the voice.

Prompt with a dry clip. Use a close-mic recording with no music behind it. If only noisy audio exists, denoise the clip before prompting rather than scrubbing the output after, because the damage happens at tokenization time.

The Quick Version

  • Speech synthesis turns text into speech by first turning sound into discrete tokens, then predicting tokens the way a language model predicts words.
  • A seconds-long voice clip sits in context and steers the output, so one model speaks in any prompted voice with no retraining.
  • Your toy budget: 2 seconds at 50 frames per second with 2 codebooks is 200 tokens in and 200 tokens out.
  • Each step samples from a softmax over next-token scores. Greedy takes the top score and temperature spreads the bets.
  • Left-to-right models mimic best and run slowest, while parallel models run fast with weaker mimicry and steadier timing.
  • The clip is the ceiling. Wrong energy, wrong language, or a noisy room in means the same out.