Skip to content
AI360XpertBeta

Text-to-Speech (TTS)

Text to speech turns writing into sound in two passes: first it sketches how each sound should look over time, then it renders that sketch into a wave a speaker can play.

Text becomes an 80 band mel spectrogram with 40 frames, then a vocoder turns those 3,200 values into 8,000 waveform samples you can play
Text becomes an 80 band mel spectrogram with 40 frames, then a vocoder turns those 3,200 values into 8,000 waveform samples you can play

Why Does This Exist?

Written words carry no sound. The letters "hello" say nothing about how long the "eh" lasts, where the pitch rises, or what the voice sounds like. So speech has to be invented from text, not read out of it.

The obvious fix was gluing recordings together. Early systems stored tiny clips of syllables and pasted them in order, which is why they sounded flat. A syllable recorded in isolation carries the wrong pitch for the middle of a question, and no library of clips covers every name, number, and emphasis you will ever type.

Classical neural TTS answers by generating instead of pasting. It works in two stages: an acoustic model sketches how the utterance should look over time, and a vocoder renders that sketch into a playable wave. You will see both stages below, with one small fixture carried through the whole page.

Know where this page sits. The language model path, where a seconds long voice clip steers a token model into a new voice with no retraining, lives in speech synthesis. The reverse trip, sound back into words, is automatic speech recognition. The token machinery that path depends on is neural audio codecs. This page covers the classical pipeline: text in, fixed voice out.

A little grounding first, since the rest assumes it. A microphone measures air pressure thousands of times per second, so half a second of speech at 16 kHz is already 8,000 numbers. Sample rates, frames, and why models prefer a time frequency picture are covered in audio preprocessing.

Think of It Like This

A player piano roll and the piano

You hand a tune request to an operator. The operator punches holes in a long paper roll: each hole says which note plays and how long it lasts, and the position along the roll says when. The roll itself makes no sound. Then the piano reads the roll and hammers its strings, turning hole positions into music you can hear.

The pipeline maps part for part. Your text is the request. The acoustic model is the operator, and the mel spectrogram is the punched roll: a picture of which frequencies should sound at each moment. The vocoder is the piano, turning that picture into pressure numbers a speaker can play.

Where it stops: a piano roll replays exactly what was punched, but the acoustic model has to guess. It has never heard your sentence before, so it guesses the timing, the stress, and the tune from patterns in its training data. Guess wrong and the voice rushes, drags, or reads exciting news like a weather report.

How It Actually Works

Three moves: turn letters into timed sounds, turn timed sounds into a spectrogram picture, turn the picture into a wave. Each step changes the shape of the data, and the shapes have to line up.

From letters to sounds you can time

Raw text is a trap, because spelling is not pronunciation. "Read" rhymes with "seed" today and "bed" yesterday, and "$100" should be spoken as "one hundred dollars". So the pipeline normalizes first: numbers expand to words, abbreviations expand, and then a lexicon plus a small model converts graphemes (written letters) into a phoneme sequence pp, where each phoneme pip_i is one contrastive sound unit such as HH or EH, and NN is the count of phonemes in the sentence.

Then each phoneme gets a duration did_i, measured in spectrogram frames, saying how many time slices that sound occupies. Fast speech uses small durations, stressed vowels use large ones. The total frame count is their sum, T=d1+d2+…+dNT = d_1 + d_2 + \ldots + d_N. This is where timing lives, and timing is most of what makes speech sound human.

From timed sounds to a mel spectrogram

The acoustic model reads the phonemes and their durations and writes a mel spectrogram MM, a matrix in FF rows by TT columns. Here FF is the number of mel bands, frequency bins warped so low pitches get more detail the way your ear does, and TT is the frame count from above. Each entry M[f,t]M[f, t] is the energy in band ff at time slice tt: bright where the voice is loud, dark where it is quiet.

Older models set durations with attention, letting the decoder look back at the phoneme string while writing each frame. That sounds natural when it works and mumbles when it does not, because nothing forces the gaze to move forward. Newer setups predict did_i explicitly and stretch each phoneme to its allotted frames, which is steadier and easier to control. Either way, the output shape is fixed: FF times TT numbers, and if your tensor is any other shape you have a bug.

From the picture to a playable wave

A spectrogram is an image, and speakers cannot play images. The vocoder converts MM into a waveform x[n]x[n], where x[n]x[n] is the air pressure at sample nn and nn runs from 11 to NsN_s, with NsN_s equal to seconds times the sample rate fsf_s. This step invents fine detail the picture threw away, because one spectrogram frame summarizes hundreds of samples.

That is the core trade off. Sample by sample models predict each x[n]x[n] from the ones before it, which sounds clean and runs slowly. Parallel models write whole chunks at once, which runs in real time and needs careful training to avoid buzzing or hiss. Both assume the training voice matches the deployment voice: one speaker, one microphone, one room. Change the speaker and you retrain or switch paths.

So pick on purpose. When you need exact phoneme timing, a fixed voice, and a small latency budget, this pipeline usually wins. When you need a brand new voice from seconds of audio, skip it and read speech synthesis instead, which pays the autoregressive price to clone from a clip.

Worked example with small numbers

Say the word "hello" lasts half a second, with sample rate fs=16,000f_s = 16{,}000 samples per second, hop length H=200H = 200 samples per frame, and F=80F = 80 mel bands. The hop HH is the step between frames, so each frame covers 200/16,000=0.0125200 / 16{,}000 = 0.0125 seconds, or 12.5 ms.

Ns=0.5×16,000=8,000 samples, and T=8,000/200=40 framesN_s = 0.5 \times 16{,}000 = 8{,}000 \text{ samples, and } T = 8{,}000 / 200 = 40 \text{ frames}

M holds F×T=80×40=3,200 numbersM \text{ holds } F \times T = 80 \times 40 = 3{,}200 \text{ numbers}

Split the 40 frames across four phonemes the way a duration predictor might: HH gets 6, EH gets 8, L gets 8, OW gets 18. Check: 6+8+8+18=406 + 8 + 8 + 18 = 40. The stressed OW owns nearly half the word, which matches how you say it.

The vocoder then turns those 3,200 picture values into 8,000 pressure numbers, about 8,000/3,200=2.58{,}000 / 3{,}200 = 2.5 samples per mel value. That ratio is the whole job of the vocoder in one number: invent plausible fine detail at two and a half times the density of the sketch.

Code

This snippet recomputes the fixture above from scratch: frames, picture values, samples, and the expansion ratio the vocoder must deliver. It uses only the standard library.

def frame_budget(seconds: float, sample_rate: int = 16000, hop: int = 200, mel_bands: int = 80) -> tuple[int, int, int]:    """Return (frames, mel values, waveform samples) for `seconds` of speech."""    total = int(seconds * sample_rate)  # waveform samples N_s    frames = total // hop  # time slices T    return frames, mel_bands * frames, total

frames, mel_values, samples = frame_budget(0.5)print(frames)  # -> 40print(mel_values)  # -> 3200print(samples)  # -> 8000print(samples / mel_values)  # -> 2.5print(6 + 8 + 8 + 18)  # -> 40

Forty frames, 3,200 picture values, 8,000 samples, and the duration split sums back to 40. If you change the hop to 256, the frame count stops dividing evenly, which is why toy fixtures pick round numbers and production code pads the remainder.

Watch Out For

Attention that skips or loops on long sentences

You feed in a three clause sentence and out comes the first clause twice with the middle clause missing. Short test lines sounded fine, so you blame the vocoder. The symptom points upstream: the acoustic model's attention gaze jumped backward or leapt forward, and the spectrogram it wrote already contains the repeat or the gap before the vocoder ever runs.

Fix it at the source. Plot the attention alignment of phonemes against frames and look for a clean diagonal; jumps and blobs mark the fault. Prefer explicit duration prediction with a left to right constraint for long inputs, keep one synthesis call to one or two sentences, and split paragraphs before synthesizing rather than after.

Raw text fed straight into the model

"Meet Dr. Smith at 12 St. with $100" comes out as "dexter smith at twelve saint with dollar one hundred hundred" and you assume the model is weak. It never saw the right input. "$100" is four characters with a dozen valid readings, and "St." can be "street" or "saint" depending on context.

Normalize before you synthesize. Expand numbers, currency, dates, and abbreviations with a text normalizer, then convert to phonemes with a lexicon, and only then run the acoustic model. Keep a small test list of nasty inputs (currency, times, abbreviations, proper names) and listen to all of them after any pipeline change, because this failure is silent in every loss curve.

The Quick Version

  • Text to speech generates sound in two stages: an acoustic model writes a mel spectrogram picture, then a vocoder renders that picture into a playable wave.
  • Raw text is normalized and converted to phonemes first, because spelling never tells you directly how a word sounds.
  • A mel spectrogram has FF frequency rows by TT time columns, and TT is the sum of the per phoneme durations.
  • Your toy fixture: half a second at 16 kHz with hop 200 gives 40 frames, an 80 by 40 picture with 3,200 values, and 8,000 waveform samples.
  • Sample by sample vocoders sound clean and run slowly, while parallel vocoders run in real time and need care to avoid artifacts.
  • For a fixed voice with exact timing needs, use this pipeline; for cloning a new voice from seconds of audio, use the token language model path instead.