Text-to-Speech (TTS)
Text to speech turns writing into sound in two passes: first it sketches how each sound should look over time, then it renders that sketch into a wave a speaker can play.
Why Does This Exist?
Written words carry no sound. The letters "hello" say nothing about how long the "eh" lasts, where the pitch rises, or what the voice sounds like. So speech has to be invented from text, not read out of it.
The obvious fix was gluing recordings together. Early systems stored tiny clips of syllables and pasted them in order, which is why they sounded flat. A syllable recorded in isolation carries the wrong pitch for the middle of a question, and no library of clips covers every name, number, and emphasis you will ever type.
Classical neural TTS answers by generating instead of pasting. It works in two stages: an acoustic model sketches how the utterance should look over time, and a vocoder renders that sketch into a playable wave. You will see both stages below, with one small fixture carried through the whole page.
Know where this page sits. The language model path, where a seconds long voice clip steers a token model into a new voice with no retraining, lives in speech synthesis. The reverse trip, sound back into words, is automatic speech recognition. The token machinery that path depends on is neural audio codecs. This page covers the classical pipeline: text in, fixed voice out.
A little grounding first, since the rest assumes it. A microphone measures air pressure thousands of times per second, so half a second of speech at 16 kHz is already 8,000 numbers. Sample rates, frames, and why models prefer a time frequency picture are covered in audio preprocessing.
Think of It Like This
A player piano roll and the piano
You hand a tune request to an operator. The operator punches holes in a long paper roll: each hole says which note plays and how long it lasts, and the position along the roll says when. The roll itself makes no sound. Then the piano reads the roll and hammers its strings, turning hole positions into music you can hear.
The pipeline maps part for part. Your text is the request. The acoustic model is the operator, and the mel spectrogram is the punched roll: a picture of which frequencies should sound at each moment. The vocoder is the piano, turning that picture into pressure numbers a speaker can play.
Where it stops: a piano roll replays exactly what was punched, but the acoustic model has to guess. It has never heard your sentence before, so it guesses the timing, the stress, and the tune from patterns in its training data. Guess wrong and the voice rushes, drags, or reads exciting news like a weather report.
How It Actually Works
Three moves: turn letters into timed sounds, turn timed sounds into a spectrogram picture, turn the picture into a wave. Each step changes the shape of the data, and the shapes have to line up.
From letters to sounds you can time
Raw text is a trap, because spelling is not pronunciation. "Read" rhymes with "seed" today and "bed" yesterday, and "$100" should be spoken as "one hundred dollars". So the pipeline normalizes first: numbers expand to words, abbreviations expand, and then a lexicon plus a small model converts graphemes (written letters) into a phoneme sequence , where each phoneme is one contrastive sound unit such as HH or EH, and is the count of phonemes in the sentence.
Then each phoneme gets a duration , measured in spectrogram frames, saying how many time slices that sound occupies. Fast speech uses small durations, stressed vowels use large ones. The total frame count is their sum, . This is where timing lives, and timing is most of what makes speech sound human.
From timed sounds to a mel spectrogram
The acoustic model reads the phonemes and their durations and writes a mel spectrogram , a matrix in rows by columns. Here is the number of mel bands, frequency bins warped so low pitches get more detail the way your ear does, and is the frame count from above. Each entry is the energy in band at time slice : bright where the voice is loud, dark where it is quiet.
Older models set durations with attention, letting the decoder look back at the phoneme string while writing each frame. That sounds natural when it works and mumbles when it does not, because nothing forces the gaze to move forward. Newer setups predict explicitly and stretch each phoneme to its allotted frames, which is steadier and easier to control. Either way, the output shape is fixed: times numbers, and if your tensor is any other shape you have a bug.
From the picture to a playable wave
A spectrogram is an image, and speakers cannot play images. The vocoder converts into a waveform , where is the air pressure at sample and runs from to , with equal to seconds times the sample rate . This step invents fine detail the picture threw away, because one spectrogram frame summarizes hundreds of samples.
That is the core trade off. Sample by sample models predict each from the ones before it, which sounds clean and runs slowly. Parallel models write whole chunks at once, which runs in real time and needs careful training to avoid buzzing or hiss. Both assume the training voice matches the deployment voice: one speaker, one microphone, one room. Change the speaker and you retrain or switch paths.
So pick on purpose. When you need exact phoneme timing, a fixed voice, and a small latency budget, this pipeline usually wins. When you need a brand new voice from seconds of audio, skip it and read speech synthesis instead, which pays the autoregressive price to clone from a clip.
Worked example with small numbers
Say the word "hello" lasts half a second, with sample rate samples per second, hop length samples per frame, and mel bands. The hop is the step between frames, so each frame covers seconds, or 12.5 ms.
Split the 40 frames across four phonemes the way a duration predictor might: HH gets 6, EH gets 8, L gets 8, OW gets 18. Check: . The stressed OW owns nearly half the word, which matches how you say it.
The vocoder then turns those 3,200 picture values into 8,000 pressure numbers, about samples per mel value. That ratio is the whole job of the vocoder in one number: invent plausible fine detail at two and a half times the density of the sketch.
Code
This snippet recomputes the fixture above from scratch: frames, picture values, samples, and the expansion ratio the vocoder must deliver. It uses only the standard library.
def frame_budget(seconds: float, sample_rate: int = 16000, hop: int = 200, mel_bands: int = 80) -> tuple[int, int, int]: """Return (frames, mel values, waveform samples) for `seconds` of speech.""" total = int(seconds * sample_rate) # waveform samples N_s frames = total // hop # time slices T return frames, mel_bands * frames, total
frames, mel_values, samples = frame_budget(0.5)print(frames) # -> 40print(mel_values) # -> 3200print(samples) # -> 8000print(samples / mel_values) # -> 2.5print(6 + 8 + 8 + 18) # -> 40Forty frames, 3,200 picture values, 8,000 samples, and the duration split sums back to 40. If you change the hop to 256, the frame count stops dividing evenly, which is why toy fixtures pick round numbers and production code pads the remainder.
Watch Out For
Attention that skips or loops on long sentences
You feed in a three clause sentence and out comes the first clause twice with the middle clause missing. Short test lines sounded fine, so you blame the vocoder. The symptom points upstream: the acoustic model's attention gaze jumped backward or leapt forward, and the spectrogram it wrote already contains the repeat or the gap before the vocoder ever runs.
Fix it at the source. Plot the attention alignment of phonemes against frames and look for a clean diagonal; jumps and blobs mark the fault. Prefer explicit duration prediction with a left to right constraint for long inputs, keep one synthesis call to one or two sentences, and split paragraphs before synthesizing rather than after.
Raw text fed straight into the model
"Meet Dr. Smith at 12 St. with $100" comes out as "dexter smith at twelve saint with dollar one hundred hundred" and you assume the model is weak. It never saw the right input. "$100" is four characters with a dozen valid readings, and "St." can be "street" or "saint" depending on context.
Normalize before you synthesize. Expand numbers, currency, dates, and abbreviations with a text normalizer, then convert to phonemes with a lexicon, and only then run the acoustic model. Keep a small test list of nasty inputs (currency, times, abbreviations, proper names) and listen to all of them after any pipeline change, because this failure is silent in every loss curve.
The Quick Version
- Text to speech generates sound in two stages: an acoustic model writes a mel spectrogram picture, then a vocoder renders that picture into a playable wave.
- Raw text is normalized and converted to phonemes first, because spelling never tells you directly how a word sounds.
- A mel spectrogram has frequency rows by time columns, and is the sum of the per phoneme durations.
- Your toy fixture: half a second at 16 kHz with hop 200 gives 40 frames, an 80 by 40 picture with 3,200 values, and 8,000 waveform samples.
- Sample by sample vocoders sound clean and run slowly, while parallel vocoders run in real time and need care to avoid artifacts.
- For a fixed voice with exact timing needs, use this pipeline; for cloning a new voice from seconds of audio, use the token language model path instead.