Whisper Architecture
Whisper listens to a whole 30-second clip once, then writes the words one by one. A few short tokens up front tell it whether to transcribe or translate to English.
Why Does This Exist?
You get a voice note in Spanish, a meeting recording in English, and a voicemail where the speaker switches halfway. You want text for all three, plus an English version of the Spanish one. The old answer was a chain: detect the language, transcribe with one model, then translate with another. Each step added its own errors, and a wrong language guess poisoned everything after it.
That chain also had no good way to share what it learned. A model that transcribes English well knew nothing about translating French, even though both jobs start from the same sounds. Teams kept one model per language pair and paid for each one separately.
Whisper replaces the chain with one encoder-decoder Transformer that reads sound as a picture and writes text one token at a time. A short prefix of task tokens tells the same decoder whether to copy the words or turn them into English. You do not swap models. You swap four tokens.
This page covers that Whisper loop: how audio becomes a mel picture, what the encoder keeps, how the tokens steer the decoder, and how cross-attention aims each word at its sound. The frame pipeline it builds on lives in automatic speech recognition. The general listen first, spell later skeleton has its own page in encoder-decoder ASR. The search that picks the final sentence lives in beam search.
One grounding note, since the rest assumes it. Sound enters as frames: short overlapping slices summarized as energies per pitch band. If frames, spectra, or the blank collapse feel new, read the ASR page first, then come back.
Think of It Like This
A stenographer with colored cover sheets
You hire a stenographer who knows many languages. You hand her a tape plus a colored cover sheet. Blue means type exactly what you hear. Yellow means type it in English. She listens to the whole segment first, then types sentence by sentence, rewinding the tape to check each fuzzy name before she commits.
The full listen is the encoder. Typing sentence by sentence is the decoder. Rewinding to check is cross-attention. The cover sheet is the task prefix: same person, same ears, different job because the sheet says so.
Where it stops: your stenographer understands the trial and will tell you when a passage was inaudible. Whisper understands nothing. It matches sound patterns to text patterns, so it types a wrong word with full confidence and will invent polite closing lines over pure silence rather than admit it heard nothing.
How It Actually Works
Call the sampled wave , where is air pressure as a number and counts samples from the start. Whisper never reads directly. It repictures the wave, reads the picture once, then writes text while looking back at that reading.
From wave to picture
Cut 30 seconds of audio into overlapping slices. The standard slice is 25 milliseconds wide with a new slice every 10 milliseconds. At 16,000 samples a second, one slice holds numbers and the hop is numbers.
Count the slices, call that count . Thirty seconds is samples, so , about 3,000 frames. For each frame, measure energy in 80 pitch bands spaced for human hearing and compress loudness with a logarithm. Stack the columns and you hold an log-Mel spectrogram: time along one axis, pitch along the other. That picture is the encoder input, not the raw wave.
The encoder reads once
The encoder is a Transformer stack that takes the whole picture before a single word is written. Write its output as , where each is a -dimensional vector and is just the width of the model. Because every is built with access to past and future frames, carries what happened at frame 10 and frame 90. A mumbled word at the start gets represented together with the later words that disambiguate it.
That full read is the accuracy win and the latency price. Nothing is emitted until the 30 seconds are encoded, which is why this shape fits recorded clips and prepared captions and fails live captioning where words must appear mid sentence.
Task tokens steer the decoder
The decoder is an autoregressive language model: it writes token after seeing the earlier tokens through , one step at a time. Before it writes anything, you feed it a fixed prefix. A Spanish to English request looks like this:
<|startoftranscript|> <|es|> <|translate|> <|notimestamps|>In plain words: start marker, then language tag saying the audio is Spanish, then job tag saying translate rather than transcribe, then format tag saying plain text without timestamps. Swap <|translate|> for <|transcribe|> and the same weights copy Spanish instead. Swap <|es|> for the detect tag and the model writes its own language guess first. The tokens are ordinary vocabulary items with learned embeddings. Training showed the model millions of such prefixes, so the prefix sets the output distribution from step zero.
To pick token , the decoder builds a query from its state , a summary of what it has written so far. It scores that query against every encoder vector , turns the scores into weights that are positive and sum to 1, and reads the weighted average . Call the context: the slice of audio this word listens to. The next token distribution is , where is a learned output matrix and means the two vectors joined. In short: self-attention remembers the text so far, cross-attention points at the sound that matters now.
Worked example: one picture size plus one token choice
Keep the picture math from above: 30 seconds gives about 2,998 frames, so the encoder input is . That is the number drawn in the diagram: 80 pitch rows, roughly 3,000 time columns, read once.
Now one decoding step. Say the decoder scores three candidate next tokens with logits , where a logit is just an unnormalized score. Exponentiate: , , , for a sum of about . Divide:
The three numbers sum to . With prefix <|es|> <|translate|>, the top token spells the English piece, say hello, with weight . With prefix <|es|> <|transcribe|>, the same audio state favors the Spanish spelling instead. Same encoder reading, same scoring rule, different first tokens because the prefix moved the query. The diagram draws these exact weights beside the output word.
The honest trade follows. Future context helps accuracy but forces waiting, gold prefixes in training differ from model guesses at test time so one wrong word skews the next query, and one clip yields one transcript. Overlapping voices break that last assumption completely. Split speakers first with speaker diarization, then transcribe each turn.
Code
The point here is the check, not the formula. This snippet recomputes both numbers from the worked example: the frame count and the token probabilities.
import math
def n_frames(duration_s: float, sr: int = 16000, win_ms: float = 25.0, hop_ms: float = 10.0) -> int: total = int(duration_s * sr) # samples in the clip win = int(win_ms / 1000.0 * sr) # samples per slice hop = int(hop_ms / 1000.0 * sr) # step between slices return 1 + (total - win) // hop
def softmax(xs: list[float]) -> list[float]: m = max(xs) # shift by the max; the result is unchanged exps = [math.exp(x - m) for x in xs] total = sum(exps) return [e / total for e in exps]
print(n_frames(30.0)) # -> 2998print([f"{p:.3f}" for p in softmax([2.0, 1.0, 0.0])]) # -> ['0.665', '0.245', '0.090']Thirty seconds gives 2,998 frames, so the mel picture is . Logits give , which sum to and match the diagram. Change the prefix and the winning index changes even though these odds stay the same shape.
Watch Out For
Feeding an hour and losing the middle
You pass a 90 minute recording in one call and the middle comes back thin, repeated, or with drifting timestamps. That is the 30 second window, not a bug in your audio. The encoder only ever saw one window at a time, so a single pass silently slices or truncates the rest.
Chunk on purpose instead. Cut the audio into 30 second pieces with a few seconds of overlap, decode each piece, then merge on the overlapped words and timestamps. If a sentence straddles a boundary, the overlap gives both sides the context to agree. Never score long form accuracy from one uncut pass.
Polite fiction over silence
You transcribe a clip with long pauses and find full sentences where nobody spoke, often captions-style closings. The symptom is fluent text aligned to near silence. The cause sits in the decoder: when cross-attention finds no speech, the language model half of the network fills the gap with likely next words.
Gate the silence before it reaches the model. Run voice activity detection first and skip non speech spans, or set a no speech threshold so low confidence spans return empty. Then spot check one silent span by hand. If it still returns words, your threshold is too loose, not your microphone.
The Quick Version
- Whisper turns a 30-second clip into an 80 by 2998 mel picture, encodes it once, then writes text one token at a time.
- Task tokens pick the job up front: language tag plus transcribe or translate plus timestamp format, all as ordinary tokens.
- Each output word reads the audio through cross-attention weights that sum to 1, so every word points at the frames it came from.
- Logits 2, 1, 0 give odds 0.665, 0.245, 0.090, which is why decoding blends candidates instead of locking the first guess.
- The price is latency and one transcript per clip: no streaming mid sentence, and overlapping speakers need diarization first.
- Never feed hours in one pass and never trust text aligned to silence without a voice gate.