Skip to content
AI360XpertBeta

Whisper Architecture

Whisper listens to a whole 30-second clip once, then writes the words one by one. A few short tokens up front tell it whether to transcribe or translate to English.

Whisper encodes a 30-second mel spectrogram once, then decodes text token by token with cross-attention while task tokens choose transcribe or translate
Whisper encodes a 30-second mel spectrogram once, then decodes text token by token with cross-attention while task tokens choose transcribe or translate

Why Does This Exist?

You get a voice note in Spanish, a meeting recording in English, and a voicemail where the speaker switches halfway. You want text for all three, plus an English version of the Spanish one. The old answer was a chain: detect the language, transcribe with one model, then translate with another. Each step added its own errors, and a wrong language guess poisoned everything after it.

That chain also had no good way to share what it learned. A model that transcribes English well knew nothing about translating French, even though both jobs start from the same sounds. Teams kept one model per language pair and paid for each one separately.

Whisper replaces the chain with one encoder-decoder Transformer that reads sound as a picture and writes text one token at a time. A short prefix of task tokens tells the same decoder whether to copy the words or turn them into English. You do not swap models. You swap four tokens.

This page covers that Whisper loop: how audio becomes a mel picture, what the encoder keeps, how the tokens steer the decoder, and how cross-attention aims each word at its sound. The frame pipeline it builds on lives in automatic speech recognition. The general listen first, spell later skeleton has its own page in encoder-decoder ASR. The search that picks the final sentence lives in beam search.

One grounding note, since the rest assumes it. Sound enters as frames: short overlapping slices summarized as energies per pitch band. If frames, spectra, or the blank collapse feel new, read the ASR page first, then come back.

Think of It Like This

A stenographer with colored cover sheets

You hire a stenographer who knows many languages. You hand her a tape plus a colored cover sheet. Blue means type exactly what you hear. Yellow means type it in English. She listens to the whole segment first, then types sentence by sentence, rewinding the tape to check each fuzzy name before she commits.

The full listen is the encoder. Typing sentence by sentence is the decoder. Rewinding to check is cross-attention. The cover sheet is the task prefix: same person, same ears, different job because the sheet says so.

Where it stops: your stenographer understands the trial and will tell you when a passage was inaudible. Whisper understands nothing. It matches sound patterns to text patterns, so it types a wrong word with full confidence and will invent polite closing lines over pure silence rather than admit it heard nothing.

How It Actually Works

Call the sampled wave x[n]x[n], where xx is air pressure as a number and nn counts samples from the start. Whisper never reads x[n]x[n] directly. It repictures the wave, reads the picture once, then writes text while looking back at that reading.

From wave to picture

Cut 30 seconds of audio into overlapping slices. The standard slice is 25 milliseconds wide with a new slice every 10 milliseconds. At 16,000 samples a second, one slice holds 0.025×16000=4000.025 \times 16000 = 400 numbers and the hop is 0.010×16000=1600.010 \times 16000 = 160 numbers.

Count the slices, call that count TT. Thirty seconds is 30×16000=48000030 \times 16000 = 480000 samples, so T=1+⌊(480000−400)/160⌋=2998T = 1 + \lfloor (480000 - 400)/160 \rfloor = 2998, about 3,000 frames. For each frame, measure energy in 80 pitch bands spaced for human hearing and compress loudness with a logarithm. Stack the columns and you hold an 80×299880 \times 2998 log-Mel spectrogram: time along one axis, pitch along the other. That picture is the encoder input, not the raw wave.

The encoder reads once

The encoder is a Transformer stack that takes the whole picture before a single word is written. Write its output as H=(h1,…,hT)H = (h_1, \ldots, h_T), where each hih_i is a dd-dimensional vector and dd is just the width of the model. Because every hih_i is built with access to past and future frames, h50h_{50} carries what happened at frame 10 and frame 90. A mumbled word at the start gets represented together with the later words that disambiguate it.

That full read is the accuracy win and the latency price. Nothing is emitted until the 30 seconds are encoded, which is why this shape fits recorded clips and prepared captions and fails live captioning where words must appear mid sentence.

Task tokens steer the decoder

The decoder is an autoregressive language model: it writes token yty_t after seeing the earlier tokens y1y_1 through yt−1y_{t-1}, one step at a time. Before it writes anything, you feed it a fixed prefix. A Spanish to English request looks like this:

<|startoftranscript|> <|es|> <|translate|> <|notimestamps|>

In plain words: start marker, then language tag ll saying the audio is Spanish, then job tag jj saying translate rather than transcribe, then format tag saying plain text without timestamps. Swap <|translate|> for <|transcribe|> and the same weights copy Spanish instead. Swap <|es|> for the detect tag and the model writes its own language guess first. The tokens are ordinary vocabulary items with learned embeddings. Training showed the model millions of such prefixes, so the prefix sets the output distribution from step zero.

To pick token tt, the decoder builds a query qtq_t from its state sts_t, a summary of what it has written so far. It scores that query against every encoder vector hih_i, turns the scores into weights αt,i\alpha_{t,i} that are positive and sum to 1, and reads the weighted average ct=∑iαt,ihic_t = \sum_i \alpha_{t,i} h_i. Call ctc_t the context: the slice of audio this word listens to. The next token distribution is p(yt∣y<t,X)=softmax(Wo[st;ct])p(y_t \mid y_{<t}, X) = \text{softmax}(W_o [s_t; c_t]), where WoW_o is a learned output matrix and [st;ct][s_t; c_t] means the two vectors joined. In short: self-attention remembers the text so far, cross-attention points at the sound that matters now.

Worked example: one picture size plus one token choice

Keep the picture math from above: 30 seconds gives about 2,998 frames, so the encoder input is 80×299880 \times 2998. That is the number drawn in the diagram: 80 pitch rows, roughly 3,000 time columns, read once.

Now one decoding step. Say the decoder scores three candidate next tokens with logits z=[2,1,0]z = [2, 1, 0], where a logit is just an unnormalized score. Exponentiate: e2≈7.389e^2 \approx 7.389, e1≈2.718e^1 \approx 2.718, e0=1.0e^0 = 1.0, for a sum of about 11.10711.107. Divide:

p=[7.389/11.107, 2.718/11.107, 1.0/11.107]≈[0.665, 0.245, 0.090]p = [7.389/11.107,\ 2.718/11.107,\ 1.0/11.107] \approx [0.665,\ 0.245,\ 0.090]

The three numbers sum to 1.01.0. With prefix <|es|> <|translate|>, the top token spells the English piece, say hello, with weight 0.6650.665. With prefix <|es|> <|transcribe|>, the same audio state favors the Spanish spelling instead. Same encoder reading, same scoring rule, different first tokens because the prefix moved the query. The diagram draws these exact weights beside the output word.

The honest trade follows. Future context helps accuracy but forces waiting, gold prefixes in training differ from model guesses at test time so one wrong word skews the next query, and one clip yields one transcript. Overlapping voices break that last assumption completely. Split speakers first with speaker diarization, then transcribe each turn.

Code

The point here is the check, not the formula. This snippet recomputes both numbers from the worked example: the frame count and the token probabilities.

import math

def n_frames(duration_s: float, sr: int = 16000, win_ms: float = 25.0, hop_ms: float = 10.0) -> int:    total = int(duration_s * sr)  # samples in the clip    win = int(win_ms / 1000.0 * sr)  # samples per slice    hop = int(hop_ms / 1000.0 * sr)  # step between slices    return 1 + (total - win) // hop

def softmax(xs: list[float]) -> list[float]:    m = max(xs)  # shift by the max; the result is unchanged    exps = [math.exp(x - m) for x in xs]    total = sum(exps)    return [e / total for e in exps]

print(n_frames(30.0))  # -> 2998print([f"{p:.3f}" for p in softmax([2.0, 1.0, 0.0])])  # -> ['0.665', '0.245', '0.090']

Thirty seconds gives 2,998 frames, so the mel picture is 80×299880 \times 2998. Logits [2,1,0][2, 1, 0] give [0.665,0.245,0.090][0.665, 0.245, 0.090], which sum to 1.01.0 and match the diagram. Change the prefix and the winning index changes even though these odds stay the same shape.

Watch Out For

Feeding an hour and losing the middle

You pass a 90 minute recording in one call and the middle comes back thin, repeated, or with drifting timestamps. That is the 30 second window, not a bug in your audio. The encoder only ever saw one window at a time, so a single pass silently slices or truncates the rest.

Chunk on purpose instead. Cut the audio into 30 second pieces with a few seconds of overlap, decode each piece, then merge on the overlapped words and timestamps. If a sentence straddles a boundary, the overlap gives both sides the context to agree. Never score long form accuracy from one uncut pass.

Polite fiction over silence

You transcribe a clip with long pauses and find full sentences where nobody spoke, often captions-style closings. The symptom is fluent text aligned to near silence. The cause sits in the decoder: when cross-attention finds no speech, the language model half of the network fills the gap with likely next words.

Gate the silence before it reaches the model. Run voice activity detection first and skip non speech spans, or set a no speech threshold so low confidence spans return empty. Then spot check one silent span by hand. If it still returns words, your threshold is too loose, not your microphone.

The Quick Version

  • Whisper turns a 30-second clip into an 80 by 2998 mel picture, encodes it once, then writes text one token at a time.
  • Task tokens pick the job up front: language tag plus transcribe or translate plus timestamp format, all as ordinary tokens.
  • Each output word reads the audio through cross-attention weights that sum to 1, so every word points at the frames it came from.
  • Logits 2, 1, 0 give odds 0.665, 0.245, 0.090, which is why decoding blends candidates instead of locking the first guess.
  • The price is latency and one transcript per clip: no streaming mid sentence, and overlapping speakers need diarization first.
  • Never feed hours in one pass and never trust text aligned to silence without a voice gate.