RNN Transducer (RNN-T)

Why Does This Exist?

Automatic Speech Recognition (ASR) used to force a choice between accuracy and latency.

If you wanted low latency—like live captions while someone is speaking—you used CTC loss (ctc-loss). CTC processes audio frame by frame, but it assumes every frame is independent. It has no idea what words make sense together. To get readable text, you had to strap a separate, massive language model onto the end of it.

If you wanted high accuracy, you used an encoder-decoder model (encoder-decoder-asr) with attention. It reads the whole audio sequence, looks at all the context, and translates it to text. But attention needs the entire audio sequence before it can start translating. If the user talks for 30 seconds, they wait 30 seconds to see the first word.

RNN Transducer (RNN-T) exists to break that tradeoff. It achieves the accuracy of an encoder-decoder model but emits words instantly as the audio arrives, making it the dominant architecture for streaming speech recognition in production today.

Think of It Like This

Imagine a court stenographer transcribing a trial. They are doing two things simultaneously:

1. Listening to the sounds the witness is making right now.
2. Remembering the sentence that is currently being built.

If the witness mumbles a sound that could be "recognize" or "wreck a nice", the stenographer uses the memory of the sentence ("We need to...") to lock in the correct word ("recognize") the instant they hear it.

RNN-T builds exactly this structure. It runs two separate networks—an ear (the encoder) and a memory (the prediction network)—and feeds them into a third network (the joint) that makes the final call.

How It Actually Works

The RNN-T architecture consists of three distinct neural networks working together.

1. The Encoder Network (The Ear)

The encoder reads the incoming acoustic frames (the mel spectrograms). Unlike a standard sequence-to-sequence encoder, it doesn't wait for the sentence to finish. It processes chunks of audio and produces an acoustic representation, $h^{enc}_t$, for the current time step $t$.

2. The Prediction Network (The Memory)

The prediction network acts as an internal language model. It takes the sequence of text tokens the model has emitted so far (up to step $u$) and outputs a representation of the linguistic context, $h^{pred}_u$. Crucially, it only looks at the text, completely blind to the audio.

3. The Joint Network (The Decider)

The joint network takes the acoustic vector $h^{enc}_t$ and the linguistic vector $h^{pred}_u$, combines them (usually with a simple addition or feed-forward layer), and applies a softmax to predict the next token $y_{t,u}$:

$$h^{joint}_{t, u} = \text{FeedForward}(h^{enc}_t + h^{pred}_u)$$ $$P(y_{t, u} | t, u) = \text{Softmax}(h^{joint}_{t, u})$$

The Blank Token and the Lattice

Because speech is continuous, the model might process several audio frames before it has enough confidence to emit a word. RNN-T handles this with a special [BLANK] token.

At every step, the joint network can either:

• Emit a text token. The prediction network updates its state with the new word, but the encoder stays on the current audio frame.
• Emit [BLANK]. This means "I need to hear more." The prediction network freezes its state, and the encoder steps forward to consume the next audio frame.

This creates a 2D lattice of time steps vs. output tokens. The model traverses this lattice, perfectly aligning variable-length audio with variable-length text without ever needing an explicit alignment map.

Watch Out For

The memory explosion during training. During inference, RNN-T is incredibly efficient because it only computes a single path through the lattice. But during training, the loss function must sum the probabilities of all possible valid paths through the lattice.

If your audio has $T$ frames and the transcript has $U$ tokens, the joint network must compute a tensor of size $(B \times T \times U \times V)$ where $B$ is batch size and $V$ is the vocabulary size. A batch of long audio clips will instantly OOM a standard GPU unless you use specialized, memory-efficient RNN-T loss implementations (like pruned RNN-T or recurrent-free transducers).

The Quick Version

• The Problem: CTC models have no language context, and attention models can't stream.
• The Fix: RNN-T splits the job into three parts: an encoder for audio, a prediction network for text history, and a joint network to combine them.
• The Execution: It uses a [BLANK] token to consume audio frames until it has enough confidence to emit a word.
• The Result: Highly accurate, perfectly streaming speech recognition.

What to Read Next

• CTC Loss (ctc-loss) — The precursor to RNN-T, and how the [BLANK] token originated.
• Encoder-Decoder ASR (encoder-decoder-asr) — The offline alternative that uses cross-attention.