RetNet

The Problem

Standard Transformers scale quadratically with sequence length during training and suffer from a growing KV-cache during inference, causing significant memory bottlenecks. While various linear attention approximations and recurrent architectures (like RNNs) attempted to solve this, they usually traded away training parallelism or raw modeling performance. The field was stuck in an "impossible triangle": architectures could achieve two out of three among parallel training, low-cost inference, and strong performance, but never all three.

The Idea

The authors proposed the Retentive Network (RetNet), which replaces standard softmax attention with a new retention mechanism. The core insight is that by dropping the non-linear softmax operation from the attention calculation and replacing it with an explicit exponential decay factor, the sequence modeling math can be re-written in three mathematically equivalent forms: parallel, recurrent, and chunkwise.

How It Works

1. The Retention Mechanism Instead of the standard $Softmax(QK^T)V$, RetNet computes retention using a Hadamard product with an exponential decay matrix. The attention score between token $i$ and token $j$ decays exponentially based on their distance $(i - j)$. This explicit decay replaces the need for complex positional encodings.

2. Three Equivalent Forms Because the operation is linear, RetNet can be computed in three ways:

• Parallel representation: Computed just like standard self-attention using matrices. Used during training to fully utilize GPUs.
• Recurrent representation: Computed state-by-state like an RNN, passing a fixed-size hidden state forward. Used during inference to achieve $O(1)$ memory and $O(1)$ time per token, entirely eliminating the KV-cache.
• Chunkwise recurrent representation: Breaks long sequences into blocks, computing parallel retention within each block and recurrent retention across blocks. This efficiently handles ultra-long sequences.

3. Multi-Scale Retention (MSR) Similar to multi-head attention, RetNet uses Multi-Scale Retention. Different heads use different exponential decay rates. Some heads focus on short-term, local context (fast decay), while others maintain long-term context (slow decay).

Why It Mattered

RetNet broke the "impossible triangle." It demonstrated that a model could match or exceed Transformer-level scaling and zero-shot performance while maintaining $O(1)$ inference complexity and enabling highly efficient parallel training. The retention mechanism proved that softmax is not strictly necessary for high-quality sequence modeling, paving the way for linear-time architectures that are hardware-friendly.

What Came After

RetNet, along with other concurrent works like RWKV and Mamba, kicked off a renaissance in sub-quadratic architectures. Its chunkwise recurrent formulation heavily influenced subsequent linear attention models and hybrid architectures that attempt to combine the exact retrieval capabilities of attention with the constant-memory inference of state space models.