ModernBERT

The Problem

The original BERT architecture (2018) was revolutionary for understanding text, but it was frozen in time. It was strictly limited to a 512-token context window (using absolute positional embeddings), and it lacked modern architectural improvements like Rotary Position Embeddings (RoPE), SwiGLU activations, and FlashAttention. While the world moved on to massive decoder-only LLMs for generation tasks, there was still a massive need for fast, cheap encoder models for classification, retrieval, and routing—but the available encoders were outdated and couldn't read long documents.

The Idea

The authors of ModernBERT asked a simple question: What happens if we rebuild an encoder-only model using all the architectural advancements discovered by decoder models over the last six years? They stripped out the legacy components of BERT, replaced them with modern equivalents, and trained it on 2 trillion tokens (a massive increase over the original BERT's 3.3 billion).

How It Works

ModernBERT is essentially a classic encoder upgraded with modern deep learning primitives:

Rotary Position Embeddings (RoPE) Absolute position embeddings (where position 1 is always exactly position 1) were replaced with RoPE. This allows the model to understand relative distances between words, enabling the context window to scale to 8,192 tokens natively.

GeGLU Activations The old GELU activation function in the feed-forward network was replaced with GeGLU (a variant of the GLU architecture used in models like LLaMA), which provides better gradient flow and expressivity.

Unpadding and FlashAttention In standard batch processing, short sentences are padded with zeros to match the longest sentence in the batch, wasting massive amounts of compute. ModernBERT uses sequence unpadding and FlashAttention-2 to compute attention only on actual tokens, dramatically accelerating training and inference.

Why It Mattered

ModernBERT proved that the encoder architecture wasn't dead—it just needed an update. By extending the context window to 8k tokens and drastically reducing latency, it unlocked the ability to run classification and retrieval on entire documents, not just single paragraphs. It achieved state-of-the-art results on encoder benchmarks while running faster than equivalent legacy models.

What Came After

ModernBERT became the immediate backbone for a new generation of "System 1" models. Because it can process a massive context window in a single, fast forward pass, models like Convai's Laya and TypeSafe's Jev use ModernBERT (or its multilingual variant mmBERT) as their core engine to read user states and output fast, typed routing decisions.