Core ML

BERT & Masked Language Modeling

BERT reads a sentence in both directions at once and learns by guessing words hidden behind a mask, building representations rather than generating new text.

Predicting a masked token by pulling in context from both the left and the right at once — bidirectional context is what an encoder-only model has that a left-to-right model doesn't

Why Does This Exist?

Before BERT, the strongest language models read in one direction only — left to right, predicting each word from everything before it, because that's what a generation objective naturally requires. That's a real limitation for understanding a sentence, not generating one. Take "the bank raised its rates" versus "the bank overflowed after the storm" — the word "bank" only resolves once you've seen "rates" or "overflowed", both of which come after it. A left-to-right model deciding what "bank" means has to guess before it's allowed to look at the word that would tell it. Nothing forces a model built purely to understand text to accept that constraint; it only exists because generation forces it.

BERT, published by Google in 2018, dropped the constraint by dropping generation as the goal entirely. It's built on an encoder-decoder-architectures-style encoder-only stack: no causal mask anywhere, every position free to attend to every other position, left and right, at once. To train something that reads in every direction simultaneously, you can't use "predict the next word" — there is no meaningful "next" when the model already sees the whole sentence. BERT's answer was masked language modeling: hide some words, make the model guess them back using everything around the gap, in both directions.

Think of It Like This

Filling in a crossword clue you can see both sides of

Picture a crossword puzzle where one square is blank and every other square around it, before and after, is already filled in. You don't guess the missing letter by only reading leftward — you read the whole clue, both directions, and the answer becomes obvious from context on either side. A left-to-right guesser, forced to commit before seeing what comes after the blank, would have a much harder time.

That's the entire idea behind masked language modeling. Hide a word, let the model see everything else in the sentence — left and right — and train it to fill the gap. The task itself forces the model to build a representation that genuinely understands both directions, because guessing right requires using both.

How It Actually Works

Masking the input

During pretraining, BERT randomly replaces about 15% of input tokens with a special [MASK] token — the diagram above shows one such position — and trains the model to predict the original token at each masked position, using the surrounding unmasked tokens as context. Because there's no causal mask restricting attention, every masked position can pull information from tokens on both sides simultaneously. This is genuinely different from a left-to-right model filling in a blank: BERT sees the actual right-context tokens during training, not a prediction of them.

Why masking, and not just removing

A subtlety worth naming: BERT doesn't always replace the masked 15% with the literal [MASK] token. A fraction get replaced with a random token, and a smaller fraction are left unchanged, even though they're still counted as "masked" positions the model has to predict. That mix exists because [MASK] never appears at actual use time — a model over-tuned to expect [MASK] specifically would behave oddly once you use it for real classification or extraction on ordinary text that contains no mask tokens at all. Mixing in random and unchanged tokens keeps the model from leaning too hard on the mask symbol itself as a signal.

The original second objective: next-sentence prediction

Alongside masked language modeling, the original BERT paper also trained on next-sentence prediction: given two sentences, decide whether the second genuinely follows the first in the source text, or was swapped in from somewhere else. The intent was to teach sentence-pair relationships useful for tasks like question answering. Later work found this objective added less than expected — some strong BERT variants trained afterward drop it entirely and rely on masked language modeling alone, refining the masking strategy instead.

What you get out of it

The end result of pretraining is a stack of encoder blocks that produces a rich contextual representation of every token, built from bidirectional context. Fine-tune that representation with a small task-specific head, and BERT-style models remain a strong choice for classification, named-entity extraction, and any task that needs to score or label a complete piece of text — as opposed to generating new text, which is squarely GPT's territory.

Show Me the Code

A minimal masked-language-modeling training example: mask a position, then compute a "prediction" from the average of every other token's vector — a simplified stand-in for the model reading full bidirectional context.

1import numpy as np2
3
4def bidirectional_context(x: np.ndarray, mask_pos: int) -> np.ndarray:5    n = x.shape[0]6    weights = np.ones(n)7    weights[mask_pos] = 0.0            # exclude the masked position itself8    weights = weights / weights.sum()   # average over every remaining token9    return weights @ x                  # (d,) — context pulled from both sides10
11
12rng = np.random.default_rng(0)13sentence = rng.normal(size=(6, 4))       # 6 tokens, embedding dim 414context = bidirectional_context(sentence, mask_pos=3)15print(np.round(context, 4))  # -> [-0.3573  0.0028  0.2135  0.4975]

Position 3's context vector is built from positions 0, 1, 2, 4, and 5 — both before and after it — which is exactly the bidirectional access a causally-masked decoder would not have.

Watch Out For

Trying to generate text with a BERT-style model

Encoder-only models have no mechanism for autoregressive generation — there's no causal mask forcing left-to-right structure, and no notion of "the next token" the way a decoder-only model has. Asking a BERT-style model to write a paragraph doesn't fail loudly; it just isn't the tool for the job. Use an encoder-only model when the task ends in a label, a span, or a score, and a decoder-only model when the task ends in generated text.

Assuming masked language modeling means 15% of tokens are always literally [MASK]

The commonly cited "15%" is the total fraction of tokens selected for prediction, not the fraction replaced with the literal mask symbol. Some of that 15% get swapped for a random token, some are left as-is — both still count as prediction targets. Skipping that detail when reimplementing pretraining from scratch produces a model that overfits to recognizing the [MASK] symbol itself, rather than learning to use context robustly.

The Quick Version

BERT is an encoder-only transformer: no causal mask, every position attends to every other position, left and right.
It pretrains with masked language modeling — hide tokens, predict them from bidirectional context.
The masked 15% is split between the literal [MASK] symbol, random tokens, and unchanged tokens, so the model doesn't over-rely on the mask symbol.
The original paper also used next-sentence prediction, though later work found it added less value than masking alone.
BERT-style models remain the right choice for classification and extraction — tasks that need a representation of a complete input, not generated text.