Skip to content
AI360XpertBeta

BERT: Bidirectional Encoder That Fills Masks

BERT reads the whole sentence at once and learns by filling in hidden words, so each word ends up carrying context from both sides.

One masked position pulls context from both sides to score mat at 0.665 over rug at 0.245
One masked position pulls context from both sides to score mat at 0.665 over rug at 0.245

Why Does This Exist?

A left to right reader must decide what bank means before the clue arrives. Take the bank raised its rates and the bank overflowed after the storm. Rates points to money while overflowed points to a river, and both clues sit after bank. A model that predicts one word at a time has already guessed by the time the useful word shows up.

BERT drops that limit by giving up generation as the goal. It stacks transformer blocks with no causal mask, so each position reads all the others at once, left and right. You can't train that layout with next-word prediction because the answer is already visible, so the job becomes hide-and-guess: cover some tokens and predict them from full surrounding context. The original run learned from about 3.3 billion words of books and encyclopedia text, which is why such a plain job taught so much.

This page covers the hide-and-guess job and the encoder-only layout behind it. GPT covers the sibling branch that keeps the causal mask and generates one token at a time. Encoder-decoder architectures lines up all three layouts when you want the full map. If queries, keys, and values still feel fuzzy, read self-attention first, since each BERT layer runs it with no restriction.

Think of It Like This

A worksheet with one word covered by a thumb

You get a printed sentence with one word covered by your thumb. You can still read everything left of your thumb and everything right of it, and you guess the covered word from both sides at once. The cat sat on the blank mat is easy because sat and mat agree from opposite sides.

In the picture, your thumb is the [MASK] symbol, the visible words are the unmasked tokens, and your guess is the score over the vocabulary. You pull clues from both directions, which a left to right guesser can't do because its thumb also covers everything to the right.

Where it stops: you read whole words, but BERT reads WordPiece pieces from a 30,000-piece list, so mat might be one piece while a rare name splits in two. And your worksheet always shows a blank, while BERT shows a blank on only 80 percent of the chosen spots, with 10 percent swapped to a random token and 10 percent left unchanged, so it can't lean on the blank itself as a crutch.

How It Actually Works

Call the vocabulary VV, a fixed WordPiece list of about 30,000 pieces. Call the input tokens x1x_1 through xnx_n, with nn capped at 512 positions. Call the hidden width HH, meaning the size of each token vector (768 in BERT-Base), the layer count LL (12 in BERT-Base and 24 in BERT-Large), and the head count AA (12 and 16 respectively). Each layer runs full self-attention with no mask, so position ii reads every xjx_j, left and right, in one step.

What goes into each position?

Three learned tables add up at each spot. Token embedding says which piece this is. Segment embedding says sentence A or sentence B for pair tasks. Position embedding says which of the 512 slots this sits in. The sum is the input vector:

ei=eitok+eiseg+eipose_i = e_i^{tok} + e_i^{seg} + e_i^{pos}

eie_i is the input vector at position ii, eitoke_i^{tok} the piece identity, eisege_i^{seg} the sentence tag, and eipose_i^{pos} the slot code. A [CLS] piece opens the sequence and a [SEP] piece closes each sentence, so classification can later read the [CLS] vector as the summary of the whole input.

Which tokens does the model predict?

Pick 15 percent of the tokens for prediction. Of those, 80 percent become [MASK], 10 percent become a random vocabulary token, and 10 percent stay unchanged, yet all three groups count as targets with a cross-entropy loss. The mix keeps the model from trusting the [MASK] symbol itself, since that symbol never appears when you later fine-tune.

Counts stay whole on a round batch. Take 200 tokens: 200×0.15=30200 \times 0.15 = 30 prediction spots, then 30×0.8=2430 \times 0.8 = 24 shown as [MASK], 30×0.1=330 \times 0.1 = 3 swapped to random tokens, and 30×0.1=330 \times 0.1 = 3 left as is. Only those 30 spots pay loss. The other 170 pass through and teach only by serving as context.

Worked example: one hidden word, three scores

Say the sentence is the cat sat on the [MASK] mat and the hidden piece is mat. The mask vector hmh_m scores every vocabulary entry through one output matrix WW, giving a logit zkz_k per candidate kk:

z=hmW+bz = h_m W + b

hmh_m is the final hidden vector at the masked spot, WW the vocabulary projection, bb a bias, and zz the raw scores. Keep three candidates to check by hand: mat at 2.02.0, rug at 1.01.0, and moon at 0.00.0.

Softmax turns logits into shares that sum to 1. Write exp⁡\exp for the exponential:

pk=exp⁡(zk)∑jexp⁡(zj)p_k = \frac{\exp(z_k)}{\sum_j \exp(z_j)}

pkp_k is the predicted share for candidate kk. Exponentiate: exp⁡(2.0)≈7.389\exp(2.0) \approx 7.389, exp⁡(1.0)≈2.718\exp(1.0) \approx 2.718, and exp⁡(0.0)=1.000\exp(0.0) = 1.000. Add: 7.389+2.718+1.000=11.1077.389 + 2.718 + 1.000 = 11.107. Divide:

pmat=7.389/11.107≈0.665,prug≈0.245,pmoon≈0.090p_{mat} = 7.389 / 11.107 \approx 0.665, \quad p_{rug} \approx 0.245, \quad p_{moon} \approx 0.090

The diagram draws these same three bars: mat longest at 0.665, rug middle at 0.245, and moon short at 0.090. Both sides voted, and mat wins without ever seeing itself, only its neighbours.

What does the loss actually see?

Loss is negative log likelihood on masked spots only:

L=−log⁡pcorrectL = -\log p_{correct}

LL is the loss for one masked spot and pcorrectp_{correct} the predicted share for the true piece. Here pcorrect=0.665p_{correct} = 0.665, so L=−log⁡(0.665)≈0.41L = -\log(0.665) \approx 0.41. Unmasked spots pay zero. That single number is the whole training signal for this sentence in our tiny demo. Real pretraining averages it over all 30 spots in the batch, over BooksCorpus plus English Wikipedia at about 3.3 billion words.

The original run added a second job, next-sentence prediction. Feed two sentences with a [SEP] between them and ask from the [CLS] vector: did B follow A, yes or no. Later work found this check added little, and strong variants dropped it while keeping masked modeling and tuning the masking itself. Treat it as history, not as the reason BERT worked.

Code

The point is the check: run the same three logits through softmax and confirm 0.665, 0.245, 0.090 and a loss of 0.41.

import math

def softmax(logits: list[float]) -> list[float]:    exps: list[float] = [math.exp(z) for z in logits]    total: float = sum(exps)    return [e / total for e in exps]

logits: list[float] = [2.0, 1.0, 0.0]probs: list[float] = softmax(logits)print(f"{probs[0]:.3f} {probs[1]:.3f} {probs[2]:.3f}")  # -> 0.665 0.245 0.090print(f"{-math.log(probs[0]):.2f}")  # -> 0.41

The printout matches the hand computation above: exps 7.389, 2.718, and 1.000 sum to 11.107, shares 0.665, 0.245, and 0.090, and negative log likelihood 0.41 on the correct word.

Watch Out For

Asking BERT to write a paragraph

Symptom: you prompt an encoder-only checkpoint for a continuation and get one filled blank or the same word repeated, never a fluent paragraph. There is no next-token slot because no causal mask ever forced left to right order, so generation has no defined starting point.

Fix: generate with a decoder-only model such as GPT, and keep BERT-style models for labels, spans, and scores on complete inputs.

Coding 15 percent as always MASK

Symptom: custom pretraining that writes [MASK] on all 15 percent trains well, then stalls on real text with no masks, because the model leaned on the mask symbol itself instead of the surrounding context.

Fix: copy the 80-10-10 split: 80 percent [MASK], 10 percent random token, and 10 percent unchanged, with all three counted as prediction targets.

The Quick Version

  • BERT is an encoder-only transformer with no causal mask, so each of its 12 Base layers reads all 512 positions left and right at once.
  • Pretraining selects 15 percent of tokens for prediction, showing 80 percent as [MASK], 10 percent as random tokens, and 10 percent unchanged.
  • Each input vector sums token, segment, and position embeddings, with [CLS] opening the sequence and [SEP] marking sentence ends.
  • The worked mask scores mat at 2.0, rug at 1.0, and moon at 0.0, giving shares 0.665, 0.245, and 0.090 with a loss of 0.41 on mat.
  • Encoder-only models fit labeling and scoring of complete text, while generation fits decoder-only models such as GPT.