Masked Language Modeling
A pre-training objective where random words in a sequence are hidden, and the model must predict them using context from both the left and the right.
Think of It Like This
Like a fill-in-the-blank test where you use the rest of the sentence to guess the missing word in the middle.
MLM is the foundation of encoder-only architectures like BERT. By forcing the model to reconstruct corrupted text, it learns deep bidirectional representations of language. This contrasts with autoregressive models (like GPT) which only use left-to-right context for next-token prediction.