Perplexity
A crucial evaluation metric for language models that quantifies how surprised the model is by a sequence of text; lower perplexity indicates better understan...
Think of It Like This
Like measuring a psychic's skill by how rarely they are completely shocked by an unfolding sequence of completely predictable events.
Mathematically, perplexity is the exponentiated average negative log-likelihood of a sequence. If a model has a perplexity of 10, it is essentially as confused as if it were guessing uniformly from a list of 10 words at each step. It is the gold-standard intrinsic metric for evaluating the quality of LLM pre-training.