Semantic Entropy
An advanced metric used to measure a language model's uncertainty by evaluating if its varied generated responses actually mean the same thing.
Think of It Like This
Like asking a witness the same question five times; if their answers use different words but tell the exact same story, they are certain.
Standard entropy just measures if the model outputs different words, but semantic entropy clusters those words by meaning. If an LLM generates 'Paris', 'The capital of France', and 'Paris, France', the semantic entropy is very low. High semantic entropy is a highly reliable indicator that the model is hallucinating or guessing.