Hallucination Mechanisms
A language model is trained to write the most plausible next word, not the truest one. When the fact is missing from its weights, it answers anyway, fluently and wrong.
Why Does This Exist?
Ask a model about something it never saw in training and it answers anyway. Not with silence, and not with "I don't know." You get a fluent, specific, confident falsehood instead: a researcher who never existed, a paper that was never written, a refund policy your company never had. The sixty-second version of the term lives at hallucination.
This is the failure mode that looks nothing like a failure. The grammar is perfect and the details are crisp, so nothing in the tone marks the answer as a guess. And the places where you'd most want certainty, legal citations, dosages, policies, are exactly where a confident fabrication costs the most.
This page covers why the miss happens: the training objective that rewards plausibility, the decoding step that crowns one token, and the loop that locks the fiction in. The fixes live next door, in Reduce LLM Hallucinations, which is about grounding, retrieval, and checking claims before anyone reads them.
The one idea you need first. A model learns by next-token prediction: look at everything so far, guess the next token, get corrected, repeat billions of times. Fluency, facts, and style all get packed into the same weights by that single task. Nothing in the objective says "stay silent when unsure." Silence was never rewarded, so the model never learned it.
Think of It Like This
A pub quiz teammate who never passes
Your team has one rule: someone must shout an answer within five seconds, every round, no passing. Your loudest teammate knows thousands of facts, so most rounds they nail it. Then comes a question about a 2019 paper nobody at the table has read. The rule still holds. They shout the most plausible-sounding name with total conviction, the team writes it down, and the next round's discussion now treats that name as settled.
Every part maps. Never passing is the missing abstention option. The loudest shout is the highest logit. The conviction in their voice is the RLHF-polished tone. Writing it down is the autoregressive loop: once emitted, the fiction becomes context.
Where it stops: your teammate knows they are guessing. The model has no such sense. There is no inner light that says "I know this" versus "I'm inventing this." There is only the distribution over next tokens, and a guess it must make.
How It Actually Works
Plausibility is the objective, and truth is a side effect
For each position, the model produces raw scores called logits, written , one per candidate token. Logits are not probabilities: they can be any size, and bigger means more plausible in this context. A decoding rule turns them into probabilities , one per candidate , with the softmax:
In plain words: is the temperature, a knob you set, where lower values sharpen differences and higher values flatten them. is the exponential function, which turns score gaps into ratios. The sum over in the denominator is what forces every onto the same scale, so the probabilities always add to 1. Nothing in this formula asks which candidate is true. It only asks which scores highest.
Worked example: the fluent wrong token wins
Say you ask about a paper the model never saw, right after the words "...led by Dr." Three candidate surnames get logits for Harrison, Smith, and Achebe. Harrison is a famous name that flows well here, so it scores highest. The true author's name is rare in training, so it never makes the shortlist at all.
Exponentiate at : , , . The sum is . Divide:
Check the invariant: . Greedy decoding picks the top entry, so the model emits "Harrison" with probability mass 0.665 behind a name it has no evidence for. That is the whole mechanism in one line: the fluent token wins because decoding rewards the score, and the score measures plausibility.
Raise the temperature to and the mass spreads to about . Harrison still leads. This is worth sitting with, because it kills a comforting myth: sampling tricks redistribute the mass, but they cannot surface knowledge the weights never stored.
One wrong token locks the rest in
Generation is autoregressive: each emitted token becomes part of the context for every token after it. Once "Harrison" is out, the next distribution is , and continuations consistent with a Dr. Harrison now score highest. The model defends its own fiction, detail by detail, because each new token is chosen to fit the story so far. A single invented name snowballs into a biography.
Confidence is a tone, not a measurement
Two separate forces make hallucinations sound certain. First, high-probability tokens read as fluent, and readers hear fluency as confidence. Second, preference tuning rewards helpful, authoritative-sounding answers, so hedging gets trained away even where hedging would be honest. The result: the model's tone tells you how its training data usually sounds, not how likely this particular claim is to be true.
So here is the honest trade-off. Decoding controls (temperature, top-, top-, all covered in temperature) buy you variety and shape, but they spend the same fixed mass from the same fixed logits. Grounding in retrieved text, covered in RAG, adds knowledge the weights lack, but it costs a retrieval step and is only as good as its sources. And the standing assumption behind every raw answer is that your question resembles the training distribution. Ask about something sparse, recent, or fabricated, like a paper that never existed, and that assumption fails silently. Do not use an ungrounded model for cited facts, full stop.
Code
The point of this snippet is the mechanism made runnable: one fixed logit vector becomes probabilities that sum to 1, the fluent token wins, and warming the temperature spreads the mass without changing the winner.
import math
def softmax(logits: list[float], temperature: float = 1.0) -> list[float]: scaled = [z / temperature for z in logits] # temperature softens the gaps m = max(scaled) # subtract the max so exp cannot overflow exps = [math.exp(z - m) for z in scaled] total = sum(exps) return [e / total for e in exps]
logits = [2.0, 1.0, 0.0] # Harrison, Smith, Achebe after "...led by Dr."print([round(p, 3) for p in softmax(logits)]) # -> [0.665, 0.245, 0.09]print([round(p, 3) for p in softmax(logits, temperature=2.0)]) # -> [0.506, 0.307, 0.186]Both lines agree with the hand computation above, including the invariant that each sums to 1. The second line is the myth-killer: at temperature 2 the wrong token still leads at 0.506.
Watch Out For
Treating a low temperature as a truth serum
The symptom: you set temperature to 0, the answers turn crisp and repeatable, and you start trusting them as facts. Then a crisp, repeatable, fabricated citation shows up in front of a user.
Temperature 0 just means "always pick the top logit." When the top logit is a fluent fiction, you now emit that fiction deterministically. Deterministic is not the same as verified. If the answer needs to be true, ground it in a source or let the model abstain; the sampling knob was never the fix.
Grading answers by polish instead of evidence
The symptom: a beautifully formatted answer with bolded terms, tidy bullets, and an authoritative tone sails through review, while a hedged correct answer gets questioned. Polish correlates with training-data style, not with truth.
Judge the claims, never the presentation. Ask every factual sentence "says who," require a retrieved source or a citation for the ones that matter, and treat unhedged confidence on an obscure question as a warning sign rather than reassurance.
The Quick Version
- A language model outputs the most plausible next token, not the truest one, because next-token training never rewards silence or abstention.
- Softmax turns one fixed logit vector into probabilities that sum to 1, so decoding can only redistribute mass, never create knowledge.
- In the worked example, logits give , and the fluent wrong name wins at 0.665 while the true answer is not even a candidate.
- Generation is autoregressive, so one invented token becomes context that later tokens must stay consistent with, and the fiction snowballs.
- Confidence is a trained tone, not a truth meter, so polish and certainty carry zero evidential weight.
- Sampling knobs reshape variety; only grounding or abstention fix factuality, which is why fixes live in the mitigation page.