How LLMs Work

An LLM reads your text as numbered tokens, refines them through stacked transformer blocks, and turns the last position into odds for what comes next. Then it draws one token, appends it, and repeats.

Tokens become vectors, transformer blocks refine them, and the last position's logits become odds for the next token, which is sampled and appended

Why Does This Exist?

You want a machine that writes fluent answers to prompts you have never seen before. Hand-writing grammar rules and facts fails fast: language has too many corners, and every new rule breaks two old ones. So the field reframed writing as a prediction game with unlimited free answer keys. Take any text, hide the next chunk, ask the model to guess it, and check against what actually came next. The web supplies endless training pairs, and getting good at this one game quietly forces grammar, facts, and style into the same weights.

This page traces the inference loop end to end: tokens in, odds out, one token drawn, repeat. The training side of the same idea gets its own treatment in next-token prediction. The block internals live in transformers. How you pick from the odds lives in decoding strategies.

The one idea you need first: a token is the integer id the model actually reads, usually a word fragment rather than a whole word (the merge rules are in byte-pair encoding). An embedding turns each id into a list of numbers with a position signal added, so the blocks that follow can tell first from fifth. That is the whole prerequisite.

Think of It Like This

A phone keyboard that read the whole library

You know the suggestion bar above a phone keyboard. You type a few words and it offers what usually comes next. Now imagine that bar had read an enormous library before it ever met you, and instead of looking up canned phrases it computed fresh odds from everything it absorbed. You tap a suggestion, the sentence grows by one word, and the bar recomputes for the new, longer sentence. Thousands of those taps per minute and the output reads like a considered reply.

Where it stops: your keyboard replays phrases somebody typed before, while an LLM computes new odds at every step and can write sentences nobody has ever typed. But like the keyboard, it knows what usually comes next, not what you meant. Fluency is guaranteed. Truth is not.

How It Actually Works

From text to token ids to vectors

Tokenization cuts your text into a sequence of token ids $t_1, t_2, \ldots, t_n$ , where each $t_i$ (the id at position $i$ ) is an integer in $[0, V)$ , and $V$ (the vocabulary size) is the count of distinct tokens, typically tens of thousands. Each id looks up an embedding vector $\mathbf{e}_i$ (a list of $d$ numbers, where $d$ is the model width), plus a position signal so order survives. Nothing has been understood yet. The text is just $n$ vectors in a row.

Blocks refine, the last position decides

A stack of $N$ transformer blocks (self-attention plus a feed-forward network, repeated) rewrites those vectors so every position gathers context from the positions before it. Call the refined vectors $\mathbf{h}_1, \ldots, \mathbf{h}_n$ . Only the last one matters for what comes next: a final linear layer maps $\mathbf{h}_n$ to a logit vector $\mathbf{z}$ (one raw score per vocabulary entry, so $\mathbf{z}$ has length $V$ ). Softmax turns scores into probabilities:

p_i = \frac{\exp(z_i)}{\sum_j \exp(z_j)}

where $p_i$ (the probability of token $i$ ) is the model's stated chance that token $i$ continues the sequence, and the denominator sums over all $V$ entries so the $p_i$ values add to 1. Then you draw one token from that distribution, append it to the input, and run the whole pipeline again. That loop is what autoregressive means: each output becomes the next input. A 500 token reply costs about 500 full passes, each over a longer sequence than the last, which is why long generations are slow and why a KV cache exists. Positions outside the context window fall out of sight as the sequence grows.

Worked example: logits $[2, 1, 0]$ become odds

Shrink the vocabulary to three words, Paris, London, Rome, and say the last position produces logits $\mathbf{z} = [2, 1, 0]$ . Exponentiate each entry: $\exp(2) \approx 7.389$ , $\exp(1) \approx 2.718$ , $\exp(0) = 1.000$ . Add them: $7.389 + 2.718 + 1.000 = 11.107$ . Divide each by the sum:

\mathbf{p} = [7.389 / 11.107,\ 2.718 / 11.107,\ 1.000 / 11.107] \approx [0.665,\ 0.245,\ 0.090]

Paris wins with about two thirds of the mass, London takes a quarter, Rome keeps a sliver, and the three sum to 1.000. Sample once and you most likely append Paris, making it part of the input for the next pass. Same odds, new run, possibly London: that is why one prompt yields different completions.

The trade is baked in. Reprocessing the full sequence per token buys coherence across long passages and charges compute that grows with length. When you need a guaranteed fact or a fresh event, do not lean on this loop alone: retrieve first, then generate, because the objective rewards the most plausible continuation, not the truest one. That assumption, plausibility standing in for usefulness, is exactly what breaks in hallucination mechanisms. And because every output becomes input, an early wrong token steers everything after it.

Code

Softmax is the hinge of the whole page, so check it directly. This snippet recomputes the worked example and takes the greedy pick, the token a temperature of zero would return.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e = np.exp(z - np.max(z))  # shift by max so exp cannot overflow6    return e / np.sum(e)7
8
9z: np.ndarray = np.array([2.0, 1.0, 0.0])10p: np.ndarray = softmax(z)11print([round(float(v), 3) for v in p])  # -> [0.665, 0.245, 0.09]12print(int(np.argmax(p)))  # -> 0

The printed list is the same $0.665$ , $0.245$ , $0.090$ as the hand computation above (Python prints the last as $0.09$ ), and index $0$ is Paris. Sampling would draw index $0$ about two thirds of the time instead of always.

Watch Out For

Reading the top token as the true token

The winning token is the most plausible continuation, not the correct one. The symptom is a fluent, confident falsehood: a made up date or citation delivered at probability $0.665$ , where the number measures fit, not truth. You catch it by noticing how smooth the wrong answer sounds, which is exactly what makes it dangerous.

Fix it by refusing to treat open loop generation as retrieval. Ground the prompt with sources first, ask for the uncertain parts to be flagged, and verify names, numbers, and dates against the source. The mechanism behind the failure is in hallucination mechanisms.

Forgetting your output becomes the input

Every sampled token is appended and conditions all later tokens, so one early slip steers the rest of the passage. The symptom is a reply that starts slightly off and drifts further the longer it runs, while lowering the temperature only locks in the wrong path faster. Regenerating the tail never helps, because the poisoned token is still in the context.

Fix it at the head, not the tail. Restate the constraint in the prompt, verify or pin down the first few tokens, and for structured output use constrained decoding so invalid tokens never enter the sequence. The rules for each draw are in decoding strategies.

The Quick Version

An LLM maps text to token ids, embeds them with position signals, refines them through stacked transformer blocks, and reads odds off the last position only.
Softmax turns the last position's logits into probabilities that sum to 1. With logits $[2, 1, 0]$ , the split is about $[0.665, 0.245, 0.090]$ .
Generation is a serial loop: draw one token, append it, rerun everything. Token 500 cannot exist before token 499.
Each new token costs a full pass over the growing sequence, so long replies are slow and old tokens eventually fall outside the context window.
The loop rewards the most plausible continuation, not the truest one, and every output becomes later input, so early errors stick.