How LLMs Work

Why Does This Exist?

Ask someone how a large language model works and the honest answer sounds almost too simple to be the whole story: it predicts the next word, over and over, one token at a time. That's not a simplification for beginners — it's literally the entire mechanism. Everything else, the apparent reasoning, the code generation, the ability to hold a conversation, is what happens when that one mechanism runs at enough scale, on enough text, for long enough.

That simplicity is exactly why it's worth walking through carefully. If the whole system really is just "predict the next token, then do it again," a lot of behavior that looks mysterious from the outside — why it sometimes makes things up, why longer prompts cost more, why it can't un-see what it just generated — turns out to follow directly from that one fact, once the pipeline is laid out end to end.

Think of It Like This

How It Actually Works

The pipeline, start to finish

The diagram above is the entire architecture, read left to right. Tokenization splits the input text into discrete units the model can index — usually word fragments, not whole words. Embeddings turn each token id into a vector, with a positional signal added so the model knows which token came first. A stack of transformer blocks — the architecture covered in the previous band, self-attention and a feed-forward network, repeated many times — refines those vectors, letting every position gather context from every other position it's allowed to see. The final layer's output at the last position becomes logits: one raw score per possible next token, out of a vocabulary that typically runs from tens of thousands to over a hundred thousand entries. Those logits pass through a softmax to become a probability distribution, and a sampled token gets drawn from it.

Why "generate" means "loop"

Here's the detail that makes an LLM autoregressive rather than a one-shot classifier: the newly sampled token gets appended to the input, and the entire pipeline runs again — tokenize (already done, since it's appended as a token id), embed, run through every transformer block, produce new logits, sample again. Producing a 500-token response means running this full pipeline roughly 500 times, once per output token, each time conditioning on everything generated so far including its own prior outputs.

This is why generation is inherently sequential at inference time, even though the transformer's attention mechanism computes in parallel within a single forward pass: token 500 cannot be sampled before token 499 exists, because token 499 is literally part of the input to the step that produces token 500.

What the training objective actually optimizes

During training, this entire process — tokenize, embed, transform, produce logits — runs on real text, and the model is scored on how much probability it assigned to the actual next token at every position simultaneously (a detail next-token prediction covers directly). No position is treated specially; a document with 10,000 tokens supplies roughly 10,000 separate next-token predictions to learn from in one pass. That density of training signal, applied to internet-scale text, is what turns "predict the next fragment" into a system that appears to know grammar, facts, and reasoning patterns — because getting the next fragment right, over and over, quietly requires all of that.

Show Me the Code

A minimal, non-batched version of the generation loop, calling out where each named step from the diagram happens.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e = np.exp(z - z.max())6    return e / e.sum()7
8
9def generate(token_ids: list[int], model_forward, steps: int, vocab_size: int) -> list[int]:10    for _ in range(steps):11        logits = model_forward(token_ids)               # tokens -> embeddings -> blocks -> logits12        probs = softmax(logits[-1])                       # only the LAST position's logits matter13        next_id = int(np.random.default_rng(0).choice(vocab_size, p=probs))14        token_ids.append(next_id)                          # append, then run the whole pipeline again15    return token_ids16
17
18def stub_forward(ids: list[int]) -> np.ndarray:19    rng = np.random.default_rng(sum(ids))20    return rng.normal(size=(len(ids), 50))                 # stand-in for the real transformer stack21
22
23output = generate([7, 12, 3], stub_forward, steps=3, vocab_size=50)24print(output)  # -> [7, 12, 3, <new>, <new>, <new>] — three tokens appended, one per loop iteration

Every call to model_forward reprocesses the entire growing sequence — that inefficiency, run naively, is exactly what makes a KV cache worth having in a real system.

Watch Out For

The Quick Version

• The pipeline is: tokenize, embed with position, run through stacked transformer blocks, produce logits, sample a token.
• Generation is a loop — the sampled token is appended and the entire pipeline runs again for the next one.
• Training scores every position in a document simultaneously against its true next token, which is what makes the signal so dense.
• Generation is inherently sequential at inference time, even though each individual forward pass is internally parallel.
• Apparent reasoning and knowledge are downstream of this one repeated mechanism, not a separate system layered on top.

What to Read Next

• Next-Token Prediction is the training objective this page's pipeline is optimized against.
• Tokenization covers the first step of the pipeline in full.
• Context Windows covers the limit on how much of the growing sequence the model can actually condition on.
• Decoding Strategies covers the choices available at the "sample a token" step.
• Transformer Architecture is the stack of blocks this page's pipeline runs the vectors through.