Structured Outputs

Why Does This Exist?

Say you are building a system that reads customer emails and turns them into orders. One real email looks like this: "Hey, could you get me two of the standing desk converters by next Friday if possible? Thanks, Dana." You read that in two seconds. Your order system can't. It wants a record with an exact shape: customer_name as a string, item as a string, quantity as an integer, requested_delivery_date as a date. For years that gap was bridged with regex and keyword rules, and they broke every time a customer phrased things a little differently.

A language model is good at the reading part. It knows "two" means 2 and "next Friday" is a date. The trouble starts one step later, when the answer has to come back in a form code can trust. Ask the model to "reply in JSON" and you will get something that looks right most runs. Most runs is not a guarantee. You will see a markdown fence around the object, a friendly sentence before the first brace, deliveryDate where your code expects requested_delivery_date, or the string "two" where the integer 2 belongs. Any one of those makes json.loads throw. Worse, a string can slip where an integer belongs and surface three steps down the pipeline.

This page is about the guarantee: how a schema becomes a per-step rule that keeps output parseable by construction, what it costs, and what it still does not promise. The masking math underneath lives in constrained decoding. The main consumer, models filling argument shapes for real code, lives in tool use and function calling. Checking whether the values are true rather than just well shaped lives in guardrails.

The one idea you need first: a model writes one token at a time, and at each step it emits logits, raw scores with one entry per token in its vocabulary. Sampling turns those scores into probabilities that sum to 1. Everything below edits the scores before sampling runs. The full loop is in next-token prediction, and the picking rules are in decoding strategies.

Think of It Like This

How It Actually Works

You hand the system a real schema, a JSON Schema document or the equivalent model in code, naming each field and its type. The engine compiles it into a tracker that always knows where generation stands: inside the quantity value, after the digits, before the closing brace. At step $t$ that tracker outputs one thing: $A_t$, the allowed set, the list of tokens that may legally come next. $V$ is the vocabulary, every token the model can emit. $z$ in $\mathbb{R}^{|V|}$ holds the logits, one raw score per token.

From schema to allowed tokens

Say the schema defines an order with quantity as an integer. While generation sits inside that value, the tracker allows digit tokens and blocks the rest. One step later, after the digits, it allows a comma or a closing brace instead. The model never sees your schema directly. It feels the effect: some tokens are available this step and the rest are gone.

Mask, then sample

The engine builds masked logits $z'$. In plain words, $z'_i$ equals $z_i$ when token $i$ sits in $A_t$, and equals $-\infty$ when it does not. Then it applies softmax:

Here $p'$ is the new probability distribution, $\exp$ is the exponential function, and the sum runs over the whole vocabulary. Because $\exp(-\infty)$ equals $0$, banned tokens land on exactly $0$ and the rest renormalises to $1$ over the allowed tokens. Only then does sampling run. Greedy choice, temperature, and top-p all operate on $p'$, never on the raw scores. That order matters: the mask runs before the randomness, so no temperature can revive a banned token. The full treatment of that step is in constrained decoding.

Worked example: the quantity step

Four tokens are live at the quantity value: "2", "3", "two", and "}", with logits $z = [2.0, 0.5, 2.5, 1.0]$. You are inside quantity, so $A_t$ holds the two digits only.

First the unconstrained picture. Subtract $m = 2.5$, the maximum, for stability, and exponentiate to $[\exp(-0.5), \exp(-2.0), \exp(0), \exp(-1.5)]$, about $[0.607, 0.135, 1.0, 0.223]$. The sum is about $1.965$. Divide and you get about $[0.309, 0.069, 0.509, 0.114]$. Left alone, greedy decoding picks "two" with $0.509$. Fluent, and illegal.

Now mask. Set the last two entries to $-\infty$ to get $z' = [2.0, 0.5, -\infty, -\infty]$. Relative to $2.0$ that is $[1.0, 0.223, 0.0, 0.0]$. The sum is about $1.223$. Divide and you get about $[0.818, 0.182, 0.0, 0.0]$. The $0.509$ on "two" is gone. Greedy decoding now picks "2", and a random draw would split about 82 to 18 between the two legal digits. Those are the numbers drawn in the diagram above.

The trade is direct. You buy output that parses on the first try, and you pay schema build time plus a per-step legality check, plus the odd detour where the model spells something in more tokens than it would have chosen. That price is worth it for APIs, tool use and function calling, and form filling.

Do not use it for open-ended writing. Brainstorming, chat, and drafts have no shape worth enforcing, and a constraint there narrows voice for no gain. Skip it too when the real requirement cannot be written as a grammar: tone, helpfulness, or factual truth. A light post-check plus retry is cheaper there.

The method rests on one assumption: the grammar means what you meant and the character rules line up with your model's tokens. Tokenisers split text in ways a character regex cannot see, so a rule that looks right can strand generation or cut words at odd joints. When constrained output looks oddly clipped, suspect that mapping before you blame the model.

Code

The point of this snippet is the whole trick in four tokens: same logits, then a mask, then new probabilities. Standard library only.

1import math2
3
4def softmax(logits: list[float]) -> list[float]:5    m = max(logits)6    exps = [math.exp(x - m) for x in logits]7    total = sum(exps)8    return [e / total for e in exps]9
10
11logits = [2.0, 0.5, 2.5, 1.0]  # tokens: ["2", "3", "two", "}"]12allowed = [True, True, False, False]  # inside quantity: digits only13
14full = softmax(logits)15masked = softmax([v if ok else float("-inf") for v, ok in zip(logits, allowed)])16
17print([round(p, 3) for p in full])  # -> [0.309, 0.069, 0.509, 0.114]18print([round(p, 3) for p in masked])  # -> [0.818, 0.182, 0.0, 0.0]

Unconstrained, index 2 ("two") wins with about 0.509. After the mask, index 0 ("2") wins with about 0.818 against 0.182. Same numbers as the hand computation above.

Watch Out For

The Quick Version

• Structured outputs are a decoding constraint compiled from a real schema, not a sentence in the prompt asking for JSON.
• At each step the engine masks illegal tokens to zero before sampling, so the reply parses by construction.
• With logits [2.0, 0.5, 2.5, 1.0] and digits only allowed, "two" at 0.509 vanishes and 0.818 plus 0.182 stay on "2" and "3".
• A guaranteed shape is not a guaranteed truth: a valid record can still carry an invented quantity.
• Make uncertain fields nullable, validate values against the source, and keep schemas off open-ended writing and off the model's scratch thinking.

What to Read Next

• Constrained Decoding
• Tool Use & Function Calling
• Guardrails