Skip to content
AI360XpertBeta

Constrained Decoding

Constrained decoding puts a mask over the model's next-token votes so banned tokens get zero chance. You still let the model choose the words, but the shape of the answer always parses.

Constrained decoding zeroes banned tokens before sampling so only schema legal tokens keep probability
Constrained decoding zeroes banned tokens before sampling so only schema legal tokens keep probability

Why Does This Exist?

You ask for JSON and it works in testing. Then one response in production opens with chatter, drops a trailing comma, or truncates a nested object, and your parser throws. The content was fine. The shape broke it.

The obvious fixes leave the hole open. Pleading in the prompt still samples from the full distribution, so an illegal token keeps nonzero chance on every step. Retry loops pay extra latency and can still fail. Regex repair helps after the fact but can't recover a structure that was never valid. If your app calls a function, writes a row, or drives a UI, you want shape guaranteed at generation time, not patched later.

This page covers that guarantee: how a schema becomes a per-step mask over next-token choices. For the broader menu of how tokens get picked, read decoding strategies. For the sampling step that runs after the mask, read temperature and sampling. For checking meaning rather than shape, read guardrails.

The one idea you need first: at each step the model emits logits, raw scores with one entry per token in its vocabulary, and sampling turns those scores into probabilities that sum to 1. The mask you are about to meet edits those scores before sampling runs.

Think of It Like This

A stencil over a keypad

You need a form filled with digits, but your friend is a chatty poet. You lay a stencil over the keyboard so only the digit keys show through. Your friend still picks which digits to press and in what order, so the content stays theirs, yet letters can't get through because the holes simply are not there.

Where it stops: a real stencil is fixed, while constrained decoding cuts a fresh stencil at every token from its place in the schema. And a tidy form can still hold wrong numbers, so shape says nothing about truth.

How It Actually Works

You hand the engine a schema, a regex, or a small grammar. The engine compiles it into a finite-state machine, a tracker that always knows where generation stands inside the allowed shape, such as inside an integer field or before a closing brace. At step tt that tracker outputs one thing: AtA_t, the allowed set, meaning the list of tokens that may legally come next. VV is the vocabulary, the full list of tokens the model can emit. zz in R∣V∣\mathbb{R}^{|V|} holds the logits, one raw score per token.

From schema to allowed tokens

Say the schema is {age:integer}\{ age: integer \} and you're inside the value. The tracker allows digit tokens and blocks everything else. One step later, after the digits, it allows a comma or a closing brace instead. The model never sees the schema directly. It only feels the effect: some tokens are available and the rest are gone for this step.

Mask, then sample

The engine takes zz and builds masked logits z′z'. In plain words, zi′z'_i equals ziz_i when token ii sits in AtA_t, and equals −∞-\infty when it does not. Then it applies the usual softmax:

pi′=exp⁡(zi′)∑jexp⁡(zj′)p'_i = \frac{\exp(z'_i)}{\sum_j \exp(z'_j)}

Here p′p' is the new probability distribution, exp⁡\exp is the exponential, and the sum runs over the whole vocabulary. Because exp⁡(−∞)\exp(-\infty) equals 00, banned tokens land on exactly 00 and the remaining mass renormalises to 11 over the allowed tokens. Only then does the engine sample, with greedy choice, temperature, top-p, or beam search all operating on p′p' rather than the raw scores. That order matters: the mask runs before the randomness, so no temperature can revive a banned token.

Worked example

Take four tokens 4242, 77, hellohello, and }\} with logits z=[2.0,1.0,3.0,0.0]z = [2.0, 1.0, 3.0, 0.0]. The schema allows only the two numbers at this step, so At={42,7}A_t = \{42, 7\}.

First the unconstrained picture, with m=3.0m = 3.0 subtracted for stability. Exponentiate to [exp⁡(−1),exp⁡(−2),exp⁡(0),exp⁡(−3)][\exp(-1), \exp(-2), \exp(0), \exp(-3)], about [0.368,0.135,1.0,0.050][0.368, 0.135, 1.0, 0.050]. The sum is about 1.5531.553. Divide and you get about [0.237,0.087,0.644,0.032][0.237, 0.087, 0.644, 0.032]. Left alone, greedy decoding picks hellohello, the fluent but illegal answer.

Now mask. Set the last two entries to −∞-\infty to get z′=[2.0,1.0,−∞,−∞]z' = [2.0, 1.0, -\infty, -\infty]. Exponentiate relative to 2.02.0 to get [1.0,0.368,0.0,0.0][1.0, 0.368, 0.0, 0.0]. The sum is about 1.3681.368. Divide and you get about [0.731,0.269,0.0,0.0][0.731, 0.269, 0.0, 0.0]. The 0.6440.644 on hellohello is gone. Greedy decoding now picks 4242, and sampling would split about 7373 to 2727 between the two legal numbers. Those are the numbers drawn in the diagram above.

The trade-off is direct. You buy shape you can parse, and you pay grammar build time plus a per-step check, plus occasional detours where the model must spell something in more tokens than it would have chosen. That price is worth it for APIs, tool calls, and form filling.

Don't use it for open-ended writing. Brainstorming, chat, and drafts have no shape worth enforcing, and a constraint there only narrows voice for no gain. Skip it too when the real requirement can't be written as a grammar, such as tone, helpfulness, or factual truth. A light post-check plus retry is cheaper there.

The method rests on one assumption: the grammar means what you meant and the engine maps characters to your model's tokens correctly. Byte-pair encoding splits text in ways a character regex cannot see, so a rule that looks right can starve a token or strand generation. When output looks oddly cut off under a constraint, suspect that mapping before you blame the model.

Code

The point of this snippet is the whole trick in four tokens: same logits, then a mask, then new probabilities. It uses only the standard library.

import math

def softmax(logits: list[float]) -> list[float]:    m = max(logits)    exps = [math.exp(x - m) for x in logits]    total = sum(exps)    return [e / total for e in exps]

logits = [2.0, 1.0, 3.0, 0.0]  # tokens: ["42", "7", "hello", "}"]allowed = [True, True, False, False]  # schema allows only the two numbers here
full = softmax(logits)masked_logits = [v if ok else float("-inf") for v, ok in zip(logits, allowed)]narrow = softmax(masked_logits)
print([round(p, 3) for p in full])  # -> [0.237, 0.087, 0.644, 0.032]print([round(p, 3) for p in narrow])  # -> [0.731, 0.269, 0.0, 0.0]print(max(range(4), key=lambda i: narrow[i]))  # -> 0

Unconstrained, index 22 (hellohello) wins with about 0.6440.644. After the mask, index 00 (4242) wins, with about 0.7310.731 against 0.2690.269 on the only other legal token.

Watch Out For

Your grammar and the tokenizer disagree

The symptom is grim: generation stalls, returns empty strings, or cuts words at strange joints only when the constraint is on. The cause sits one layer down. Your rule speaks characters while the model speaks tokens, and a token like 4242 may not exist alone or a leading space may be glued to the word. The engine then allows strings no token can spell.

Fix it at the boundary. Use a tokenizer-aware constraint library, test with your exact model tokenizer, and spell out spaces and newlines in the schema rather than leaving them implicit.

Parseable does not mean right

The symptom is quiet success: every response parses, so wrong values sail through. A valid JSON object can still carry a made-up age, a flipped sign, or a missing field your schema marked optional. Teams notice late because the parser stopped complaining.

Treat shape and truth as two checks. Keep the schema tight, validate ranges and types after parsing, and add a semantic layer such as guardrails when wrong-but-valid output would cause harm.

The Quick Version

  • Constrained decoding masks banned next tokens to zero before sampling, so output matches your schema by construction.
  • A schema or grammar compiles to a tracker that names the allowed token set at each step.
  • Masking runs before temperature and sampling, and probabilities renormalise to 1 over the allowed tokens.
  • With logits [2.0,1.0,3.0,0.0][2.0, 1.0, 3.0, 0.0] and only the first two tokens allowed, the 0.6440.644 on hello vanishes and 0.7310.731 plus 0.2690.269 stay on the legal pair.
  • A passing parse is not a correctness check, so still validate values and skip constraints for open-ended writing.