Skip to content
AI360XpertBeta

Chat Templates

A chat template turns your list of labeled messages into the single flat string a model actually reads, using small marker tokens to say who said what.

A two-message list becomes one flat 91-character prompt when a template wraps each turn in control tokens and leaves the assistant header open
A two-message list becomes one flat 91-character prompt when a template wraps each turn in control tokens and leaves the assistant header open

Why Does This Exist?

You send a tidy list: a system note, a user question, an assistant reply, then another user question. Each entry names its role. A transformer has no slot for that list. It reads one flat sequence of token IDs and predicts what comes next, which is the whole game of next-token prediction.

So something has to translate. A chat template is that translator. It walks your message list in order, wraps each turn in short control tokens that name the role and mark where the turn ends, then joins everything into one string. Without it the model sees bare sentences with no idea which lines were standing instructions, which were user requests, and which were its own past answers. You have lost the roles before generation even starts.

This page is about that formatting step and nothing else. The words you put in the system slot belong to system prompts. What happens to the finished string belongs to tokenization. Where the model first learns the format belongs to instruction tuning. Here you learn the rule that turns roles into text.

Think of It Like This

A deli counter copying separate slips onto one ticket roll

You hand the counter three separate slips: a staff note, a customer order, a kitchen reply. The worker copies them onto one long receipt roll in order. Before each part the worker stamps STAFF, CUSTOMER, or KITCHEN, then draws a short line when that part ends. Anyone reading the roll later knows exactly who said what, even though the original slips are gone.

That worker is the template. The stamps are control tokens. The final blank line stamped for the kitchen with nothing written after it is the open assistant header. It tells the next cook exactly where to write.

Where it stops: rubber stamps do not enforce behavior. A rushed cook can still misread a stamp or scribble in the wrong section. Markers make roles visible, but they do not make anyone obey them. That gap is why a perfectly formatted prompt can still be talked into following a user instruction over a system rule, the failure named in prompt injection.

How It Actually Works

A chat template is a small formatting program, usually written in Jinja and stored beside the tokenizer. Call the message list MM, where each entry pairs a role word rr like system with its content text cc. Call the template rule TT. Rendering means computing P=T(M)P = T(M), where PP is one flat prompt string. Tokenizing then computes X=tok(P)X = tok(P), where toktok is the tokenizer and XX is the list of token IDs the model reads. Three objects, in that order: list, string, IDs.

Messages in, one string out

For each pair (r,c)(r, c) in MM, the template emits an opening marker, the role word, the content, and a closing marker. In the ChatML style that marker pair is <|im_start|> to open and <|im_end|> to close, so one turn looks like <|im_start|>user plus the text plus <|im_end|>. Those markers are control tokens: special IDs reserved at tokenizer training time that ordinary text splitting never produces. Your words can contain the letters u-s-e-r without confusion, because the model sees a dedicated marker ID, not the word user.

The trade is explicit. Markers cost budget and buy clarity. Every turn pays for its two markers plus its role word plus newlines, and a long back and forth with many short turns pays mostly for structure rather than content. You accept that tax because the alternative is worse: concatenated bare text where a user sentence that sounds like an instruction gets read as one.

The open assistant header

After the last complete turn the template appends one more opening marker with no closing marker: <|im_start|>assistant followed by a newline, then stops. That trailing header is the generation prompt. The model now does what it always does, predict the next token, and the only sensible continuation starts the assistant answer. When the first end marker appears in the output, that turn is done.

Do not use this machinery where it does not belong. A raw completion model with no chat tuning has learned no meaning for these markers, so wrapping its input in them wastes budget and adds IDs it never saw in training. Chat templates are for chat tuned models served through a chat API. Everywhere else, send plain text.

Worked example: 11 characters become 91

Take two messages. The system content is Be brief. at 9 characters. The user content is Hi at 2 characters. Content total is 9+2=119 + 2 = 11 characters. The role words are system at 6 and assistant at 9 for the final header plus user at 4.

Render with the rule above. The opening marker is 12 characters and the closing marker is 10 characters, counted straight from the strings. Turn one costs 12+6+1+9+10+1=3912 + 6 + 1 + 9 + 10 + 1 = 39 characters: open marker, role, newline, content, close marker, newline. Turn two costs 12+4+1+2+10+1=3012 + 4 + 1 + 2 + 10 + 1 = 30 characters by the same count. The open header costs 12+9+1=2212 + 9 + 1 = 22 characters with no closing marker. Add them: 39+30+22=9139 + 30 + 22 = 91 characters in PP.

So 11 content characters become a 91 character prompt. Overhead is 91−11=8091 - 11 = 80 characters: 56 in marker strings, 19 in role words, 5 in newlines. Control token occurrences total 5: 3 openings and 2 closings. The diagram above draws this exact string, and the snippet below rebuilds it. When you size a request against context windows, 91 is the number that counts, not 11. The same markers that save the roles are what push long chats over the limit, which is also why tool use messages need the same accounting once tool calls join the list.

Code

This rebuilds the worked example with plain string code, so you can see the overhead instead of taking it on faith.

def apply_chat_template(messages: list[dict[str, str]]) -> str:    """Render a message list into one flat prompt, ChatML style."""    parts: list[str] = []    for m in messages:        parts.append(f"<|im_start|>{m['role']}\n{m['content']}<|im_end|>\n")    parts.append("<|im_start|>assistant\n")  # left open: generation starts here    return "".join(parts)

messages = [    {"role": "system", "content": "Be brief."},    {"role": "user", "content": "Hi"},]prompt = apply_chat_template(messages)print(len("Be brief.") + len("Hi"))  # -> 11 content charactersprint(len(prompt))  # -> 91 rendered charactersprint(prompt.count("<|im_start|>"))  # -> 3 opening markersprint(prompt.endswith("<|im_start|>assistant\n"))  # -> True

Eleven characters in, 91 out. The three openings against two closings is the signature of a prompt left open for generation.

Watch Out For

Serving a model with another family's template

You point a Llama style checkpoint at a ChatML formatting function, or the reverse, because the server default looked fine. The model still runs. It just stops acting like itself: it rambles past stop points, answers in the voice of the user, loops on markers, or ends turns early. Nothing in the logs says template, so teams retune sampling first and lose a day.

The fix sits with the tokenizer, not the sampler. Use that checkpoint's own apply_chat_template and its bundled template from tokenizer_config.json, and pin the two together in your serving config. If you switch checkpoints, switch templates with them. Never hand copy marker strings between model families.

Budgeting context on the raw messages instead of the rendered prompt

You count the characters in the message list, see 11, and assume the request is tiny. The model counts the rendered 91 plus their tokenized form, markers included. Near the limit this gap turns into refused requests, silent truncation of early turns, or a system instruction squeezed out by overhead you never measured.

Count what the model reads. Render first with the exact template you serve, then tokenize that string and check its length. That rendered count is your budget number on every call.

The Quick Version

  • A chat template turns a labeled message list into one flat string by wrapping each turn in control tokens that mark role boundaries.
  • Control tokens are reserved marker IDs that plain text splitting never produces, so a role label cannot be confused with the word appearing in chat.
  • The rendered prompt ends with an open assistant header, and generation is simply continuation after that header.
  • In the worked example 11 content characters render to a 91 character prompt with 5 control token occurrences, so context math must use 91.
  • Inference must use the template the checkpoint trained on, because a mismatched template reads as garbled structure even when every word is right.