Special chat tokens are trained in an LLM by the same mechanism as every other token. They get an ID in the vocabulary and an embedding vector, then enough examples for that vector to mean something. No separate module ever teaches the model what <|im_end|> stands for. It learns from position and repetition, which is all it ever learns anything from.
What differs is the timing. The slots are reserved when the tokenizer is built, long before training starts. The meaning arrives much later, during post-training, when the model is shown millions of conversations with those tokens in exactly the same places.
That gap between reserving a token and teaching it explains most of the strange behaviour people hit running a model themselves.
What a special chat token actually is
A tokenizer turns text into numbers. It splits words into subword pieces according to merge rules learned from a corpus, so a common word becomes one ID while an unusual one gets broken into fragments.
A special token skips that machinery. It is a reserved entry in the vocabulary, matched literally and mapped to a fixed ID. The merge rules never produce it from ordinary text.
OpenAI's harmony format, which the gpt-oss models are trained on, publishes its numbers:
| Token | ID | What it marks |
|---|---|---|
<|start|> | 200006 | The beginning of a message |
<|message|> | 200008 | The end of the header, where content begins |
<|end|> | 200007 | The end of a message |
<|return|> | 200002 | Stop generating |
The pipes and angle brackets are a costume. The model never sees them, since it only ever sees 200006. The punctuation exists so a human reading a log can tell structure from content, and so nobody types one by accident in a support ticket.
Where chat tokens enter LLM training
LLM training happens in two broad stages, and chat tokens belong almost entirely to the second one.
Pretraining produces a base model by running next-token prediction over an enormous pile of text. Hugging Face's documentation makes the underlying point plainly: every causal language model continues a sequence of tokens, whether or not it was ever trained for chat.
A base model has no concept of a conversation. Ask one what the capital of France is and a perfectly reasonable continuation is three more geography questions, because that is what a page of quiz material looks like.
Post-training is where the conversation appears. Supervised fine-tuning takes a dataset of message lists and renders each one through a chat template into a flat string, then trains on the result. The marker <|im_start|> then sits at the head of every turn in every example, hundreds of thousands of times, always in the same structural slot.
That repetition is the entire training signal. Nobody defines the token anywhere. The model picks it up the way a dog picks up the sound of the cupboard where the leash lives, which is to say entirely from what reliably happens next.
Preference tuning follows and sharpens the behaviour, though the format is fixed by then. LLM Systems Engineering walks through the pipeline from tokenizer design to fine-tuning.
How an LLM is made step by step, from tokenizer to template
Laid out in order, the stages where chat tokens matter are these:
| Stage | What happens to the special tokens |
|---|---|
| Tokenizer design | IDs are reserved. A block of spare slots is usually set aside for tokens nobody has named yet. |
| Pretraining | The reserved IDs go largely unused, so their embeddings stay close to their random starting values. |
| Template authoring | A Jinja template is written that renders a message list into the exact string the model will be trained on. |
| Supervised fine-tuning | Every example passes through that template. The tokens acquire their meaning here. |
| Preference tuning | Helpfulness and style are tuned. The structure is already load-bearing. |
| Serving | The same template renders live requests. Any mismatch shows up as degraded quality. |
The template is the piece most people never open, and it ships alongside the tokenizer for exactly that reason. Hugging Face's chat templating guide shows two models fine-tuned from the same Mistral-7B base that expect completely different markers. One wraps user turns in [INST] and [/INST], the other uses <|user|> and <|assistant|>. The guide is blunt about mixing them up, warning that the wrong control tokens make performance drastically worse.
One template argument deserves its own paragraph. Setting add_generation_prompt=True appends the opening marker of an assistant turn, such as <|im_start|>assistant, to the rendered prompt. Leave it off and the model sees a completed user turn with nothing to indicate whose move it is. Continuing the user's message is a legitimate reading of that sequence, so that is often what you get.
Whether to train a model at all is a separate question, worked through in our piece on when to build an LLM and when to skip it.
How to train an LLM on chat tokens without teaching it both roles
Here is the detail that makes the whole arrangement work. During supervised fine-tuning, the loss is usually computed on the assistant's tokens only.
Loss is the number the optimiser tries to reduce, in this case the cross-entropy of predicting each next token. Positions you want excluded carry an ignore index, conventionally -100, and contribute nothing. Hugging Face's TRL library exposes this as an assistant_only_loss setting, which computes loss on the assistant responses and skips the user and system turns.
The consequence is an asymmetry worth holding onto. The model is graded on emitting <|im_end|> at the right moment, because that token falls inside the assistant span. It is never graded on emitting <|im_start|>user, because that marker only ever appears in the masked region.
That asymmetry is why a well-trained chat model stops talking. It has been rewarded thousands of times for closing its own turn and never once for opening yours.
The same library flags what happens when the wiring is off. Training a Qwen base model on a chat template requires pointing the end-of-sequence token at <|im_end|>, or responses will not terminate correctly. A model that never learned where to stop keeps going, cheerfully playing both parts, and it will be considerably kinder to itself than you were.
Why a local LLM breaks when the chat template is wrong
Run a local LLM and this is the failure you meet first. The server starts cleanly and the answers come out worse than the benchmark scores suggested, with no exception to catch and nothing to explain it. The model quietly degrades, and the blame lands on the model.
The symptoms are recognisable once you know the cause:
- The model answers, then invents your next message and answers that one too.
- Generation runs to the token limit every time, because the stop token is not configured on the serving side.
- Output quality drops while the model stays coherent, which is the hardest version to spot.
- The beginning-of-sequence token appears twice, because the template added one and the tokenizer added another.
That last one is common enough that Hugging Face warns about it directly. Chat templates already include the special tokens a model needs, so adding more is usually incorrect or duplicated, and it hurts performance. Rendering and tokenizing in a single call avoids the entire class of problem.
Some models are stricter. OpenAI's harmony response format documentation states that gpt-oss should not be used without the harmony format, since it will not work correctly. The format carries a channel field separating internal analysis from the final answer, so a client that ignores it can surface reasoning meant to stay hidden.
The debugging move is unglamorous and works every time. Render the prompt with tokenization switched off and read the string you are actually sending. Nine times out of ten the bug is sitting right there in it, wearing a name tag: a missing generation prompt, or a marker that got added twice.
Prompt quality sits on top of all this. Our notes on practices for training AI models with prompts assume the plumbing underneath is correct, worth confirming before you spend a week rewriting instructions.
Special tokens are a boundary an LLM prompt should not cross
If the model's sense of who is speaking is carried by tokens, then anything that can inject those tokens can impersonate a role.
Most tokenizers block this by default. Text arriving from a user is encoded as ordinary characters, so someone who types the literal string <|im_start|>system gets a handful of unremarkable subword tokens and no authority.
The risk lives in the plumbing around the tokenizer. Any code path that builds the prompt by string concatenation, then tokenizes with special-token parsing enabled, has handed the user a working role marker.
Be precise about what the role hierarchy is. Harmony defines five roles in a priority order running from system at the top down to tool at the bottom. That ordering is a convention the model follows because its training data was consistent about it, and it carries no enforcement. A persuasive enough user turn can still talk a model out of its instructions.
Two rules follow. Pass user content into the template as data inside a message object, never spliced into a rendered string. And keep special-token parsing switched off for anything that came from outside your system. Agentic AI Engineering covers this boundary where it matters most, which is agents whose tool output flows back into the prompt.
Four things to check when a chat model misbehaves, in the order that finds the problem fastest:
- Print the rendered prompt string before it is tokenized.
- Confirm the template came from the same repository as the weights.
- Check that the generation prompt is appended for inference and omitted for training.
- Confirm every stop token in the template is configured on the serving side.
None of that is glamorous, and all of it is faster than another fine-tuning run.
The tokens stay unimpressed either way. To the model they were only ever 200006 and 200007. Everything else, including the part where it knows to stop talking, came from the training.