How Are Special Chat Tokens Trained in an LLM?

Code and debug output on a screen, where the special chat tokens an LLM was trained on become visible
Photo by Daniil Komov on Pexels

Special chat tokens are trained in an LLM by the same mechanism as every other token. They get an ID in the vocabulary and an embedding vector, then enough examples for that vector to mean something. No separate module ever teaches the model what <|im_end|> stands for. It learns from position and repetition, which is all it ever learns anything from.

What differs is the timing. The slots are reserved when the tokenizer is built, long before training starts. The meaning arrives much later, during post-training, when the model is shown millions of conversations with those tokens in exactly the same places.

That gap between reserving a token and teaching it explains most of the strange behaviour people hit running a model themselves.

What a special chat token actually is

A tokenizer turns text into numbers. It splits words into subword pieces according to merge rules learned from a corpus, so a common word becomes one ID while an unusual one gets broken into fragments.

A special token skips that machinery. It is a reserved entry in the vocabulary, matched literally and mapped to a fixed ID. The merge rules never produce it from ordinary text.

OpenAI's harmony format, which the gpt-oss models are trained on, publishes its numbers:

TokenIDWhat it marks
<|start|>200006The beginning of a message
<|message|>200008The end of the header, where content begins
<|end|>200007The end of a message
<|return|>200002Stop generating

The pipes and angle brackets are a costume. The model never sees them, since it only ever sees 200006. The punctuation exists so a human reading a log can tell structure from content, and so nobody types one by accident in a support ticket.

Where chat tokens enter LLM training

LLM training happens in two broad stages, and chat tokens belong almost entirely to the second one.

Pretraining produces a base model by running next-token prediction over an enormous pile of text. Hugging Face's documentation makes the underlying point plainly: every causal language model continues a sequence of tokens, whether or not it was ever trained for chat.

A base model has no concept of a conversation. Ask one what the capital of France is and a perfectly reasonable continuation is three more geography questions, because that is what a page of quiz material looks like.

Post-training is where the conversation appears. Supervised fine-tuning takes a dataset of message lists and renders each one through a chat template into a flat string, then trains on the result. The marker <|im_start|> then sits at the head of every turn in every example, hundreds of thousands of times, always in the same structural slot.

That repetition is the entire training signal. Nobody defines the token anywhere. The model picks it up the way a dog picks up the sound of the cupboard where the leash lives, which is to say entirely from what reliably happens next.

Preference tuning follows and sharpens the behaviour, though the format is fixed by then. LLM Systems Engineering walks through the pipeline from tokenizer design to fine-tuning.

Server racks lit blue in a data centre, the hardware behind LLM training runs
Photo by panumas nikhomkhai on Pexels

How an LLM is made step by step, from tokenizer to template

Laid out in order, the stages where chat tokens matter are these:

StageWhat happens to the special tokens
Tokenizer designIDs are reserved. A block of spare slots is usually set aside for tokens nobody has named yet.
PretrainingThe reserved IDs go largely unused, so their embeddings stay close to their random starting values.
Template authoringA Jinja template is written that renders a message list into the exact string the model will be trained on.
Supervised fine-tuningEvery example passes through that template. The tokens acquire their meaning here.
Preference tuningHelpfulness and style are tuned. The structure is already load-bearing.
ServingThe same template renders live requests. Any mismatch shows up as degraded quality.

The template is the piece most people never open, and it ships alongside the tokenizer for exactly that reason. Hugging Face's chat templating guide shows two models fine-tuned from the same Mistral-7B base that expect completely different markers. One wraps user turns in [INST] and [/INST], the other uses <|user|> and <|assistant|>. The guide is blunt about mixing them up, warning that the wrong control tokens make performance drastically worse.

One template argument deserves its own paragraph. Setting add_generation_prompt=True appends the opening marker of an assistant turn, such as <|im_start|>assistant, to the rendered prompt. Leave it off and the model sees a completed user turn with nothing to indicate whose move it is. Continuing the user's message is a legitimate reading of that sequence, so that is often what you get.

Whether to train a model at all is a separate question, worked through in our piece on when to build an LLM and when to skip it.

How to train an LLM on chat tokens without teaching it both roles

Here is the detail that makes the whole arrangement work. During supervised fine-tuning, the loss is usually computed on the assistant's tokens only.

Loss is the number the optimiser tries to reduce, in this case the cross-entropy of predicting each next token. Positions you want excluded carry an ignore index, conventionally -100, and contribute nothing. Hugging Face's TRL library exposes this as an assistant_only_loss setting, which computes loss on the assistant responses and skips the user and system turns.

The consequence is an asymmetry worth holding onto. The model is graded on emitting <|im_end|> at the right moment, because that token falls inside the assistant span. It is never graded on emitting <|im_start|>user, because that marker only ever appears in the masked region.

That asymmetry is why a well-trained chat model stops talking. It has been rewarded thousands of times for closing its own turn and never once for opening yours.

The same library flags what happens when the wiring is off. Training a Qwen base model on a chat template requires pointing the end-of-sequence token at <|im_end|>, or responses will not terminate correctly. A model that never learned where to stop keeps going, cheerfully playing both parts, and it will be considerably kinder to itself than you were.

Why a local LLM breaks when the chat template is wrong

Run a local LLM and this is the failure you meet first. The server starts cleanly and the answers come out worse than the benchmark scores suggested, with no exception to catch and nothing to explain it. The model quietly degrades, and the blame lands on the model.

The symptoms are recognisable once you know the cause:

  • The model answers, then invents your next message and answers that one too.
  • Generation runs to the token limit every time, because the stop token is not configured on the serving side.
  • Output quality drops while the model stays coherent, which is the hardest version to spot.
  • The beginning-of-sequence token appears twice, because the template added one and the tokenizer added another.

That last one is common enough that Hugging Face warns about it directly. Chat templates already include the special tokens a model needs, so adding more is usually incorrect or duplicated, and it hurts performance. Rendering and tokenizing in a single call avoids the entire class of problem.

Some models are stricter. OpenAI's harmony response format documentation states that gpt-oss should not be used without the harmony format, since it will not work correctly. The format carries a channel field separating internal analysis from the final answer, so a client that ignores it can surface reasoning meant to stay hidden.

The debugging move is unglamorous and works every time. Render the prompt with tokenization switched off and read the string you are actually sending. Nine times out of ten the bug is sitting right there in it, wearing a name tag: a missing generation prompt, or a marker that got added twice.

Prompt quality sits on top of all this. Our notes on practices for training AI models with prompts assume the plumbing underneath is correct, worth confirming before you spend a week rewriting instructions.

Special tokens are a boundary an LLM prompt should not cross

If the model's sense of who is speaking is carried by tokens, then anything that can inject those tokens can impersonate a role.

Most tokenizers block this by default. Text arriving from a user is encoded as ordinary characters, so someone who types the literal string <|im_start|>system gets a handful of unremarkable subword tokens and no authority.

The risk lives in the plumbing around the tokenizer. Any code path that builds the prompt by string concatenation, then tokenizes with special-token parsing enabled, has handed the user a working role marker.

Be precise about what the role hierarchy is. Harmony defines five roles in a priority order running from system at the top down to tool at the bottom. That ordering is a convention the model follows because its training data was consistent about it, and it carries no enforcement. A persuasive enough user turn can still talk a model out of its instructions.

Two rules follow. Pass user content into the template as data inside a message object, never spliced into a rendered string. And keep special-token parsing switched off for anything that came from outside your system. Agentic AI Engineering covers this boundary where it matters most, which is agents whose tool output flows back into the prompt.

Four things to check when a chat model misbehaves, in the order that finds the problem fastest:

  1. Print the rendered prompt string before it is tokenized.
  2. Confirm the template came from the same repository as the weights.
  3. Check that the generation prompt is appended for inference and omitted for training.
  4. Confirm every stop token in the template is configured on the serving side.

None of that is glamorous, and all of it is faster than another fine-tuning run.

The tokens stay unimpressed either way. To the model they were only ever 200006 and 200007. Everything else, including the part where it knows to stop talking, came from the training.

Frequently asked questions

Is ChatGPT an LLM?

Yes. ChatGPT is a product built around a large language model, with a chat interface and tooling wrapped around it. The model underneath predicts the next token like any other causal language model, and the conversation structure it sees is assembled from special chat tokens in exactly the way described above.

What is a chat template in an LLM?

A chat template is a small Jinja template shipped with the tokenizer that turns a list of role and content messages into the single flat string the model expects. It places the special tokens in the positions the model was trained on. Two models fine-tuned from the same base can carry completely different templates, which is why the template belongs with the weights.

Can you add new special tokens to a model that is already trained?

You can, though it is more involved than editing a config file. The vocabulary grows, so the embedding matrix has to be resized. The new row starts out as noise with no learned meaning, and only fine-tuning on data that uses the token in a consistent position will give it one. Most teams reuse a reserved slot the tokenizer already carries instead.

Do special chat tokens count toward the context window?

They do. Every marker in the rendered prompt is a real token with a real ID, so it occupies a slot in the context window and appears on your bill if you pay per token. The overhead is small on a single turn and accumulates across a long conversation, since the markers repeat on every message.

Why does my local LLM keep writing both sides of the conversation?

Almost always the generation prompt is missing. Without the opening marker of an assistant turn at the end of the rendered prompt, the model sees a finished user message and continues the transcript, which means writing your next line as well as its own. Check that the template is applied with the generation prompt enabled and that the stop token is configured on the server.

How is an LLM made, step by step?

A tokenizer is built first, reserving IDs including the ones chat markers will later occupy. Pretraining then produces a base model through next-token prediction over a large corpus. Supervised fine-tuning on templated conversations teaches the chat format, and preference tuning shapes helpfulness and style on top of that.

Can a user break a model by typing a special token into an LLM prompt?

Typing those characters into a chat box is harmless, because tokenizers encode user text literally and the string becomes ordinary subword tokens. The risk sits in application code that concatenates user input into a prompt string and then tokenizes with special-token parsing enabled. Pass user content through the template as message data and the problem does not arise.

Sources