Tokenization：从文本到 ID

An application may represent a conversation as

[
  {"role": "system", "content": "You are a helpful assistant"},
  {"role": "user", "content": "How are you?"}
]

The Transformer does not receive that dictionary directly. A chat template first serializes the messages into the format seen during training. A ChatML-style rendering might be

<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

At inference time the final assistant marker has no answer after it; generation begins there. Other checkpoints may use [INST]...[/INST], header tokens, or entirely different formats. The chat template must match the checkpoint and tokenizer rather than being copied across model families.

To the model, system, user, assistant, and message boundaries eventually become token IDs. If <|im_start|> is registered as a special token, it normally maps to one ID; otherwise it may be split into ordinary pieces. Whether a role name occupies one token or several also depends on the vocabulary.

The architecture contains no hard-coded rule that says “system has highest priority.” The model learns the usual effect of these symbols from repeated combinations of role markers, message content, and target responses in its training data.

Do not copy numeric IDs from a screenshot. Special-token strings, IDs, stop sets, and embeddings are part of a specific tokenizer/checkpoint protocol and must be loaded again when the model changes.

The complete input pipeline is

$$ \boxed{ \text{structured messages} \xrightarrow{\text{chat template}} \text{serialized token text} \xrightarrow{\text{tokenizer}} \text{token IDs} \xrightarrow{\text{embedding table}} \text{vectors} } $$

Libraries often wrap template rendering and tokenization in one function, but their conceptual responsibilities remain distinct. The template determines the ordering of roles and message boundaries; the tokenizer selects vocabulary units; the embedding table turns integers into model-width vectors.

A chat template does not alter the attention equation. It gives an ordinary token sequence a learnable grammatical structure.

Avoid duplicating special tokens in actual code. In Transformers v5.20.0 (checked October 2026), if you render with apply_chat_template(tokenize=False) and tokenize separately, use add_special_tokens=False; tokenize=True combines the two steps and, because return_dict now defaults to True in v5, returns a dict with input_ids rather than a bare list of ids. Official guidance