Tokenization: text to IDs

The same sentence can become different ID sequences under different tokenizers. If a toy tokenizer splits playing into play and ing, the model receives two IDs, not one English word. That choice affects sequence length, compute, and how unfamiliar words are represented. We will follow the text through tokenization and the embedding lookup. The model only ever sees token IDs; change the split, and the sequence length, the compute, and the units the model sees all change with it.

The model receives token IDs

We say all the time that a model “reads” a sentence, but it has never actually seen text. A tokenizer first splits the string into tokens from a finite vocabulary and then maps them to integer IDs; from start to finish, those numbers are all the model really receives.

Step 1: how text is split into tokens

The segmentation and IDs below are illustrative, not output from a particular tokenizer:

"unbelievable!" → ["un", "believ", "able", "!"] → [431, 9821, 612, 5]

The model uses an embedding table to look each ID up as a vector:

$$x_t = E[\text{token\_id}_t], \qquad E \in \mathbb{R}^{|V| \times d}$$

$|V|$ is the vocabulary size and $d$ is the model dimension. The tokenizer decides how long the sequence is; the embedding decides which continuous vector each discrete symbol starts from.

Step 2: why not just split on words

The intuition is reasonable, but a real-world word list has no end: new names, spelling variants, code, emoji, and other languages cannot all be listed. Making every character a token can produce longer sequences; characters absent from the base alphabet can still be unknown inputs.

A subword tokenizer is the compromise between the two: frequent fragments stay whole, and rare words are split into smaller units.

Step 3: how BPE merges common fragments

Byte Pair Encoding knows nothing about word roots and nothing about grammar. It just keeps asking: which two adjacent symbols appear together most often? Then it glues them together.

l o w </w>
l o w e r </w>

most frequent pair: (l, o)  →  lo
next round:         (lo, w) →  low

Assume each word occurs once. In the first round, (l, o) and (o, w) both occur twice; this example breaks the tie in favor of (l, o). Training learns an ordered list of merge rules; encoding applies them in the same order. BPE is not looking up English word roots; it is compressing recurring string patterns. Original BPE paper, §3.2

For two worked merges and the differences between BPE, WordPiece, Unigram, and byte fallback, continue to tokenizer algorithms. This page first completes the input pipeline.

Four objects that are easy to confuse

1. Vocabulary: the static mapping between tokens and IDs.
2. Merge rules / model: how raw symbols are combined into tokens.
3. Normalizer / pre-tokenizer: how case, Unicode, whitespace, and the like are handled before text reaches the model.
4. Special tokens: BOS, EOS, PAD, and the role boundaries in chat templates.

decode(encode(text)) should normally recover the text, but normalization can make the round trip not byte-for-byte reversible. PAD exists only to align sequences within a batch and should not be treated as content by the model; EOS is the real signal that generation should stop.

How a conversation becomes a string of IDs the model can read

1. Chat templates serialize role structure

An application may represent a conversation as

[
  {"role": "system", "content": "You are a helpful assistant"},
  {"role": "user", "content": "How are you?"}
]

The Transformer does not receive that dictionary directly. A chat template first serializes the messages into the format seen during training. A ChatML-style rendering might be

<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

At inference time the final assistant marker has no answer after it; generation begins there. Other checkpoints may use [INST]...[/INST], header tokens, or entirely different formats. The chat template must match the checkpoint and tokenizer rather than being copied across model families.

2. Special tokens are still vocabulary symbols

To the model, system, user, assistant, and message boundaries eventually become token IDs. If <|im_start|> is registered as a special token, it normally maps to one ID; otherwise it may be split into ordinary pieces. Whether a role name occupies one token or several also depends on the vocabulary.

The architecture contains no hard-coded rule that says “system has highest priority.” The model learns the usual effect of these symbols from repeated combinations of role markers, message content, and target responses in its training data.

Do not copy numeric IDs from a screenshot. Special-token strings, IDs, stop sets, and embeddings are part of a specific tokenizer/checkpoint protocol and must be loaded again when the model changes.

3. The full boundary: messages → template → IDs → embeddings

The complete input pipeline is

$$ \boxed{ \text{structured messages} \xrightarrow{\text{chat template}} \text{serialized token text} \xrightarrow{\text{tokenizer}} \text{token IDs} \xrightarrow{\text{embedding table}} \text{vectors} } $$

Libraries often wrap template rendering and tokenization in one function, but their conceptual responsibilities remain distinct. The template determines the ordering of roles and message boundaries; the tokenizer selects vocabulary units; the embedding table turns integers into model-width vectors.

A chat template does not alter the attention equation. It gives an ordinary token sequence a learnable grammatical structure.

Avoid duplicating special tokens in actual code. In Transformers v5.20.0 (checked October 2026), if you render with apply_chat_template(tokenize=False) and tokenize separately, use add_special_tokens=False; tokenize=True combines the two steps and, because return_dict now defaults to True in v5, returns a dict with input_ids rather than a bare list of ids. Official guidance

Connecting the output shapes to the neural network

Suppose a batch holds 4 texts, the padded length is 12, and the model dimension is 768:

token_ids       (B, T)    = (4, 12)
attention_mask  (B, T)    = (4, 12)
embeddings      (B, T, d) = (4, 12, 768)

The tokenizer ends at (B, T); the neural network begins at (B, T, d).

Experiment: compare different tokenizations

Run ../code/tokenizer_from_scratch.py. It trains a miniature BPE using only the Python standard library, with no extra dependencies. Change the training corpus and you can watch directly why the same sentence gets split into completely different tokens under another corpus.

This teaching implementation collapses whitespace, maps unseen characters to <unk>, and has no byte fallback. Its decode_tokens method reconstructs token strings, not a lossless round trip through IDs. Try repeated spaces and a character absent from the training corpus to see the distinction.

Self-check

Where to read next

Text has finally become vectors, but the positions still know nothing about each other. The next page is RNN and LSTM: if you can only read from left to right, where exactly should the past be stored?

Quick learning: what a tokenizer does at the model boundary