Build-it-yourself labs: from symbols to generation

Type: lab index · Runtime: CPU is enough · Last reviewed: 2026-10

I like to write the same thing twice. The first time without PyTorch, forcing myself to see where every number comes from; the second time handing it over to tensors, modules, and autograd so that it actually learns.

High-level APIs are of course not a bad thing. But if every core computation is wrapped from the very start, then when a shape, mask, or state goes wrong later, it is hard to locate why training does not converge.

If views, broadcasting, or autograd are still unfamiliar, start with PyTorch: tensors to training. Small examples connect storage, gradients, and a complete training loop before you return here to build models.

Stage one: verify the computation with pure Python and NumPy

These first two files do not use autograd. Every hidden state, gate, and attention weight can be traced directly from the equation to the array, which makes them the place to see the mechanism clearly before the training loop drowns it out.

python tokenizer_from_scratch.py
python sequence_numpy.py

Stage two: train it with PyTorch

These implementations do not call nn.MultiheadAttention or F.scaled_dot_product_attention. PyTorch supplies tensors, autograd, and basic pieces such as Linear, Embedding, LayerNorm, cross-entropy, and the optimizer; attention, masking, and the overall structure are written out line by line.

Individual notes also link smaller check scripts from their "Verify it" sections, for example attention_numpy.py (multi-head attention in pure NumPy), attention_torch.py (the hand-written version checked against F.scaled_dot_product_attention and nn.MultiheadAttention, so the rule above does not apply to it), and norm_compare.py (reduction axes of the three normalizations and batch size 1).

Quick verification

python test_learning_path.py
python test_model.py

test_learning_path.py checks the tokenizer round-trip, the RNN/LSTM shapes, that the upper triangle of causal attention is strictly 0, and the output contract of the two PyTorch cells.

Suggested run order

tokenizer_from_scratch.py
        ↓
sequence_numpy.py
        ↓
sequence_torch.py --task delay
        ↓
sequence_torch.py --task reverse
        ↓
vanilla_demo.py
        ↓
test_model.py → train.py

Why keep two implementations

The framework-free version is good for answering "where does every quantity in the equation come from"; the PyTorch version is good for answering "how are parameters registered, how do gradients flow, and how is the training loop organized." Look only at the first and you stop at a forward-pass demo; look only at the second and it is easy to misdiagnose shape, mask, and state bugs as framework problems.

Four Transformer bugs that still "seem to run"

1. Causality: change token $t$, and the logits at positions $<t$ must stay strictly unchanged.
2. KV cache equivalence: incremental decoding must reproduce the one-shot forward pass.
3. RoPE relativity: the positional part of the attention score depends only on relative distance.
4. GQA grouping: query heads must map to the correct KV group.

make_figures.py builds the decision-boundary and hidden-space figures from real trained models; make_norm_figures.py draws the normalization-axis schematic and a measured residual-gradient plot; make_arch_figures.py draws schematic architecture diagrams (Transformer block, the three attention sites, KV cache), not measurements.