Before deriving every formula, start with three questions: what does this component compute, why does it work this way, and what does it leave unresolved?
After each note, try following one sentence through the computation. Explaining that path is more useful than remembering a list of model names.
- Tokenization: split text into units, assign IDs, then map them to vectors the model can compute with.
- Recurrent networks: RNN and LSTM: update a state as the input arrives; LSTM gates control what to keep and forget.
- Sequence-to-sequence (Seq2Seq): encode the input, then generate the output; attention lets each step revisit relevant inputs.
- Multi-head attention, residual connections, and normalization: take the Transformer apart first—how attention reads across positions, and how residuals and normalization keep deep networks trainable.
- Vanilla Transformer: replace recurrent state updates with attention between positions.
- After the Transformer, choose a task: BERT restores hidden tokens using both contexts and fine-tunes for tasks such as classification; Decoder-only generates from a prefix, one token at a time.
flowchart LR
A["Discrete input<br/>Token"] --> B["Recurrent state<br/>RNN / LSTM"]
B --> C["Conditional generation<br/>Seq2Seq"]
C --> D["Parallel attention<br/>Transformer"]
D --> E["Understand supplied text<br/>BERT"]
D --> F["Continue from a prefix<br/>Decoder-only"]
Once the main line makes sense, go to the deep dives to fill in the math; if you prefer to run code first, go straight to the build-it-yourself labs.
Fill in unfamiliar foundations
- Embeddings and similarity: distinguish input embeddings, hidden states, and retrieval vectors; compare dot product with cosine.
- FFN and SwiGLU: follow feature transformations after attention, gate gradients, and matched parameter budgets.
- One training step: connect tokens, masks, loss, and parameter updates in one computation.
- CLIP alignment: apply these ideas to images and text with a 3 × 3 example.