Vanilla Transformer

For translation, an encoder can read the entire source sentence at once. A decoder predicting the next word cannot look ahead at the unwritten translation. The original Transformer uses different attention paths and masks for those two conditions. Following one sentence through the encoder and decoder gives each block a concrete role. The original Transformer moves information between positions with attention and transforms each position with an FFN; the encoder and decoder differ mainly in their masks and in where attention reads from.

Move information, then transform it

Forget the arrow-heavy architecture diagram for a moment. A Transformer layer repeats only two operations: attention goes to other positions to fetch information; the FFN stays at the current position and processes it. The original architecture is still an encoder–decoder, but recurrence is removed completely.

A layer has only two jobs

1. Attention mixing exchanges information across token positions.
2. Channel mixing / FFN transforms each token's channels independently.

Residual connections and normalization wrap those two operations. Stacking many layers still means repeating “go out and find information → come back and process it.” This is much easier to remember than the full block diagram.

The three attention sites ask different questions

All three use the same scaled dot-product attention:

$$\text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V$$

$M$ is the mask: it adds 0 at allowed positions and $-\infty$ at forbidden positions.

Padding and a padding mask are different operations. For input lengths 3 and 5, a collator can pad both to 5; the mask then excludes the first input's two placeholder keys. The mask does not add tokens, and the batch need not use the model's maximum length.

For valid decoder self-attention queries, a key must be both non-padding and not in the future. With query rows and key columns, the strictly upper triangle is blocked; the diagonal stays visible because the current input predicts the next token. Cross-attention can read the full source sentence, not just a source prefix matching the translation position. These are the original architecture's rules; packing and local attention can impose further restrictions. Transformer §3.2.3

Why the architecture scales

• Parallel training: $Q/K/V$ for every position are computed at once.
• Shorter paths: positions allowed by the mask can communicate through one attention layer.
• One uniform structure: self-attention and cross-attention differ only in where the tensors come from.

The cost is a $T\times T$ self-attention score matrix, so the time and memory of a standard implementation grow approximately quadratically with sequence length.

The missing information: attention does not know order

Without positional information or an order-dependent mask, self-attention is permutation-equivariant: reordering inputs reorders outputs. The original Transformer adds fixed sinusoidal positions to scaled token embeddings, with the scale absorbed into E here:

$$z_t = E[x_t] + PE_t$$

This describes attention without order constraints, not a causal decoder with no sense of earlier and later. The causal mask already restricts visibility; positional encoding supplies more explicit position information.

From token IDs to the next token: the complete path

Use a tiny English-to-German example to connect the entire forward path of the 2017 Transformer. The 512-dimensional states, 8 heads, and 2048-dimensional FFN below refer to the paper's base configuration, not every original configuration. Look only at shapes first, then at each module's job.

1. How the source enters the encoder

The English sentence first goes through the tokenizer:

[I, love, you] -> [21, 135, 48]

A token ID is only an integer index into the vocabulary; it carries no notion of distance or magnitude. The embedding layer uses these IDs to look up vectors in a matrix $E$. With source length $S=3$ and the original $d_{\text{model}}=512$:

$$X_{\text{emb}}\in\mathbb{R}^{S\times d_{\text{model}}} =\mathbb{R}^{3\times512}.$$

The paper actually multiplies the embeddings by $\sqrt{d_{\text{model}}}$ and then adds a fixed sinusoidal positional encoding:

$$X=\sqrt{d_{\text{model}}}\,X_{\text{emb}}+PE,$$

$$PE(pos,2i)=\sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right),\qquad PE(pos,2i+1)=\cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right).$$

There is no need to memorize the constants. What to remember is that $PE$ is added to the embedding, not concatenated, so the shape remains $(S,512)$. Every position receives a unique but regular set of phases, which is what lets the model tell I love you apart from a sentence with the same tokens in a different order. Dropout is applied after the sum.

Attention dimensions and edge cases

For a single attention head, the more general shapes are

$$Q\in\mathbb R^{L_q\times d_k},\qquad K\in\mathbb R^{L_{kv}\times d_k},\qquad V\in\mathbb R^{L_{kv}\times d_v}.$$

Therefore

$$QK^\top\in\mathbb R^{L_q\times L_{kv}},\qquad \operatorname{softmax}(QK^\top)V\in\mathbb R^{L_q\times d_v}.$$

There are two real shape constraints here. Q and K must both have final dimension $d_k$ so that their dot product is defined. K and V must both have sequence length $L_{kv}$, because every key has to correspond to one value that gets read. $d_v$ need not equal $d_k$; the number of output channels is determined by $d_v$. Setting $d_k=d_v=64$ per head in the original model was a design choice, not a mathematical requirement of attention.

The score is divided by $\sqrt{d_k}$, the Q/K matching width, not $d_v$; the variance derivation and the example showing that saturation depends on score differences are in multi-head attention.

The individual coordinates of $W_Q,W_K,W_V$ have no fixed human meaning, but the three projections play different roles: Q expresses “what I am looking for,” K expresses “how I can be matched,” and V expresses “what content is passed on after a match.” Hence

$$QK^\top=\text{addressing / routing},\qquad V=\text{retrieved content}.$$

Separate $W_Q,W_K$ let “$i$ queries $j$” and “$j$ queries $i$” score differently, and a separate $W_V$ decouples “how the information is found” from “what is read once it is found”; what forcing $Q=K$ does is covered in the multi-head attention interview questions.

2. What one encoder layer computes

The original model stacks 6 encoder layers with identical structure. Each layer is

X -> multi-head self-attention -> Add & Norm -> FFN -> Add & Norm

In self-attention, $Q/K/V$ all come from the current source states, which is why it is called self-attention:

$$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$$

The original has $h=8$ heads, so each head has

$$d_k=d_v=\frac{512}{8}=64.$$

For one head, the attention formula from the previous section does four things in order:

$$S=QK^\top,\qquad \widetilde S=\frac{S}{\sqrt{d_k}},\qquad A=\operatorname{softmax}_{\text{key}}(\widetilde S),\qquad O=AV.$$

$S_{ij}$ is the match score between the query at source position $i$ and the key at position $j$; softmax is computed along the key dimension of each row. For example, when some head in some layer updates love, it might obtain

$$A_{\text{love}}=[0.4,\ 0.1,\ 0.5],$$

and therefore

$$o_{\text{love}}=0.4v_I+0.1v_{\text{love}}+0.5v_{\text{you}}.$$

These are the read weights of this one head in this one layer. Do not read them as the model as a whole paying 40% “semantic attention.” Each of the 8 heads has its own projections; at the end they are concatenated and passed through an output projection:

$$\operatorname{MHA}(X)= \operatorname{Concat}(\operatorname{head}_1,\ldots,\operatorname{head}_8)W^O.$$

The original is post-norm. Each sublayer's output goes through dropout first, is then added to the input through the residual connection, and LayerNorm comes last:

$$H=\operatorname{LN}\big(X+\operatorname{Dropout}(\operatorname{MHA}(X))\big).$$

Each token then passes independently through the same set of FFN parameters:

$$\operatorname{FFN}(h)=\operatorname{ReLU}(hW_1+b_1)W_2+b_2,$$

$$512\rightarrow2048\rightarrow512.$$

Attention is responsible for exchanging information between tokens; the FFN does not cross tokens and only processes the channel information each position has already collected. The second Add & Norm is

$$Y=\operatorname{LN}\big(H+\operatorname{Dropout}(\operatorname{FFN}(H))\big).$$

Repeating this for 6 layers gives the encoder memory:

$$C=\operatorname{Encoder}(X)\in\mathbb{R}^{S\times512}.$$

3. Why the target is shifted right

If the correct translation is

target labels: [ich, liebe, dich, <EOS>]
decoder input: [<BOS>, ich, liebe, dich]

the two are offset by one position, which is what lets position $t$ learn to “predict token $t$ from the tokens before it.” During training the whole correct target is known and can be fed in parallel in one pass; the causal mask is what prevents peeking at the future.

The decoder input goes through embedding, positional encoding, and dropout in the same way. The source embedding, the target embedding, and the output projection can be separate as a matter of structure; the original paper, however, uses a joint vocabulary and shares the weights of both embeddings and the pre-softmax projection, while multiplying the embeddings by $\sqrt{d_{\text{model}}}$. So “a translation model necessarily uses three different sets of weights” is not a fact about the original Transformer.

4. Why one decoder layer has three sublayers

The original also stacks 6 decoder layers; each has one more sublayer than an encoder layer, the cross-attention:

target states
  -> masked self-attention -> Add & Norm
  -> cross-attention       -> Add & Norm
  -> FFN                   -> Add & Norm

The first sublayer is self-attention over the target, but a causal mask is added to the scores. For 4 positions, the allowed region is the lower triangle:

$$ \begin{bmatrix} s_{11}&-\infty&-\infty&-\infty\\ s_{21}&s_{22}&-\infty&-\infty\\ s_{31}&s_{32}&s_{33}&-\infty\\ s_{41}&s_{42}&s_{43}&s_{44} \end{bmatrix}. $$

Since $e^{-\infty}=0$, masked positions have weight exactly 0 after softmax; position $i$ can read only target positions 1 through $i$. Write $Y\in\mathbb{R}^{T\times512}$ for the target-side input after embedding, positional encoding, and dropout (not the encoder-layer output $Y$ from step 2). The first sublayer produces

$$H_1=\operatorname{LN}\big(Y+ \operatorname{Dropout}(\operatorname{MaskedMHA}(Y))\big).$$

The second sublayer, cross-attention, is where the decoder first touches the English source:

$$Q=H_1W_Q,\qquad K=CW_K,\qquad V=CW_V.$$

That is, Q comes from the decoder, while K/V come from the encoder memory. With target length $T$ and source length $S$, the attention weights have shape $(T,S)$ and need not be square. For example, when preparing to generate “liebe”, the decoder state can use its query to read the values in the encoder that relate to love:

$$H_2=\operatorname{LN}\big(H_1+ \operatorname{Dropout}(\operatorname{CrossAttention}(H_1,C))\big).$$

The third sublayer is the same position-wise FFN as in the encoder:

$$H_3=\operatorname{LN}\big(H_2+ \operatorname{Dropout}(\operatorname{FFN}(H_2))\big).$$

After 6 layers we have

$$D\in\mathbb{R}^{T\times512}.$$

5. From decoder states to vocabulary probabilities

The 512-dimensional vector at each position is projected onto the target vocabulary. With vocabulary size $V=30{,}000$:

$$Z=DW_{\text{vocab}}+b,\qquad W_{\text{vocab}}\in\mathbb{R}^{512\times30{,}000},$$

so $Z\in\mathbb{R}^{T\times30{,}000}$. Each row holds that position's logits over all tokens; only softmax turns them into a conditional probability:

$$p(y_t\mid y_{<t},x)=\operatorname{softmax}(Z_t).$$

Training uses cross-entropy; the original paper also applies label smoothing with $\epsilon_{ls}=0.1$ rather than a purely one-hot target. Inference uses greedy decoding, beam search, or sampling to pick one token, then feeds it back into the decoder and continues generating. Softmax itself only provides probabilities; it does not require choosing the argmax.

The whole path compresses into one line:

$$ \text{source IDs}\rightarrow C\rightarrow \underbrace{p(y_1\mid x),\ p(y_2\mid y_1,x),\ldots}_{ \text{masked self-attention reads the prefix; cross-attention reads the source}} $$

Do not conflate the 2017 architecture with a modern decoder-only LLM

The original is the best place to learn what each of the three attention sites is responsible for. The modern version is the best reference for understanding how today's language models are trained and served.

Verify it: run the complete example

• Quick end-to-end run: ../code/vanilla_demo.py
• How the parts fit, the RoPE code, and checks for a from-scratch implementation: The Transformer architecture
• Attention forward pass without PyTorch: ../code/sequence_numpy.py

Self-check

Next

Choose a task next: BERT uses the encoder for bidirectional pretraining and text understanding; Decoder-only expresses conditional generation, dialogue, and code as autoregressive prediction over a token stream. You need not read every model in historical order. Original configurations and weight-sharing conventions here follow Attention Is All You Need.

Quick learning: the 2017 Transformer from input to next token