Decoder-only：自回归生成

After the user submits structured messages, the application uses a chat template and tokenizer to create token IDs with role boundaries. A typical prompt ends at the assistant-start marker:

<system> You are helpful <end>
<user> How are you? <end>
<assistant>

The model then repeats one loop: predict the next token, append it to the context, and predict again. Generation stops on an end-of-message or EOS token, another configured stop condition, or a maximum-length limit.

<assistant> → I → am fine → . → <end>

Role structure does not change the decoder equation. It makes “whose turn is next” part of the token context while the model continues ordinary causal next-token prediction.

For the second response, the sequence usually contains the system message, first user turn, first assistant turn, second user turn, and a new assistant-start marker. Standard full causal attention can read earlier tokens that remain in context and are not masked; sliding-window variants have narrower visibility. This lets the response use earlier dialogue.

This is not permanent memory independent of context. History must remain in the logical prefix: either supply its tokens again or reuse a matching KV cache and pass only uncached new tokens. History absent from both cannot be read. When the sequence exceeds the context window, the application must truncate, summarize, or retrieve important facts. Caching API, Transformers v5.20.0

KV cache stores already-computed K/V for the current inference sequence to avoid recomputation. It does not turn one session into a cross-session knowledge base or decide which history deserves long-term retention.

A typical pre-norm decoder block can be written as

$$ \begin{gathered} U=\operatorname{RMSNorm}(X),\\ H=X+\operatorname{Attention}(U). \end{gathered} $$

$$ \begin{gathered} G=\operatorname{RMSNorm}(H),\\ Y=H+\operatorname{SwiGLU}(G). \end{gathered} $$

The full path is RMSNorm → Q/K/V projections → RoPE on Q and K → causal attention → output projection → residual addition → RMSNorm → SwiGLU → a second residual addition. A final norm usually follows the entire stack before the vocabulary logits.

This is a common design, not a universal law. Relative to the 2017 encoder–decoder, modern LLMs are often decoder-only, use pre-norm instead of post-norm, RMSNorm instead of LayerNorm, RoPE instead of additive sinusoidal positions, SwiGLU instead of a ReLU FFN, and sometimes GQA or MQA instead of standard MHA.

As of October 2026, open models show three more common changes: Qwen3 and Gemma 3 add QK-norm on q and k; DeepSeek-V3 caches a 512-dimensional KV latent plus a separate 64-dimensional positional key with MLA; and in Qwen3.5-397B-A17B (February 2026), 45 of 60 layers are Gated DeltaNet layers with a fixed-size recurrent state and only 15 are Gated Attention layers, in a 3:1 pattern.

The changes listed in the previous card sit in different parts of the block. Each row records only what the part changes and the point most often stated wrong; the derivations live in the linked notes.

For long context, first ask which bottleneck you face:

• position distributions beyond the training length → RoPE scaling / frequency adjustment;
• quadratic computation and intermediate state of full attention → windowed / sparse connectivity or an efficient kernel;
• distant evidence cannot reach the query → global pathways, retrieval, or layer mixing.

The FFN row is where dimensions most often go wrong. This chapter writes a token as a row vector: the original FFN is $\operatorname{ReLU}(xW_1)W_2$ (bias omitted), and SwiGLU is

$$ \begin{gathered} g=\operatorname{SiLU}(xW_{\text{gate}}),\\ u=xW_{\text{up}},\\ \operatorname{SwiGLU}(x)=(g\odot u)W_{\text{down}}, \end{gathered} $$

with $W_{\text{gate}},W_{\text{up}}\in\mathbb R^{d\times h}$, $W_{\text{down}}\in\mathbb R^{h\times d}$, and $\operatorname{SiLU}(z)=z\sigma(z)$. The next card keeps this notation and walks through the whole block.

For layer-$l$ input $X_l$, first normalize and form Q, K, and V:

$$ U=\operatorname{RMSNorm}(X_l), $$

$$ \begin{gathered} Q=UW_Q\\ K=UW_K\\ V=UW_V. \end{gathered} $$

K and V may use GQA. The following attention equation is for one query head and its associated KV head; concatenate head outputs before the output projection. Next rotate only Q and K, then apply causal attention:

$$ \begin{gathered} Q'=\operatorname{RoPE}(Q)\\ K'=\operatorname{RoPE}(K), \end{gathered} $$

$$ \begin{gathered} S=Q'K'^\top/\sqrt{d_k},\\ A=\operatorname{softmax}(S+M_{\text{causal}}),\\ O=AV. \end{gathered} $$

FlashAttention can implement this step efficiently without changing the formula. Concatenate the outputs of all $h_q$ query heads, then apply the output projection and first residual connection:

$$ \begin{gathered} O_{\text{multi}}=\operatorname{Concat}(O_1,\ldots,O_{h_q}),\\ H=X_l+O_{\text{multi}}W_O. \end{gathered} $$

Then follow the second pre-norm branch:

$$ G=\operatorname{RMSNorm}(H), $$

$$ \begin{gathered} U_g=\operatorname{SiLU}(GW_{\text{gate}}),\\ U_u=GW_{\text{up}},\\ F=(U_g\odot U_u)W_{\text{down}}. \end{gathered} $$

$$ X_{l+1}=H+F. $$

Some models replace this SwiGLU FFN with MoE. Take the embedding output as $X_0$, so layer $l=0,\ldots,N-1$ maps $X_l$ to $X_{l+1}$. After all $N$ layers:

$$ \begin{gathered} H_{\text{final}}=\operatorname{RMSNorm}(X_N)\\ \text{logits}=H_{\text{final}}W_{\text{vocab}}. \end{gathered} $$

These terms therefore operate at different levels: RoPE changes positional relations in Q/K; GQA changes KV heads; KV cache stores historical state; FlashAttention optimizes the attention kernel; and MoE replaces the FFN.