后训练（post-training）：模型预训练之后还要学什么？

A common high-level path is

$$ \boxed{ \text{Pre-Training} \longrightarrow \text{SFT} \longrightarrow \text{Preference / RL (RLHF, DPO, RLVR)} } $$

These labels describe functional stages; an organization need not save exactly three separate checkpoints. The useful distinction is that pre-training learns language, knowledge, and foundational capability from the data distribution; SFT teaches the model to invoke that capability through demonstrations; preference learning selects which of several plausible behaviors better serves the objective. RLVR needs no preference data: its reward comes from a program check (see verifiable rewards). It often sits in the same post-SFT stage as preference learning, but uses a different signal.

Pre-training precedes classic RLHF; it is not RLHF's first stage. In the full model lifecycle it is upstream, while the classic RLHF pipeline usually begins from an SFT policy.

GPT-style decoder-only models use causal language modeling:

$$ \mathcal L_{\text{causal}} =-\sum_t\log p_\theta(x_t\mid x_{<t}). $$

They see only the left context and naturally support autoregressive continuation. BERT-style encoders commonly use masked language modeling: selected tokens are hidden and recovered using both left and right context. That objective more naturally supports bidirectional representation, classification, and extraction than generation.

A pretrained Base Model can continue text, imitate textual patterns, and encode a great deal of knowledge, yet still fail to treat a user prompt as an instruction requiring a direct answer. It may continue the question, imitate a webpage, or ignore a constraint such as “use three sentences,” because its original objective predicts text rather than behaving as a chat assistant.

“Pre-training learns capability; SFT teaches the model how to use it” is a useful approximation, not a law. SFT can also change knowledge and capability, although its scale and objective usually emphasize behavioral shaping.

Post-training broadly covers training performed after initial foundation pre-training and is larger than RLHF:

$$ \text{Post-Training}\supset \{\text{SFT, preference optimization, RL, domain/safety tuning, distillation, tool use}\}. $$

Classic RLHF trains a Reward Model from preference pairs and then optimizes the policy with an RL method such as PPO. DPO directly optimizes the policy from chosen/rejected pairs, with neither a separate Reward Model nor an online RL loop. DPO is therefore not strictly reinforcement learning, although both approaches belong to preference alignment.

Keep two orthogonal axes separate:

• what is trained (objective / data): SFT, DPO, language modeling, distillation, RL;
• how parameters are updated (parameterization): full-parameter fine-tuning or LoRA / adapters.

The same SFT or DPO objective can use full updates or LoRA. Listing “LoRA” beside “SFT” as if both were training stages confuses the objective with the update mechanism.

A common domain-adaptation path is

General Base Model
  → continue next-token prediction on a large medical corpus
  → Medical Domain Base Model
  → Medical SFT / preference alignment

The middle step is Continued Pre-Training (CPT) or Domain-Adaptive Pre-Training. It occurs after initial pre-training in time but still uses a language-modeling objective and non-instruction domain text; SFT uses demonstrations to shape behavior.

The boundary between “pre-training” and “post-training” can therefore depend on whether the speaker is classifying by chronological stage or by training objective. When terminology is ambiguous, ask four concrete questions: what is the data, what is the objective, which checkpoint is the starting point, and what kind of model should result?

English terminology also differs: pre-training is a process, while a pre-trained model has completed that process. In casual usage, “pretrained LLM” may even refer broadly to a general-purpose model that has already been aligned.