Post-training changes how the model behaves
Pretraining teaches a model what the world usually contains. Post-training teaches it how to behave and act when it faces a particular kind of task.
Want the RL foundations first? Start with the separate Deep RL route: MDPs, Bellman, policy gradients, DQN, SAC, and model-based RL. Then return here for their language-model applications.
How to read this series
First separate what supplies supervision from which parameters may change. If you already understand SFT, go straight to preference learning and return for unfamiliar concepts; there is no need to restart from page one.
I. Foundations: why more teaching is needed
- Why pretraining isn't enough (this page) — what learning problem SFT, preference learning, and RL each solve
- Model adaptation: Full Fine-Tuning, LoRA, Prompt Tuning, and Distillation — separate the learning objective from parameterization, and explain how the system constrains classification output
- SFT — why mean token loss does not replace task success, and how demonstrations teach answering, clarification, or abstention
- Preferences and reward models — how Bradley–Terry models preferences through score differences, whether the reward model remains reliable as the policy changes, and AI feedback, generative reward models, and rubric rewards
For worked calculations, continue to LoRA / QLoRA and distillation: parameters, gradients, and memory in the former; soft targets, KL, and what top-k leaves out in the latter.
For demonstrations built from model outputs, read rejection sampling and filtering: distinguish selecting an answer for a user from selecting data for another training round, then check how filtering changes the prompt mix.
II. Teaching by outcome: the RL line
- The three stages of RLHF, and what happened next — four models, which train and which are frozen
- PPO, GRPO, and DPO — starting from PPO, how GRPO / RLOO / REINFORCE++ / DAPO / DPO use different signals and objectives
- Verifiable rewards — sorting tests expose reward loopholes, followed by sparse feedback and process checks
Two frontier drafts extend this line: training multi-step agents with RL and how long chains of thought are trained.
III. Actually running it
- Post-training infrastructure: sampling, numerics, context — none of the three changes the objective; all of them decide what you can do to it. The least discussed layer, and often where results actually stall
- How far one base model can go — a rare natural experiment: base pinned, only post-training changed, showing what the three things in note 8 are worth
IV. What it costs
- Alignment trade-offs — check task-level regressions, diversity, and distribution shift; aggregate pass@1 and pass@k alone cannot establish mode collapse
Read these as a connected sequence: choose supervision and an update method, then examine training, validation, and costs.
Use the contents to jump to a specific problem; there is no need to finish everything in one sitting.
What each stage does, intuitively
- Pretraining: learn language, knowledge, and general capability from large-scale corpora.
- Continued Pretraining: keep adapting to the data distribution and terminology of one domain.
- SFT: learn instruction following and target behavior from high-quality demonstrations.
- Preference Learning: learn preferences from comparisons between responses.
- RL: adjust the policy according to task outcomes or reward signals.
They are not interchangeable buttons; they solve different learning problems.
From Base Model to Aligned Model: get the map right first
1. Three stages, each with its own supervision
A common high-level path is
$$ \boxed{ \text{Pre-Training} \longrightarrow \text{SFT} \longrightarrow \text{Preference / RL (RLHF, DPO, RLVR)} } $$
| Stage | Main data | Training signal | Common result label |
|---|---|---|---|
| Pre-training | large-scale text, code, and related corpora | token targets constructed from the text itself | Base Model |
| SFT | instruction–response demonstrations | selected ideal responses | Instruction Model |
| Preference learning (RLHF / DPO) | chosen/rejected pairs, or a reward model trained on them | which of two responses is preferred | Aligned Model / Policy |
| RL with verifiable rewards (RLVR) | prompts plus a program that checks results, such as answer matching or unit tests | whether a sampled response passes the check | RL-trained policy |
These labels describe functional stages; an organization need not save exactly three separate checkpoints. The useful distinction is that pre-training learns language, knowledge, and foundational capability from the data distribution; SFT teaches the model to invoke that capability through demonstrations; preference learning selects which of several plausible behaviors better serves the objective. RLVR needs no preference data: its reward comes from a program check (see verifiable rewards). It often sits in the same post-SFT stage as preference learning, but uses a different signal.
Pre-training precedes classic RLHF; it is not RLHF's first stage. In the full model lifecycle it is upstream, while the classic RLHF pipeline usually begins from an SFT policy.
2. GPT and BERT use different pre-training objectives
GPT-style decoder-only models use causal language modeling:
$$ \mathcal L_{\text{causal}} =-\sum_t\log p_\theta(x_t\mid x_{<t}). $$
They see only the left context and naturally support autoregressive continuation. BERT-style encoders commonly use masked language modeling: selected tokens are hidden and recovered using both left and right context. That objective more naturally supports bidirectional representation, classification, and extraction than generation.
A pretrained Base Model can continue text, imitate textual patterns, and encode a great deal of knowledge, yet still fail to treat a user prompt as an instruction requiring a direct answer. It may continue the question, imitate a webpage, or ignore a constraint such as “use three sentences,” because its original objective predicts text rather than behaving as a chat assistant.
“Pre-training learns capability; SFT teaches the model how to use it” is a useful approximation, not a law. SFT can also change knowledge and capability, although its scale and objective usually emphasize behavioral shaping.
3. Post-training is a scope; LoRA is an update mechanism
Post-training broadly covers training performed after initial foundation pre-training and is larger than RLHF:
$$ \text{Post-Training}\supset \{\text{SFT, preference optimization, RL, domain/safety tuning, distillation, tool use}\}. $$
Classic RLHF trains a Reward Model from preference pairs and then optimizes the policy with an RL method such as PPO. DPO directly optimizes the policy from chosen/rejected pairs, with neither a separate Reward Model nor an online RL loop. DPO is therefore not strictly reinforcement learning, although both approaches belong to preference alignment.
Keep two orthogonal axes separate:
- what is trained (objective / data): SFT, DPO, language modeling, distillation, RL;
- how parameters are updated (parameterization): full-parameter fine-tuning or LoRA / adapters.
The same SFT or DPO objective can use full updates or LoRA. Listing “LoRA” beside “SFT” as if both were training stages confuses the objective with the update mechanism.
4. Continued pre-training happens later but retains a language-modeling objective
A common domain-adaptation path is
General Base Model
→ continue next-token prediction on a large medical corpus
→ Medical Domain Base Model
→ Medical SFT / preference alignment
The middle step is Continued Pre-Training (CPT) or Domain-Adaptive Pre-Training. It occurs after initial pre-training in time but still uses a language-modeling objective and non-instruction domain text; SFT uses demonstrations to shape behavior.
The boundary between “pre-training” and “post-training” can therefore depend on whether the speaker is classifying by chronological stage or by training objective. When terminology is ambiguous, ask four concrete questions: what is the data, what is the objective, which checkpoint is the starting point, and what kind of model should result?
English terminology also differs: pre-training is a process, while a pre-trained model has completed that process. In casual usage, “pretrained LLM” may even refer broadly to a general-purpose model that has already been aligned.
What SFT does
SFT uses input–output demonstrations to teach the model to imitate a target behavior. It fits when you want to:
- learn a fixed task format;
- establish basic instruction following;
- distill an expert's process into the model;
- give the model a reasonably stable behavioral starting point first.
Its limit is that the objective only imitates demonstrations. The model can generalize to similar situations, but for long-tail cases the demonstrations never covered it may not know what to do, so those cases need separate tests.
What preference learning does
When the best answer is hard to write down directly but people can compare A with B, you can learn from preferences.
Methods such as DPO turn preference pairs directly into a policy training objective. That is simpler than building a full reward pipeline, but it still rests on several assumptions: that the preference labels are stable, that the candidates differ enough, and that the training distribution is close to real use.
What RL does
RL is the better fit for tasks that need multi-step action, have delayed outcomes, or require the policy to learn through exploration.
But RL does not automatically turn weak feedback into strong feedback. If the reward is sparse, ambiguous, or contaminated by policy bias, the model may only learn to exploit loopholes in the metric.
In recommendation, for example, one click depends at the same time on exposure position, the title, and how much time the user has. Use the click directly as the reward, and the model may learn to attract clicks more effectively rather than to deliver more long-term content value.
Why data often becomes the bottleneck before the algorithm does
If the data lacks longitudinal structure, so that one user's behavior is cut into many isolated samples, the model can hardly learn how long-term intent changes.
If the training data cannot say which policy produced each sample, it is also hard to tell whether the model learned the user's preference or the old system's bias.
So you need an objective-aware data layer that makes explicit:
- what the current training objective is;
- which samples fit that objective;
- how they are sampled and combined;
- how data and model versions are tracked;
- whether field semantics agree between training and serving;
- whether different experiments are actually comparable.
Working thesis. Post-training is fundamentally a problem of turning imperfect observations of human behavior into defensible learning objectives. The data-generating policy, feedback delay, selection effects, and evaluation design matter as much as the optimizer.
Questions I keep open:
- What can SFT learn that preference optimization or online RL cannot, and vice versa?
- How should objectives change when feedback is sparse, delayed, or confounded?
- How can training datasets preserve longitudinal and causal structure?
- Which behavioral gains survive distribution shift and repeated interaction?
How to choose a method
Start by asking:
| Question | More likely starting point |
|---|---|
| Do you have high-quality gold answers? | SFT |
| Is the answer hard to write, but easy for people to compare? | Preference learning / DPO |
| Does it need multi-step exploration, with a final outcome you can verify? | RL |
| Does the model lack domain knowledge and distribution coverage? | Continued pretraining |
| Is the feedback itself sparse, biased, or impossible to attribute? | Fix data and evaluation first, rather than switching algorithms first |
How to evaluate
Training loss or one aggregate reward is not enough. Also check:
- whether the new behavior comes from the intended mechanism rather than from data leakage;
- whether different user and task slices improve consistently;
- whether the model gives up diversity, calibration, or safety in exchange for one metric;
- whether offline preference translates into real task outcomes;
- whether the policy, log-probabilities, and data semantics agree between training and inference.
Continue reading
- DAPO: sampling, length, and clipping: inspect weighting and filtering on small batches.
- GSPO and ASPO: sequence ratios, token weights, and stop-gradients.
- SAO and asynchronous training: what changes when sampling lags behind policy updates.
- Data and feedback
- Evaluation
- Model Experience