Why post-training keeps coming back to data

Post-training has a way of sending you back to the data, scoring rules, or environment. You have already picked SFT, DPO, or GRPO. Why is there still so much to figure out? The algorithm settles only part of the experiment: tasks, feedback, data updates, and generation budgets still affect whether the run improves the model.

We have effective methods, but not a universal recipe that transfers unchanged across models, tasks, and budgets. SFT, preference optimization, and online RL have well-defined methods and implementations. Their combinations, data, feedback, and system settings are still evolving. Building data chooses what the model practices; evaluation checks what it learned. Neither replaces work on exploration, optimization, or systems.

Start with what public recipes are changing, then three demonstrations of the same question. If you already run experiments, jump to environment checks and testing a data change.

What are public recipes still changing?

A new acronym is not a reason to replace the entire system. Specific accounts help distinguish problems with working solutions from choices that still need experiments.

These examples show that training data need not be a file completed before training: feedback and task selection may change with the model. This is a synthesis of the cited work, not advice to update tasks or rewards dynamically in every project. The last two papers are further reading; recency alone does not earn a place on the main branches of the AI timeline.

A small example shows the change. Sample a checkable problem four times: with rewards 0, 0, 1, 1, subtracting the group mean still separates attempts worth encouraging, but an all-correct or all-wrong group has zero advantage for every attempt. The full worked example and the zero-variance handling are in when the within-group signal disappears.

As the policy improves, a problem may produce all-correct groups more often. Conversely, one all-wrong group does not make it permanently unlearnable. An unchanged prompt collection can therefore produce a changing learning signal, but deleting every all-wrong task is not a general answer. Compare demonstrations, intermediate tasks, or sampling changes; check the scorer and account for discarded generations. This illustrates group-relative signals, not measured model improvement.

Fixed data and SFT may already solve a small, well-defined task. Adding RL just to make the pipeline more elaborate is unnecessary. Conversely, more cleaning alone will not solve exploration, credit assignment, or stability problems; see Why RL is still hard to train. Automate repetitive format and environment checks. Reserve experimentation for decisions that change what the model learns.

One question can need three different responses

Suppose we are training an assistant to answer from supplied documents. The user asks, “When does registration close?”

If nearly every training example is of the first kind, most practice amounts to “find a date and answer.” We may want the model to check conflicting evidence, but barely demonstrate that behavior.

A subtler failure: the teacher sees the full notice, while the student receives only the registration link. Using the teacher's “Wednesday” as the target rewards a conclusion unsupported by the student's input. The date might even be right, but this does not teach the evidence-based behavior we wanted.

To judge a demonstration, read the input, available tools, response, and supporting evidence together—not just the final answer. This does not mean a model cannot handle anything absent from the dataset. Pretraining and generalization still matter. The question is what these particular training signals encourage.

Four choices hidden inside a dataset

What does data represent? A useful starting point is that it turns our intentions for a task into examples and feedback the model can actually encounter.

A preference label is not a context-free statement of correctness. Its meaning depends on the annotators and instructions. InstructGPT used particular demonstration and comparison-labeling procedures and discussed why the resulting preferences cannot represent everyone. InstructGPT, 2022, §§3, 5.2

Behavioral feedback needs similar care: not clicking a result does not necessarily mean disliking it. Feedback to objectives develops that example. For now, an observed record and the objective we care about need an explicit connection; they are not interchangeable.

The algorithm can stay fixed while the objective changes

Consider one explicit SFT convention: average over supervised response tokens within each example, then take a weighted sum across examples.

$$ \begin{aligned} \ell_i(\theta)&=-\frac{1}{|T_i|}\sum_{t\in T_i}\log p_\theta(y_{i,t}\mid x_i,y_{i,<t})\\ L(\theta)&=\sum_i\alpha_i\ell_i(\theta),\quad \alpha_i\geq 0,\quad \sum_i\alpha_i=1. \end{aligned} $$

Here $x_i$ is the visible input, $y_i$ the demonstration, $T_i$ the supervised positions, and $\alpha_i$ the example weight. Every example must have at least one valid target. This is the normalization convention chosen here, not a claim about every trainer's default.

Changing inputs changes what the model can use. Changing responses changes what it imitates. Changing masks changes which tokens are supervised. Changing the sampling mixture changes how often particular mistakes contribute to parameter updates. None requires renaming the algorithm.

For example, take two examples with 1 and 3 supervised tokens and mean losses of 0.2 and 1.0. Equal example weighting gives 0.6; averaging all supervised tokens gives 0.8. The latter weights the second example more heavily—it is not merely a different loss display. Multi-GPU execution and gradient accumulation must preserve the intended denominator too; see Data and objectives.

That is why “another 100,000 examples” is not enough information. What are they? Which examples do they displace? How often will they be sampled under the same training budget?

A tidier dataset can lose important practice

Return to the registration assistant. For easy arithmetic, assume 100 demonstrations: 90 with sufficient evidence and 10 without it. We add a filter: “Remove short responses.”

It retains 81 supported-answer examples but only 2 missing-evidence examples. “The deadline isn't listed here. Could you share the notice?” happens to be short.

from fractions import Fraction

before = {"supported": 90, "missing_evidence": 10}
after = {"supported": 81, "missing_evidence": 2}
before_share = Fraction(before["missing_evidence"], sum(before.values()))
after_share = Fraction(after["missing_evidence"], sum(after.values()))

assert before_share == Fraction(1, 10)
assert after_share == Fraction(2, 83)
print(f"Before: {float(before_share):.1%}; after: {float(after_share):.1%}")

The output is Before: 10.0%; after: 2.4%. Under uniform sampling of the retained examples, the model gets less practice handling missing evidence. This calculation establishes a change in the data mixture, not a particular decrease in model accuracy.

Inspect the rejected short responses first: distinguish inadequate answers from appropriate clarification. Repair or remove incorrect labels, then check retention by situation. Collect or resample examples where needed, but do not retain wrong answers just to meet a quota. Nor is 50:50 a universal recipe: overemphasizing missing-evidence cases may encourage unnecessary abstention on answerable questions.

Row count, duplication rate, and average length are useful, but add retention rates by situation. Otherwise you know how much you removed without knowing which practice you lost.

Different training stages need different data checks

For offline DPO, chosen means preferred within the pair, not necessarily correct. A relative preference can exist between two wrong answers. If factual accuracy matters, verify it separately when constructing and reviewing pairs. DPO, 2023, §§3–4

For RL, inspecting the initial prompt file is not enough. As the policy changes, so do its responses, tool calls, and failures. Understanding a batch also requires the policy version, sampling settings, environment, and reward rules that produced it.

If success on a coding task means passing tests, the tests' boundary-case coverage matters. Passing an incomplete suite is not proof of solving the task. DeepSeek-R1's rule-based rewards include answer checking and code execution: not requiring human-written reasoning demonstrations does not remove the need for tasks and reliable feedback. This is a reference to the v2 paper's reward design, not a recipe for all RL tasks. DeepSeek-R1, v2, §2.2 Why answers grow longer when only the final answer is rewarded is covered in how long chains of thought are trained.

Low-reward trajectories are not automatically dirty data. An incorrect answer may be exactly the attempt RL should learn from. A failed call because the tool service was down is different. Treating both as the same negative reward may teach the model to avoid a useful tool. Define retry, exclusion, and failure rules for the task, record the cause, and do not quietly keep only successful attempts afterward.

Environment setup has two meanings

One is getting the code to run: drivers, PyTorch, CUDA, attention backends, training frameworks, and dependencies. The other is making sure the running experiment still means what we intended: the same inputs, the same objective, and feedback under the same task rules.

For instance, as of October 2026, TRL v1.16.0's assistant_only_loss=True still requires the chat template to mark assistant output with {% generation %} and {% endgeneration %}; for known model families such as Qwen3, TRL swaps in a patched training template on its own. TRL SFT documentation, v1.16.0 That training template need not render the same way as the inference template; Qwen3's, for example, keeps the thinking block in every assistant turn. TRL Chat Templates, v1.16.0 So a True in a configuration file is not evidence that the labels are right. Decode a sample and inspect the actual supervised positions.

Fixing a random seed does not guarantee identical results across machines and versions either; PyTorch documents this limitation explicitly, and the statement is unchanged in the 2.14 docs checked in October 2026. PyTorch 2.14: Reproducibility

You should not have to rediscover these issues on every run. Pin dependencies, retain dataset and template versions, and keep a small fixed set of smoke-test samples. For concrete input and gradient checks, continue to From one demonstration to a training batch.

Quality is not the same as polished writing

A fluent, long response may lack evidence. A short clarification may be exactly right. A hard example is not automatically useful either: an ambiguous problem with unreliable scoring can teach noise. At the other extreme, filtering everything down to easy tasks leaves little practice on the behavior that needs improvement.

Review correctness and training usefulness separately:

• Can we use it? Check provenance, permissions, privacy, support for the label, and evaluation contamination.
• What does it add? A new situation, a new failure, or dozens of paraphrases of the same template? How much is already easy for the model, and how much is not?
• What does selection cost? Which labels become more trustworthy, and which languages, difficulty levels, or uncommon situations disappear?

Apply the same questions to synthetic data. A teacher produces candidates for review, not guaranteed ground truth. Verify against calculators, executable tests, or source documents where possible; subjective preferences need a rubric and spot checks. If the same judge filters training data and establishes the improvement, check whether the model has simply learned to please that judge; how a verifier's blind spots stall the generate–verify–train loop is in self-improvement: why the verifier is the bottleneck.

LIMA is often cited as evidence that a small curated set can help. Its main experiment and ablations used particular base models, tasks, and evaluation procedures; they do not establish that every post-training problem needs only 1,000 examples. The useful habit is to compare quality, coverage, and quantity separately, rather than memorize a magic dataset size. LIMA, 2023, §§2, 5

How to tell whether a data change helped

Suppose the hypothesis is “More missing-evidence demonstrations reduce unsupported answers.” Fix an evaluation set with no near-duplicates from training and appropriate separation by question or document family. Then compare the original dataset with the modified version.

Start with a matched training-token or compute budget and state which one you use; different length distributions may prevent matching both exactly. If you want to test one new demonstration category, change that first rather than several things together. Iterate on a development set and reserve the final test set, rather than using it to choose each new filter.

“Cleaned several million rows” is less informative than explaining which supervision problem you found, which selection rule you changed, what you retained or removed, which unseen situations improved, and what costs or failures remain. A change that does not help is worth recording too.

Spending time on data is not surprising. The more useful question is whether today's work merely made a file readable or gave the model a better chance to learn the intended behavior. Automate the former where possible; test the latter.

Where to go next

• Feedback to objectives: why behavior, preference, and supervision are different things.
• SFT: how a conversation becomes a training signal: conditional probabilities, masks, and updates.
• Post-training engineering: connecting the data pipeline, distributed training, checkpoints, and evaluation.