Compare experiments and replace models safely

Original teaching examples · Checked: 2026-10-10. Responses, pass rates, and traces below are illustrative, not model measurements. The code reproduces data processing, not model inference.

After training, the tempting question is “How much did it improve?” An aggregate move from 60% to 70% doesn't tell you which questions improved, which regressed, or whether evaluation added an extra hint.

We'll follow an experiment from saved settings to paired results, then open a failure and work out what to change. For debugging, start with “The retriever found it. Why was the answer wrong?”. For a new experiment, start with the manifest below.

Save enough information to repeat the experiment

You don't need a large platform first. A manifest saved with the results prevents many avoidable mix-ups:

{
  "run_id": "citation-sft-demo-01",
  "base_revision": "local-fixture-v1",
  "data_snapshot": "examples-v3",
  "split_rule": "question-family-v1",
  "template_revision": "chat-template-v2",
  "objective": "assistant-target-token-mean",
  "eval_snapshot": "fixed-evidence-v2",
  "decode": {"sampling": false, "max_new_tokens": 128}
}

These are illustrative fields, not downloadable checkpoint names. Real runs also need content hashes, dependency versions, seeds, optimizer settings, training steps, and valid-token counts. A run ID indexes artifacts; it cannot replace a data snapshot.

If A gets 128 output tokens and B gets 512, the comparison can still be useful—but it compares systems with different budgets, not training alone.

Six questions reveal what an aggregate hides

1 means passing a predefined check; 0 means failing. Put old and new outcomes for the same question on one row:

Overall performance rises from 4/6 to 5/6. But q5, previously handled correctly without evidence, now fails. That slice drops from 2/2 to 1/2. The new model might answer better—or simply be more willing to answer. Inspect q5's actual output, then check the quantity and quality of missing-evidence demonstrations.

A paired comparison retains the change on each question: two wins, one loss, three ties. Six cases illustrate regression detection, not a stable improvement estimate. Real comparisons need sample sizes and uncertainty; resampling should respect related questions from the same user or document.

The retriever found it. Why was the answer wrong?

Suppose someone asks, “When does North Hall close on Sunday?” The knowledge base says 18:00 on Sundays and 20:00 on weekdays. The assistant answers 20:00. It's tempting to add more date-related training examples. First, look at what happened to this request:

We have found an evidence-assembly problem: retrieval logs contain the right passage, but the final input doesn't. Giving a confident answer without Sunday evidence is another behavior worth testing. What this case cannot show is that the model ignores correct evidence when it receives it. Both failures may exist; investigate them separately.

A screenshot of the answer isn't enough to replay the request. Keep the original question, retrieved document revisions, assembled input, actual output, and reason for the verdict. Where available, include token IDs, truncation boundaries, generation stop reasons, and tool state. Logs containing user data need access controls, redaction, and retention limits; don't paste them into public issues.

Change one part and replay the request

Hold the model and decoding settings fixed, then try small interventions. These are proposed experiments, not reported results:

Manually supplying the needed passage is an oracle-context comparison. It asks whether the downstream component can work when upstream supplies the right material. Keep that result separate from end-to-end scores. One correct answer isn't enough; repeat across cases. Continued failure isn't a capacity diagnosis either—conflicting instructions, formatting, and grading may still be wrong.

For sampled generation, keep repeated outcomes rather than selecting one successful screenshot. Greedy decoding still requires pinned versions and execution settings; hardware, kernels, or service updates can change results. Single-component changes help isolate clues, but the final combination still needs an end-to-end test.

from pathlib import Path
import sys

sys.path.insert(0, str(Path("practice/rag/code").resolve()))
from evidence_pipeline import Chunk, pack_evidence

weekday = Chunk(
    "weekday", "hours", 1,
    "North Hall closes at 20:00 on weekdays.",
    frozenset({"public"}),
)
sunday = Chunk(
    "sunday", "hours", 1,
    "North Hall closes at 18:00 on Sundays.",
    frozenset({"public"}),
)
retrieved = [weekday, sunday]
provided, used = pack_evidence(retrieved, budget=7)
oracle, oracle_used = pack_evidence([sunday, weekday], budget=7)

assert sunday in retrieved and sunday not in provided
assert sunday in oracle
assert used == oracle_used == 7
print("retrieved:", [chunk.chunk_id for chunk in retrieved])
print("provided:", [chunk.chunk_id for chunk in provided])
print("oracle:", [chunk.chunk_id for chunk in oracle])

Before training more, inspect the targets

Return to q5, where the model answers without enough evidence. Read a small batch of actual training examples, not just the config. Do missing-evidence demonstrations acknowledge uncertainty? Do similar inputs have conflicting labels? After templating and truncation, which answer tokens still contribute to loss?

Suppose the intended demonstration says, “The material doesn't give Sunday hours, so I can't confirm.” A long input truncates away the answer, leaving no valid target tokens. The run is called SFT, but that example contributes none of the intended answer supervision. An inverted mask can be equally misleading: loss falls on user questions while the answers you wanted to teach are ignored.

Decode a batch and inspect inputs, targets, loss masks, and valid-target counts together. Causal-LM logits and labels generally require a one-position shift; count the positions actually used by the loss. This project supervises assistant answers. That is a task choice, not a requirement for every pretraining or fine-tuning objective.

TRL v0.29.0 requires a compatible template that returns an assistant mask for assistant_only_loss=True. A flag in a config is not evidence that the resulting batch is correct. See data to batches and masks and loss denominators.

Loss falls, but generation doesn't improve

Start with a tiny, internally consistent training set. Check trainable parameters, nonzero finite gradients, and weight changes. If the model cannot learn those examples, inspect the objective, optimization, and implementation first. Once it can, test unseen cases. Memorization and generalization answer different questions.

• It predicts well after correct prefixes but fails when generating. Teacher forcing supplies the correct history; free generation may already have taken a wrong turn. Measure held-out loss and generated behavior separately.
• It works locally but fails in the service. Verify the base model, adapter, tokenizer, template, and stopping settings. A local checkpoint name doesn't identify the revision actually being served.
• It repeats itself or stops early. Inspect EOS handling, stop reasons, output limits, and the actual context before changing decoding. A higher temperature can add variation; it cannot restore missing evidence or correct a bad label.

One inexpensive check is whether templating and tokenization added special tokens twice. The Transformers chat-template documentation warns about this exact mismatch. Checking the input path gives you a clearer next step than immediately changing the training algorithm.

Let the failure determine the fix

You don't have to progress through prompt changes, then SFT, then RL. If a document is stale, update it. If a tool executes twice, fix execution. Training isn't a substitute for those components doing their jobs.

LoRA, QLoRA, and full-parameter updates are a separate decision. A serious failure doesn't imply that full fine-tuning is required. The LoRA paper reports competitive results on its evaluated tasks, not a guarantee across all tasks. Compare key slices and resource costs under controlled data and tuning budgets; see training choices.

Input and output checks have tradeoffs too. Rules are inspectable and cheap but cover limited cases. Another model offers flexibility at the cost of latency, missed violations, and false rejections. Tool permissions should be enforced reliably by the execution layer, not granted solely because a second model approves an action.

Fix the case, then test nearby cases

Once the assistant answers 18:00 correctly, try a new phrasing or change one condition. The following original examples use the behavioral test types in CheckList (ACL 2020):

Add missing evidence, conflicting rules, and irrelevant long text to test when an answer is justified. Ten cosmetic paraphrases may be less useful than one change to a condition that should alter the answer.

Keep repaired failures in a regression set to catch recurrence. Once used during development, they are no longer independent test evidence. Group close paraphrases by document or question template when splitting. Validate the final choice on untouched cases; success on a rehearsed question is not a generalization result.

Don't rank fixes by counts from a hand-collected failure folder alone. Ten complaints are not ten randomly sampled requests. Estimate frequency on data with a known denominator, then consider impact, cost, and strength of evidence. Severe permission violations need separate treatment rather than being traded away in an average score.

Record incomplete cases separately

If the old model completes 100 questions and the new one completes only 80, averaging each model's successful responses can introduce selection bias. Our small implementation rejects mismatched IDs so incomplete pairs require explicit handling.

A real evaluator can retain ok / timeout / invalid_output per case and report two views:

• Did the user get a usable result? Use all requests as the denominator; timeouts and malformed outputs affect task success.
• Which model answered better when both completed validly? Compare that subset, with counts and reasons for exclusions. This is conditional performance, not an estimate automatically valid for all requests.

Both views matter. A timeout isn't a factual error, but it cannot disappear from the report. If the task requires JSON, invalid formatting can itself count as task failure.

python3 practice/post-training/code/experiment_checks.py

The code prints the six-case overall and slice results. Removing a new-model result causes a missing-pair error; assigning a question family to both training and testing causes a leakage error.

Increase cost in stages

If a few examples cannot be learned, inspect inputs, objectives, and implementation before adding GPUs. Memorizing those examples doesn't establish generalization either; it is a training-path check.

See training choices, multi-GPU batches, and recovery comparisons for implementation decisions. GPU count alone is not a resource report: include valid target tokens, end-to-end time, peak memory, and failed-work replay costs.

If only offline evaluation is available, keep the conclusion offline. User simulation can stress-test behavior, but it cannot establish real-user product benefit.

Release a compatible bundle

Weights belong with the tokenizer, template, generation settings, preprocessing, and tool/output schemas. Rolling back weights while retaining a new template may not restore the old system.

First run a shadow version on matching inputs without changing user-visible outputs, checking format, latency, and failure types. Then decide whether controlled online validation is appropriate. Shadow traffic cannot measure changed user behavior because users never saw the new outputs. Define stop conditions in advance, such as key-slice regressions, excessive invalid outputs, or latency beyond the product budget. Thresholds follow risk, not this six-question table.

“Don't release this run” is a valid conclusion. Explaining why, and what to test next, is more useful than presenting every training run as a win.

Project introduction · Compare the recommender release path