Post-training infrastructure: sampling, numerics, context

Last reviewed: 2026-10 · Prerequisites: RL and language models, PPO clipping

Follow one response into training

A model receives a multiplication question, calls a calculator, reads the result, and answers. That gives us an execution trace. Training on it requires more: which tokens did the model choose? Which weights generated them? Did the task finish, or did a length limit stop collection?

These are not optional debugging logs. Without them, we may not even know where to apply the loss. Start with the rollout record, then examine throughput, log-probs, and context. All numerical examples below are teaching assumptions, not framework benchmarks.

A rollout need not be a complete answer

A rollout runs a policy and collects actions, observations, and outcomes. It can run one task or a batch, and collect a whole task or a fragment. The term alone specifies neither the newest policy nor a learned reward model.

An end marker may end only the current assistant turn; a pending tool result means the task is not yet complete. Conversely, for a task requiring just one answer, a completion can be the entire episode.

What a usable training record contains

Consider these invented span lengths; they do not represent a real tokenizer's output:

There are 20 input positions but only 9 valid actions. Dividing policy loss by 20 would change its scale. Tool results still condition later actions even though they receive no direct policy loss. Excluding their loss does not mean deleting them from context. Whether role or control markers are sampled actions depends on the generation protocol, not their appearance in a decoded string.

A record needs the following kinds of information; field names vary by implementation:

Rewards may come from tests, the environment, humans, or a learned scorer. Reference scores, values, and advantages need not be computed during generation—or all be required. Extra fields should not obscure algorithmic roles. A PPO rollout buffer generally supports a limited set of updates, not unrestricted replay of historical experience. The PPO paper describes alternating collection with multiple epochs of minibatch updates.

def action_mean(values, mask):
    if len(values) != len(mask) or any(flag not in (0, 1) for flag in mask):
        raise ValueError("aligned values and binary mask required")
    count = sum(mask)
    if count == 0:
        raise ValueError("no policy actions")
    return sum(value for value, flag in zip(values, mask) if flag) / count

mask = [0] * 6 + [1] * 4 + [0] * 3 + [1] * 5 + [0] * 2
values = [99.0] * 6 + [2.0] * 4 + [99.0] * 3 + [4.0] * 5 + [99.0] * 2
assert len(mask) == 20 and sum(mask) == 9
assert abs(action_mean(values, mask) - 28 / 9) < 1e-12

Did the task end, or did collection run out of time?

Suppose a step gives reward 0.2, the next-state value is 0.8, and the discount is $\gamma=0.9$. For a one-step TD target:

• True task termination leaves no future return: target = 0.2.
• An external collection limit interrupts a task that could continue: retain the bootstrap, giving 0.2 + 0.9 × 0.8 = 0.92.

This does not mean every length limit requires a value bootstrap. If the task is “finish within 10 steps or fail,” exhausting the budget is an endpoint of the task itself. A collector that merely pauses every 10 steps is different. Gymnasium's time-limit guide distinguishes termination and truncation; finite-horizon tasks also need sufficient observation information, such as remaining time.

Bootstrap from the interrupted trajectory's final observation, not the first observation after a reset. With GAE, collection boundaries usually also stop advantage recursion so errors from the next episode do not leak backward. The bootstrap mask and recursion-boundary mask can differ. Methods without a Critic should follow their own truncation and reward convention rather than add an unnecessary value field.

Find out where sampling time goes

A batch can pass through generation, tool waits, scoring, training, and weight synchronization. GPU utilization alone can make “waiting for a tool” look like “the model is too small.” As NVML defines it, the utilization in nvidia-smi is only the fraction of the sample period during which some kernel was running; it says nothing about how much compute is used. See Performance: the roofline, arithmetic intensity, and MFU.

Take hypothetical durations: 30 seconds sampling, 10 scoring, 20 training, and 5 synchronizing. Serial execution takes 65 seconds. If the first three stages have independent resources and overlap perfectly across batches, while the 5-second synchronization cannot overlap, the steady-state interval has a lower bound of max(30, 10, 20) + 5 = 35 seconds: about 1.86× faster. This is not a general asynchronous speedup. The first batch still traverses all stages; contention, stragglers, and networking add time.

Track generated tokens/s, loss-bearing tokens/s, queue-wait quantiles, policy-version lag, timeout / discard rates, and task quality at equal wall time. Collecting invalid answers faster does not improve learning efficiency. See asynchronous policy learning for differences between objectives.

Identical weights can still produce different log-probs

Separate the raw model distribution, the sampling distribution after temperature / top-p processing, and the distribution recomputed by the trainer. A field named logprob does not establish which one it represents.

For example, raw probabilities [0.8, 0.2] become [2/3, 1/3] under temperature 2. If the new model gives the second action probability 0.25, its ratio to actual behavior is 0.25 / (1/3) = 0.75. Using the raw 0.2 denominator instead gives 1.25—even reversing whether that action has become more likely. Top-p truncation also changes support; ordinary importance sampling cannot recover actions that collection never sampled.

When actual behavior policy $b$ is the chosen anchor, the action ratio is:

$$ r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{b(a_t\mid s_t)}. $$

Other implementations use a trainer-recomputed old policy as the PPO anchor and separately correct the rollout-to-old mismatch. The denominator neither always comes from the inference engine nor can always be replaced by a trainer score. Check how distributions enter the objective. verl's rollout-correction guide distinguishes these designs; this discussion checks its 2026-09-23 revision, not a promise that configuration names will stay fixed.

Several primary sources sit behind that guide and are worth reading directly. The blog post “Your Efficient RL Framework Secretly Brings You Off-Policy RL Training” points out that an inference engine (such as vLLM) and a training framework compute different probabilities for the same token, so “on-policy” training actually carries an off-policy bias. Its fix truncates the training-to-inference probability ratio to an interval and multiplies it into the gradient: truncated importance sampling (TIS). “When Speed Kills Stability” (Liu et al., 2025) links this mismatch to RL training collapse and compares token-level with sequence-level corrections. IcePop, in the Ring-1T report (Ling Team, 2025), masks and clips tokens whose mismatch is too large. Frameworks are following: as of October 2026, TRL v1.16.0 turns on this kind of correction by default when generating with vLLM (vllm_importance_sampling_correction=True; the default mode in the config signature is sequence_mask, which masks whole sequences, with vllm_importance_sampling_clip_max=3.0). A default being on does not mean the problem is solved: find where the mismatch comes from before choosing between truncating and discarding.

After aligning sampling semantics, inspect kernels, precision, templates, positions, and masks. Score identical tokens and weights on both sides, examining valid-action differences by quantile, extremes, and length. Bitwise equality across hardware is not the requirement. Large ratios alone do not prove data staleness: alignment errors and numerical differences can also cause them.

Removed context is information the policy no longer has

Return to the tool task. The user initially says “do not modify the original file.” Several turns later, the system removes that sentence to fit the context window. If the model then changes the file, reward design is not the only issue: the constraint was missing when it made the decision. For how truncation and compression affect scoring and the loss in multi-turn agent training, see the section “When episodes get long” in the frontier draft on agent credit assignment and training.

These choices affect completion, answer quality, and latency—not just whether context fits. Changing compression changes the policy's observations, not necessarily the underlying environment transitions. Losing necessary history can also make observations non-Markov. Version templates, summarizers / retrievers, raw tool records, and external state so experiments can be compared and tasks resumed.

Trace a small batch before scaling up

You do not need the most elaborate asynchronous system first. Fix a small set of tasks and trace sampling through loss: reconcile action counts, explain stop reasons, and verify rewards before optimizing waits. The point is not that algorithms matter less than engineering; it is to rule out code that does not implement the objective you think it does.

Nor are these changes all objective-preserving. Changing truncated-response rewards, filtering, or sampling transformations can change the training distribution, estimator, or task definition. Record what changed rather than grouping everything under “performance optimization.”

Before a larger run, I would check

1. Can one raw record reconstruct tokens, roles, the policy mask, and next-token logits alignment?
2. Have unscored data, tool failure, task failure, and length truncation quietly become the same zero reward?
3. What are old, current, reference, and actual behavior distributions? Is sampling configuration recorded?
4. Does the synchronous / asynchronous comparison report throughput, data age, and quality at equal time?
5. Beyond weights, is there an explicit resume policy for queues, samplers, tools, and context state?

Where to read next

• Experiments and debugging: check the loop with small experiments.
• Checkpoints and resuming: verify the next update, not merely file loading.
• Distillation: connect student rollouts to teacher supervision.
• The same base, different post-training: algorithm and system changes in a specific report.

The one thing to retain

A rollout is more than generated training text. It must preserve enough information to explain which choices the model made and under what conditions. Establish that record before making collection faster.