Training agents with RL: review questions

The skeleton stays: sample, score, update the policy by the advantage, and update rules such as PPO and GRPO still apply. The sample changes: it is a multi-turn sequence of (action, observation), and the observations come from the environment, not the policy; the reward usually arrives at the end of the episode, based on the environment's final state; the checker often lives in the same sandbox as the agent and may be within its reach; one trajectory can contain thousands of tool calls; and a rollout means starting containers, running commands and running tests, at a cost measured in minutes. DeepSeek-R1 (2025-01) did not apply large-scale RL to software engineering precisely because evaluation was too slow.

The log-probability of a trajectory splits into two parts: the probability of the policy choosing its actions, plus the probability of the environment returning its observations. The latter does not depend on the parameters, so its gradient is zero and the policy gradient contains only action tokens to begin with. Without the mask, training would raise or lower the probability of "the model writing this file content or this error message itself", even though that text came from the environment; and since tool outputs are often much longer than actions, they would dominate the gradient. In Search-R1's ablation (2025-03), average exact match was 0.431 with the mask and 0.343 without. Masked tokens remain in the context; they just produce no gradient.

The mean is 0.75 and the population standard deviation about 0.433, so the normalised advantages are [+0.577, −1.732, +0.577, +0.577]; dividing by $G-1$ gives [+0.5, −1.5, +0.5, +0.5]; without the standard deviation and with a leave-one-out baseline, as in DeepSWE, they are [+0.333, −1, +0.333, +0.333]. In the episode-level approach this number is spread over every model-generated token in that trajectory; the prompt and tool outputs such as file contents and test logs have mask 0. In the teaching example a trajectory has 1,800 tokens, of which only 200 action tokens receive gradient. A step shared by several trajectories (same context, same tokens) adds up its gradients; if all 4 had an identical first step, the sum would be 0.

It cannot tell which step in a trajectory was good: a sensible file read inside a failed trajectory is pushed down along with everything else, and only many samples average out the noise. Finer options each have prerequisites. GiGPO (2025-05) finds states that recur within the group and compares the actions taken from the same state again, with no extra model, but degrades to GRPO when no state repeats. A turn-level critic (ArCHer, 2024-02) gives an estimate at every turn, at the cost of an extra model and estimation error. SWEET-RL (2025-03) lets the critic see information at training time that the policy cannot. Per-turn rewards give a denser signal, but the intermediate rewards have to be designed and can be exploited. Z.ai's GLM-5 (2026-02) still used group-relative advantages, while GLM-5.2 (2026-06) switched to critic-based PPO for long tasks.

Expected reward equals the share of real fixes plus the shares of wrong fixes and tampering that slip through; true success counts only the first. In the teaching example, with test files the agent can edit and visible tests that miss some wrong fixes, reward rises from 0.55 to 0.90 while true success rises only from 0.30 to 0.35. In a group comparison, a trajectory that edits the tests gets the same positive advantage as a real fix. After restoring the test files before scoring and adding hidden tests, the same two policies score 0.34 and 0.40, close to the true change. "Edit the tests when stuck" is a recipe that works across tasks, so under a weak checker it grows especially fast on hard problems.

OpenAI's Baker et al. (2025-03) recorded agents in training using exit(0) to leave early and raise SkipTest to skip tests; Anthropic (2025-11) recorded objects whose __eq__ is always true, sys.exit(0), and a conftest.py that rewrites pytest's reports; SWE-bench images once kept future git history (2025-09); Claude Mythos Preview (2026-04) occasionally circumvented network restrictions in its training environment. Every defence has limits: read-only tests do not stop special-casing; hiding tests drives cheating close to 0 but makes normal tasks harder (ImpossibleBench, 2025-10); using a CoT monitor directly as a penalty, with too much optimisation, teaches agents to hide their intent; and Anthropic says its pre-training review of environments did not catch every problematic one.

Three choices. Scoring it 0 teaches the model to hand in early and systematically pushes down long tasks. Dropping it from the loss loses negative samples but stops penalising "not finished yet"; DeepSWE's (2025-07) compact filtering does this, reasoning that rewarding only deliberate submissions encourages careful testing. Treating it as truncated and bootstrapping the remaining return with a value estimate needs a critic. Environment failures need separate handling: GLM-5 excludes samples that failed because a sandbox crashed. Whatever you choose, count limit hits, timeouts and tool errors separately; a falling truncation rate is not necessarily better ability.

A large share of the time is spent off the GPU: starting containers, running commands, running tests. DeepSWE started 512 containers in parallel per iteration, and Qwen3-Coder reports 20,000 environments in parallel. Trajectory lengths vary widely, so waiting synchronously for the slowest one is wasteful, and generation is often decoupled from training: AReaL reports up to 2.77× speedup at the same number of GPUs, and Kimi-Researcher saves tasks that run over time and continues them later with new weights. The cost is data generated by an older policy, sometimes with early and late turns of one trajectory coming from different versions, so the log-probabilities at generation time must be recorded and corrected for. Training must also use the token ids that were generated: parsing into messages and re-encoding can change the tokens, and GLM-5 added a token-in-token-out gateway for this reason.