Training agents with RL: from single answers to multi-step tasks

When you train on maths problems with RLVR, one sample is a prompt, one answer and one score. Now you want to train an agent that fixes bugs: it reads files, edits code, runs tests, reads the error and edits again, and after dozens of turns it submits; hidden tests decide whether it gets 1 or 0. The skeleton of the algorithm does not change: sample, score, update the policy by the advantage. What changes is what a sample looks like: it is an interaction run inside a sandbox, the reward usually arrives once at the end, the text returned by tools sits inside the sequence without being the model's actions, and one rollout can take minutes to tens of minutes. So the hard part moves away from "which loss" to three places: whether the environment and reward can be trusted, how one trajectory's success or failure is split across its dozens of actions, and how to afford the rollouts.

Draft, pending review. The site owner has not reviewed this note yet; facts, emphasis and wording may change. Check the primary sources linked in the text before relying on it.

From single answers to multi-step tasks: what changes

Why tool outputs carry no gradient follows directly from the probability of a trajectory. Let $h_k$ be the whole history before the $k$-th action, $a_k$ the $k$-th action (a span of tokens), and $o_k$ the observation the environment returns next:

$$ \log p_\theta(\tau)=\sum_{k=1}^{T}\log\pi_\theta(a_k\mid h_k)+\sum_{k=1}^{T}\log P_{\rm env}(o_k\mid h_k,a_k) $$

The second sum is the environment's transition. It does not depend on the parameters $\theta$, so it vanishes under the gradient, and the policy gradient keeps only the first term:

$$ \nabla_\theta J(\theta)=\mathbb E_{\tau}\Big[R(\tau)\sum_{k=1}^{T}\nabla_\theta\log\pi_\theta(a_k\mid h_k)\Big] $$

A number to make it concrete: a two-turn trajectory where the policy picks $a_1$ with probability 0.5, the tool returns $o_1$ with probability 0.9, the policy picks $a_2$ with probability 0.2, and $o_2$ is deterministic; the whole trajectory has probability $0.5\times0.9\times0.2\times1=0.09$. Changing $\theta$ can only change the 0.5 and the 0.2; the 0.9 belongs to the environment. This assumes the environment does not use the same weights: if subagents run on the model being trained, their outputs do depend on $\theta$; Kimi K2.5 (2026-02) freezes its subagents and treats their outputs as observations. Masking tool-output tokens out of the loss in an implementation does exactly this. Real training replaces $R(\tau)$ with a baseline-subtracted advantage and adds probability ratios and clipping; see The LLM bridge.

Timeline

Only events that can be checked against a primary source. Every score a team reports is self-reported, under its own evaluation settings.

What the labs publish differs a lot. OpenAI's product announcements state that agents are trained with RL but give no algorithmic or environment detail. Anthropic's system cards do not describe training algorithms, but they keep reporting hacking in training environments and the defences against it: from Claude 3.7 Sonnet's special-casing to Claude Opus 5.5 (2026-09-22) screening training episodes with classifiers. Google DeepMind's Gemini 2.5 technical report (2025-07) says its RL environments added tasks that need multi-step actions and tool use. Algorithms, environment counts and infrastructure details come mainly from teams releasing open-weight models (Moonshot AI, DeepSeek, Qwen, Z.ai) and from open projects such as Agentica.

What is settled, what is still moving

Each cell rests on a dated public record; the "what to watch" column says which kind of result would change the judgement, not a prediction that it will happen.

How to read this series

For background: how policy gradients, GRPO and critics apply to LLMs is in The LLM bridge; what tests miss when used as rewards is in Verifiable rewards; termination versus truncation is in MC and TD; correcting for stale data is in Asynchronous RL; the basic roles in RLHF are in RLHF. The structure of an agent is in Agents, and when splitting into several agents pays off is in One agent or several; training on self-generated data and why it stalls at the verifier is in Generate, verify, retrain.

1. Where environments and rewards come from: code sandboxes, browsing, simulated users, tool APIs and computer-use environments; four sources of reward; documented exploits; how far expected reward and true success drift apart when tests catch only some failures
2. Credit assignment over multi-step episodes, and the training system: group-relative advantages for four 3-turn trajectories; which tokens receive gradient; GiGPO, turn-level critics and per-turn rewards; long contexts, truncation and asynchronous rollouts
3. Review questions: interview questions and a self-check