GRPO, DPO, RLVR: which part does each change?

Suppose your model can already answer questions and you want to improve its answers. You might have a dataset of responses with labels saying which ones are better. Or you might generate new responses during training and score them as you go.

Standard offline DPO uses the first setup. PPO and GRPO are commonly used in the second, but turn scores into learning signals differently. RLVR answers a separate question: can a program check the result and provide a reward? It can be combined with PPO or GRPO; it is not the next algorithm in that sequence.

Start with the diagram to see which problem each method addresses. Then follow a group of answers from rewards to a training signal, and calculate DPO loss for a preference pair. Derivations and code are there to expand when you want them.

Separate three decisions first

For example, “GRPO with verifiable rewards” is a valid combination. RLVR can also use a Critic-based method; it does not inherently mean “only two models.” Weight sharing, caching, and offloading further change the actual memory footprint.

GRPO: compare answers to the same question

Classic PPO typically trains a Critic to estimate expected return. GRPO instead samples several answers to the same prompt and constructs a relative signal from their rewards, without a separately trained Critic. The DeepSeekMath paper introduces this approach; it does not require rewards to come from a program.

The group contains several responses sampled from one policy for one prompt, not a team of different models. The comparison updates that same policy; it does not aim to make every response in the group more likely.

Four answers, one correct

For example, a model writes four implementations for the same coding problem, and only the second passes the current test suite. Assigning 1 for a pass and 0 otherwise gives [0, 1, 0, 0]. This is a teaching example, not an experimental result; passing those tests does not prove correctness on every input.

Using the population standard deviation within a group of size $G$, and temporarily ignoring the stabilizer $\epsilon$:

$$ \begin{aligned} \bar r &= \frac{1}{G}\sum_i r_i,\\ s &= \sqrt{\frac{1}{G}\sum_i(r_i-\bar r)^2},\\ \hat A_i &= \frac{r_i-\bar r}{s+\epsilon}. \end{aligned} $$

The mean is $0.25$ and the standard deviation is about $0.433$. The implementation that passes receives an advantage of about $1.732$; each other answer receives $-0.577$. This does not identify useful lines of code. It says which result was better than the group average under the current scoring rule.

import math

rewards = [0.0, 1.0, 0.0, 0.0]
mean_reward = sum(rewards) / len(rewards)
variance = sum((reward - mean_reward) ** 2 for reward in rewards) / len(rewards)
deviation = math.sqrt(variance)
advantages = [(reward - mean_reward) / deviation for reward in rewards]
assert math.isclose(advantages[1], math.sqrt(3))
assert math.isclose(sum(advantages), 0, abs_tol=1e-12)
print([round(advantage, 3) for advantage in advantages])

If all four rewards are zero, or all four are one, centering produces zeros. That group provides no distinguishing policy-gradient signal. The total gradient need not vanish if other terms, such as KL, remain active. For how common such groups are in a batch, and how to resample them, see DAPO's dynamic sampling.

Reward, advantage, and loss are different numbers

The passing answer earned a reward of 1. Why did 1.732 then appear? The first number records a test result; the second compares that answer with its group. Updating the model also involves how its probability of producing the answer has changed.

Take one token from that passing response. Given the same prompt and response prefix, suppose the old policy assigned probability 0.20 and the current policy assigns 0.26. The ratio is 1.3. Its denominator is neither the reward nor a probability from the Reference model.

With a clipping interval of [0.8, 1.2] and this positive advantage, the clipped objective takes the smaller of 1.3 × 1.732 and 1.2 × 1.732, approximately 2.078. That is a single-token policy objective to maximize; a loss to minimize uses its negative. It is not the whole batch loss and does not include KL or other terms. DeepSeekMath §4.1.1–4.1.2

import math

def clipped_policy_term(ratio, advantage, epsilon=0.2):
    clipped_ratio = min(max(ratio, 1 - epsilon), 1 + epsilon)
    return min(ratio * advantage, clipped_ratio * advantage)

advantage = math.sqrt(3)
old_probability, current_probability = 0.20, 0.26
ratio = current_probability / old_probability
objective = clipped_policy_term(ratio, advantage)
loss_term = -objective
assert math.isclose(objective, 1.2 * math.sqrt(3))
assert math.isclose(clipped_policy_term(1.6, advantage), objective)
print(round(objective, 3), round(loss_term, 3))

Two roles are easy to confuse here. The old policy generated the batch, and its saved log-probabilities supply the ratio denominator. The Reference supplies a baseline for a KL constraint when one is enabled. Their weights may coincide at some point, but their jobs differ. With log-probabilities, compute the ratio as exp(current_logp - old_logp), not by dividing them. If sampling settings change the generation distribution, check which distribution those probabilities describe.

The Reference is neither the training dataset nor a gold answer. Given the same prompt and response prefix, the current policy might assign [0.8, 0.2] to two candidate tokens while the Reference assigns [0.5, 0.5]. KL compares those conditional distributions, not generated text directly against a dataset. See the two-token KL example for exact sums and sampled estimates. The Reference stays fixed during an optimization phase; if an algorithm refreshes it between outer iterations, record that schedule rather than assuming it stays unchanged throughout training.

Returning to that one-point reward: it records a pass on the current tests. The advantage gives a relative signal, and the probability ratio and clipping determine its contribution to the objective. None guarantees that the next answer will be correct. That still needs evaluation on new problems not used for training.

Removing the Critic does not make training free

That last distinction matters. A correct final answer does not imply that every intermediate step was correct. Applying a sequence reward to its tokens is a training estimator, not stepwise causal attribution.

The TRL GRPO documentation exposes choices for reward scaling, KL, and loss aggregation. Record those settings when reading code: implementations called GRPO may differ in group standard deviation, length normalization, and KL handling. Evaluate the useful group signal alongside its sampling cost.

DPO: put preference comparisons directly into the policy loss

Suppose each prompt already has a preferred answer $y_w$ and a less preferred answer $y_l$. Standard offline DPO trains on these pairs without a fresh rollout, separate RM training, or a Critic at every training step. That does not mean the dataset never required generation, or that preference labels were free.

Ask, for example, “Explain overfitting in one sentence.” One answer says, “The model learns incidental details in the training examples and performs worse on new data”; another says, “The model has not learned the training data yet.” An annotator prefers the first. DPO learns directly from that relative preference, without first asking a reward model to score each sentence separately. Learning this pair does not guarantee correct choices on every new question.

Why does the Reference appear in the loss?

The Reference is a frozen model, not a reference answer. It assigns probabilities to both responses in the dataset, giving us a starting point for comparison. It does not need to generate another answer to imitate. The preference label—not the Reference—says which response is better.

For our overfitting example, score the same prompt and response with the current model and the Reference. This tells us whether that response has become more or less likely relative to the starting point. DPO compares that change for the preferred answer against the change for the rejected one. Original paper §4

Calculate the loss for one pair

Suppose the current model assigns sequence log-probabilities of $-2,-4$ to the preferred and rejected answers, while the Reference assigns $-3$ to both. These are illustrative numbers; a higher log-probability means the model assigns more probability to that response:

Subtracting the two changes gives $1-(-1)=2$. With $\beta=0.2$, the sigmoid input is $0.4$ and loss is about $0.513$. If the current model equals the Reference, both changes are zero and loss is $\log 2\approx0.693$—even when the two responses originally had different probabilities.

import math

chosen_logp, rejected_logp = -2.0, -4.0
reference_chosen, reference_rejected = -3.0, -3.0
beta = 0.2
relative_gap = (chosen_logp - reference_chosen) - (rejected_logp - reference_rejected)
preference_logit = beta * relative_gap
loss = math.log1p(math.exp(-preference_logit))
assert math.isclose(preference_logit, 0.4)
assert math.isclose(loss, 0.5130152523999526)
assert loss < math.log(2)
print(round(loss, 4))

These small values are convenient for hand calculations; production implementations should use stable logsigmoid / softplus operations. Standard sequence log-probability sums answer-token log-probabilities, with correct prompt, padding, and EOS handling. Replacing the sum with a token mean changes the objective. The TRL DPO documentation helps connect the equations to data formats and loss implementations.

DPO can improve the relative gap without necessarily increasing the chosen answer's absolute probability: lowering the rejected answer's probability can also reduce this loss, as a three-output counterexample works out. A fixed dataset also cannot reveal failures that never appear in it.

RLVR: programs give feedback, but programs have blind spots

RLVR uses verifiable outcomes as rewards, such as answer matching, compilation results, or test passes. It specifies where feedback comes from, not a unique policy-update algorithm. DeepSeek-R1 is one public example using rule-based rewards.

“Deterministic” does not mean “complete”: a “sorting function” that always returns [1, 2, 3] passes a test containing only [3, 1, 2]. Verifiable rewards starts from this example and takes apart, item by item, what the verifier failed to check. Other gaps can lie in answer extraction, empty test sets, exception handling, or timeouts. An optimizer receives the score you implemented, not the requirements you forgot to implement. A deterministic verifier can remove some subjective scoring errors without eliminating specification gaming. Anthropic's controlled study illustrates how gaps between reward and intent can be exploited; it does not establish that every training run will do so.

Separate training checks from independent evaluation: reserve unseen problems and boundary cases, test new input types, and add human audits where needed. Changing the question while retaining the same faulty checker is not enough independence.

Choose based on what you actually have

Before asking which name is newer, sketch where data comes from, who scores it, how the loss is computed, and who independently checks the result. Those four questions tell you what an algorithm change actually changes.

Next, use the evaluation checklist to test whether a higher score supports your claim; for data reuse, revisit on-policy and off-policy learning.