GSPO and ASPO: how should one answer share its gradient?

Last reviewed: 2026-10 · Prerequisites: PPO clipping and GRPO

An answer is correct overall, yet one token has become less likely. Should training raise the whole answer together, or give that token extra attention? GSPO and ASPO take different approaches. Start with where their weights enter the update, rather than the acronyms.

Separate 3 policies

For an already sampled response $y$, the token probability ratio is:

$$ \rho_t=\exp\left(\log\pi_\theta(y_t\mid x,y_{<t})-\log\pi_b(y_t\mid x,y_{<t})\right). $$

Both policies must see the same prefix. Dividing probabilities for different freely generated responses does not have this meaning. Prompt tokens, tool outputs, and padding are not automatically policy actions either.

GSPO: aggregate the change over a response

GSPO uses a length-normalized sequence ratio and sequence-level clipping. The response tokens share a group-relative advantage $A$:

$$ s=\exp\left(\frac{1}{T}\sum_{t=1}^{T}\log\rho_t\right),\qquad J=\min\left(sA,\operatorname{clip}(s,1-\epsilon,1+\epsilon)A\right). $$

This is the geometric mean of token ratios, not their arithmetic mean or the raw sequence ratio $\prod_t\rho_t$. Taking the $T$-th root reduces length sensitivity in the numerical scale but changes the weight. It is not an unbiased whole-trajectory importance-sampling identity.

When reading the paper's motivation, separate an unstable estimate from an invalid identity. With the required support, even a single importance-weighted sample can be unbiased, although its variance may be large. Sampling one token per position is not, by itself, a proof of invalidity. A token ratio leaving the old prefix distribution uncorrected is a different issue: in this two-step example, every step-2 ratio is 1 while the response success rate changes from 0.35 to 0.65. GSPO's experimental results are worth examining; they do not establish that every token-level surrogate is invalid.

Two tokens moving in opposite directions

These are invented numbers, not a model measurement. Let $A=1$ and the token ratios be [2, 0.5]:

On the unclipped GSPO branch, with fixed advantage:

$$ \nabla J=\frac{sA}{T}\sum_t\nabla\log\pi_\theta(y_t\mid x,y_{<t}). $$

Every token has the same external coefficient $sA/T$, not the same parameter gradient: their network Jacobians still differ. The example also reveals a tradeoff: sequence aggregation can conceal large local changes.

GSPO does not simply keep high-scoring answers. Change this example's $A$ to −1: $s$ remains 1, and the local derivative with respect to each selected log-probability is −0.5. The negative signal is still present. Clipping depends on the ratio and the direction of the advantage, not just the answer's score.

MoE adds an implementation concern: the paper's §5.3 reports training instability with routing changes. Different selected experts do not, by themselves, make the output probability ratio undefined. Check the shared prefix, actual sampling distribution and support, and whether rollout and training implementations compute consistent probabilities. Switching to GSPO does not replace those checks.

import math

def clipped_surrogate(ratio, advantage, lower=0.8, upper=1.2):
    if not 0 < lower <= 1 <= upper or not math.isfinite(ratio) or ratio <= 0:
        raise ValueError("invalid ratio or clipping interval")
    return min(ratio * advantage, min(upper, max(lower, ratio)) * advantage)

def sequence_ratio(log_ratios, action_mask):
    if len(log_ratios) != len(action_mask) or any(type(flag) is not bool for flag in action_mask):
        raise ValueError("one Boolean action mask per token is required")
    selected = [value for value, active in zip(log_ratios, action_mask) if active]
    if not selected or not all(math.isfinite(value) for value in selected):
        raise ValueError("at least one finite action log-ratio is required")
    return math.exp(sum(selected) / len(selected))

ratios = [2.0, 0.5]
assert math.isclose(sequence_ratio([math.log(value) for value in ratios], [True, True]), 1)
assert math.isclose(sum(clipped_surrogate(value, 1) for value in ratios) / 2, 0.85)
assert clipped_surrogate(0.5, -1) == -0.8

This checks objective values; it is not a trainer. GSPO thresholds need to be chosen for sequence ratios. The 0.2 margin is only convenient arithmetic, not a training recommendation. Identical group rewards, broken action masks, and misleading rewards do not disappear with GSPO.

ASPO: reverse the positive-advantage weight

ASPO (retitled “When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL” in v2, 2026-05-15; the method is in §6) changes token-level updates: retain an advantage-directed mask, use reciprocal weights for positive advantages and ordinary ratios for negative advantages, and bound extreme weights. The motivation is to treat tokens already moving strongly and those lagging behind differently.

The gradient is easy to get wrong. Write $\ell=\log p_\theta$, with behavior probability $p_b$. The positive branch uses:

$$ \widehat\rho=\frac{p_b p_\theta}{\operatorname{sg}(p_\theta^2)}, \qquad \text{forward value}=\frac{p_b}{p_\theta}, \qquad \frac{\partial\widehat\rho}{\partial\ell}=+\frac{p_b}{p_\theta}. $$

$\operatorname{sg}$ means stop-gradient. A directly differentiable 1 / ratio has a negative derivative, reversing the intended direction for a good token. Equal forward values do not imply equal training behavior.

One local update

Suppose behavior probability is 0.2, current probability is 0.1, and $A=1$. The ordinary ratio is 0.5; ASPO's positive-branch weight is 2. If neither masking nor a weight cap is triggered, their coefficients on $\nabla\log p_\theta$ are therefore 0.5 and 2.

This checks the local stop-gradient interpretation by holding the denominator fixed during differentiation, rather than recomputing it after each perturbation.

import math

behavior_probability = 0.2
current_probability = 0.1
frozen_square = current_probability ** 2
current_logp = math.log(current_probability)
step = 1e-6

def detached_denominator_value(log_probability):
    return behavior_probability * math.exp(log_probability) / frozen_square

derivative = (
    detached_denominator_value(current_logp + step)
    - detached_denominator_value(current_logp - step)
) / (2 * step)
assert math.isclose(derivative, 2.0, rel_tol=1e-6)

Without masking, dual clipping, and loss reduction, these lines are not a complete ASPO implementation. Continue with the authors' code. The v1 paper's token-masking condition printed the positive-advantage upper bound as $1-\epsilon_{high}$; v2 §6 corrects it to $1+\epsilon_{high}$, matching PPO. Still check the implementation before turning a sentence from the paper into a branch condition.

Decide with diagnostics, not names

Record rewards, lengths, ratio distributions, and valid-token counts on the same rollouts before changing the loss. Keep data, generation budget, initialization, and evaluation matched where possible.

These methods change an update rule, not the validity of a reward. Continue with DAPO's sampling and normalization, then stale data in asynchronous training.