Actor–Critic and GAE: making credit assignment concrete

A high-scoring trajectory does not make every action in it good. Actor–Critic asks: at that state, how much more expected return does this action offer than following the existing policy's action choice? The Actor chooses actions; the Critic estimates value. Here we use a state-value Critic with a stochastic policy. TD3 and SAC later use Q Critics instead.

What each network receives

The networks can share an encoder, but then their losses interact through shared parameters. A small value MSE does not establish a good policy; the target itself may be wrong.

From a TD residual to advantage

The advantage $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s)$ measures how much better this action is than the policy's average choice at that state. If the Critic were exactly $V^\pi$, the expected one-step TD residual conditional on $(s_t,a_t)$ would equal $A^\pi(s_t,a_t)$:

$$ \delta_t=r_t+\gamma V^\pi(s_{t+1})-V^\pi(s_t),\qquad \mathbb E[\delta_t\mid s_t,a_t]=Q^\pi(s_t,a_t)-V^\pi(s_t). $$

In practice $V_\phi$ is approximate. Generalized Advantage Estimation (GAE) combines future TD residuals with geometrically decreasing weights:

$$ \hat A_t^{\rm GAE}=\sum_{l\ge0}(\gamma\lambda)^l\delta_{t+l} =\delta_t+\gamma\lambda\hat A_{t+1}. $$

At $\lambda=0$, only one step contributes. At $\lambda=1$, a complete terminated trajectory telescopes to $G_t-V(s_t)$. A truncated rollout retains its bootstrap tail and is not full MC. Intermediate values trade value-estimation error against sampling noise; no $\lambda$ is universally best.

Why GAE has these weights

Sum $n$ discounted residuals. Intermediate value terms cancel, leaving:

$$ \hat A_t^{(n)}=\sum_{l=0}^{n-1}\gamma^l\delta_{t+l} =\sum_{l=0}^{n-1}\gamma^l r_{t+l}+\gamma^n V(s_{t+n})-V(s_t). $$

This is an $n$-step return minus the starting value estimate. Rather than choose one $n$, GAE mixes lengths. Take $N$ remaining steps within the same episode, without crossing a reset. Retain a tail bootstrap if the segment ends before the task does:

$$ \hat A_t^{\rm GAE} =(1-\lambda)\sum_{n=1}^{N-1}\lambda^{n-1}\hat A_t^{(n)} +\lambda^{N-1}\hat A_t^{(N)}. $$

The final term takes the remaining weight. Do not mechanically multiply it by another $(1-\lambda)$ in a finite trace. These weights sum to 1; collecting coefficients of each residual gives $(\gamma\lambda)^l$.

The endpoints become intuitive: $\lambda=0$ selects one step; $\lambda=1$ selects the longest segment. Longer does not automatically mean more accurate: reward noise and Critic error both matter.

Calculate a trace, then move the slider

For $\delta=[1,2,-1]$, $\gamma=0.9$, and $\lambda=0.8$:

$$ \hat A_0=1+0.72\times2+0.72^2\times(-1)=1.9216. $$

The residuals are fixed teaching data. Moving the slider does not retrain a Critic; it changes only the estimator.

Keep the two masks separate

One mask asks whether the next state has future value. The other asks whether later residuals belong to this trajectory. True termination disables both. External truncation can bootstrap, but the GAE recurrence must not cross a reset into another episode.

If the task itself is defined to end when time runs out, that is true termination; remaining time should generally be part of a finite-horizon task's state. External truncation here means the task could continue but the collector stopped.

def gae(rewards, values, next_values, terminated, truncated,
        gamma=0.99, trace_decay=0.95):
    lengths = {len(part) for part in
               (rewards, values, next_values, terminated, truncated)}
    if len(lengths) != 1:
        raise ValueError("trajectory arrays must have equal lengths")
    result = [0.0] * len(rewards)
    carry = 0.0
    for step in reversed(range(len(rewards))):
        bootstrap = 0.0 if terminated[step] else next_values[step]
        residual = rewards[step] + gamma * bootstrap - values[step]
        same_episode = not (terminated[step] or truncated[step])
        carry = residual + gamma * trace_decay * same_episode * carry
        result[step] = carry
    return result

Each next_values entry must use that transition's final observation, not a reset observation. Under Gymnasium 1.x's default next-step autoreset, the reset observation arrives on the step after the episode ends; mask that step out rather than continuing GAE through it. See MC, TD, and boundaries and Farama's explanation. A rollout buffer that ends before the task does can also bootstrap its final value while stopping recurrence at the buffer boundary.

How does Critic error reach the Actor?

Write $V_\phi(s)=V^\pi(s)+e(s)$. The expected one-step residual becomes:

$$ \mathbb E[\delta_t\mid s,a]=A^\pi(s,a)+\gamma\mathbb E[e(s')\mid s,a]-e(s). $$

If the current state is overestimated by 2 and the successor of a particular action underestimated by 3 on average, with $\gamma=0.9$, that action's advantage estimate has error $0.9(-3)-2=-4.7$. This may change a single sampled update's direction.

The errors do not play the same role in the expected policy gradient. Fixed $-e(s)$ is an action-independent baseline, whose expected gradient contribution cancels under the current policy. Successor error can depend on the action and generally does not cancel. Advantage-estimation error is not itself gradient bias. This follows the previous chapter's baseline argument; nonlinear operations such as PPO clipping need separate analysis.

Increasing $\lambda$ reduces dependence on intermediate short-horizon estimates but adds noise from longer observed returns. Truncated segments still retain final bootstrap error. Even on complete on-policy terminated trajectories, $\lambda=1$ gives $\mathbb E[\hat A_t\mid s,a]=A^\pi(s,a)-e(s)$, not necessarily an unbiased advantage estimate. If the Critic was fixed before sampling, however, $-e(s)$ cancels in the expected policy gradient. An unbiased policy-gradient estimate need not use unbiased advantage estimates. Discounted objectives still require the previous chapter's $\gamma^t$ weighting convention.

Padding and averaging change the objective too

Suppose trajectory A contributes valid token losses $[1,1,1]$ and B contributes one loss of 9. Token averaging gives 3; averaging each trajectory and then trajectories gives 5. Both are definable, but weight trajectories differently.

Centering $[1,2]$ gives $[-0.5,0.5]$. This is a within-batch relative transformation, not the true $Q-V$ of each action. It may help optimization, but it changes the numbers, so do not treat it as a harmless identity transformation.

Separate the Actor and Critic updates

The Actor minimizes $-\log\pi_\theta(a_t\mid s_t)\,\mathrm{stopgrad}(\hat A_t)$. The Critic often fits $\mathrm{stopgrad}(\hat A_t+V_{\rm old}(s_t))$. Fix this target before optimization rather than letting it drift with the value being trained.

Use unnormalized GAE when adding back the old value. Old values $[10,10]$ and raw advantages $[1,3]$ give value targets $[11,13]$. Standardizing advantages to $[-1,1]$ first gives $[9,11]$, changing the return units. The Actor may use a separately normalized copy; do not change the Critic target with it.

PPO adds a new/old probability ratio and clipping to the Actor objective. GAE is not PPO itself. The existing four-case PPO clipping interactive shows that additional mechanism.

Reference: Original GAE paper. Next, see how far a policy should update. For the Q-based route, continue with DQN.