DQN: why learning Q with a network can be unstable

With actions such as “left, right, stop,” a network can output 3 Q-values and choose the largest. It resembles regression, except the labels are not fixed ground truth: the target contains the network's own estimates.

For a concrete case where regression loss improves while values drift, start with the two-state fitted Q counterexample. This chapter explains how DQN mitigates related problems.

Turn a transition into a loss

A replay buffer stores $(s,a,r,s',d)$. The online network $Q_\theta$ predicts the current action's value; a slower target network $Q_{\bar\theta}$ constructs its label:

$$ y=r+\gamma(1-d)\max_{a'}Q_{\bar\theta}(s',a'),\qquad L(\theta)=\mathbb E[(Q_\theta(s,a)-\mathrm{stopgrad}(y))^2]. $$

Here $d$ means true termination, not every reset. Replay reduces strong sequential correlations and reuses interaction. A target network slows target drift. These mitigate problems; they do not guarantee convergence with neural approximation.

This is a semi-gradient update: differentiate the current Q, not the target. It is not a full gradient through every occurrence in a Bellman residual.

Double DQN separates selection from evaluation

A maximum tends to select actions whose noisy estimates are too high. Double DQN selects with the online network and evaluates with the target network:

$$ a^*=\arg\max_a Q_\theta(s',a),\qquad y_{\rm DDQN}=r+\gamma(1-d)Q_{\bar\theta}(s',a^*). $$

Let online Q be $[5,4]$, target Q be $[2,6]$, and $r=1,\gamma=0.9$. DQN uses the target maximum of 6, giving 6.4. Double DQN selects the first action using online Q, then uses its target value of 2, giving 2.8.

We do not know the true $Q^*$ here, so a smaller target is not evidence of greater accuracy. This example illustrates decoupling only. The networks remain correlated and can also underestimate.

Inspect shapes and gradients in code

The code uses the same Double DQN target but replaces squared error with Smooth L1—equivalent here to Huber loss with threshold 1—to limit the influence of large TD errors on gradients. Action selection and evaluation are unchanged.

import torch

def double_dqn_loss(online, target, states, actions, rewards,
                    next_states, terminated, gamma=0.99):
    if rewards.ndim != 1 or rewards.numel() == 0:
        raise ValueError("rewards must be a nonempty vector")
    if not rewards.is_floating_point():
        raise ValueError("rewards must be floating-point")
    if actions.shape != rewards.shape or terminated.shape != rewards.shape:
        raise ValueError("actions, rewards, and terminated must have shape [batch]")
    if actions.dtype != torch.long or terminated.dtype != torch.bool:
        raise ValueError("actions must be int64 and terminated must be boolean")
    if states.ndim != 2 or states.shape != next_states.shape or states.shape[0] != rewards.numel():
        raise ValueError("states and next_states must have matching [batch, state_dim] shapes")
    if not 0 <= gamma <= 1:
        raise ValueError("gamma must be in [0, 1]")
    prediction = online(states).gather(1, actions[:, None]).squeeze(1)
    with torch.no_grad():
        next_q = torch.zeros_like(rewards)
        continuing = ~terminated
        if gamma > 0 and continuing.any():
            successors = next_states[continuing]
            next_actions = online(successors).argmax(dim=1, keepdim=True)
            next_q[continuing] = target(successors).gather(1, next_actions).squeeze(1)
        expected = rewards + gamma * next_q
    return torch.nn.functional.smooth_l1_loss(prediction, expected)

Here states is [batch, state_dim], actions is int64 [batch], rewards is floating-point [batch], and terminated is boolean [batch]. The example uses a plain MLP with one device and matching floating-point precision. Shape checks reject [batch, 1] rewards rather than silently broadcasting them with [batch] Q values into a [batch, batch] table.

True terminal successors never enter the network: their target is the reward. Computing NaN and then multiplying by zero still produces NaN. External truncation is different and requires the valid final observation before reset. Networks with dropout or BatchNorm need explicit mode handling: no_grad is not eval(), and changing the valid batch affects BatchNorm statistics. This small implementation does not manage those training states for you.

Similar names, different changes

Exploration often uses $\epsilon$-greedy. Random training actions collect information; document whether evaluation disables them. Otherwise low scores from exploration and low scores from the policy itself become hard to separate.

Why does max turn zero-mean noise into overestimation?

Both actions have true value 0, with independent errors of +1 or −1, each equally likely:

Each action’s error has mean zero, but the maximum has expectation 0.5. Selection retains favorable noise. Evaluating the chosen action with independent zero-mean errors returns expectation 0 in this example.

Double DQN networks are not truly independent, so this explains motivation rather than proving unbiasedness. Frequent synchronization can correlate errors; infrequent synchronization can leave targets stale.

Replay and target networks are slow in different ways

Replay stores transitions, not permanent targets. Sample $(s,a,r,s')$, then construct labels using the specified target network. Shuffling reduces consecutive-sample correlation but does not ensure coverage of the new policy’s states.

Freezing the target network does not freeze every label. For the same transition, vanilla DQN's label stays fixed between hard synchronizations, assuming deterministic network outputs. Double DQN still selects actions with the changing online network. With $r=1,\gamma=0.9$, keeping target Q at $[2,6]$ but changing online Q from $[5,4]$ to $[3,4]$ changes its label from 2.8 to 6.4 without synchronizing the target.

Targets may be copied periodically or softly updated as $\bar\theta\leftarrow(1-\tau)\bar\theta+\tau\theta$. Specify whether synchronization counts environment or gradient steps: 1000 steps means different things with ten updates per interaction versus one.

Compare environment interactions and gradient updates, not just an hour of training. Otherwise more data and more computation are indistinguishable.

Check these before tuning

First test that terminal targets equal reward. Then verify that targets have no gradients, replay actions align with their states, and target synchronization uses the intended step units. If loss falls while return does not improve, inspect these semantics before enlarging the network.

Reference: Double DQN (arXiv v3, 2015). Next: What if continuous actions make enumeration impossible?