You can take 2 points now, or wait one step for a possible 4. Which should you choose? Reinforcement learning asks not just what a decision earns now, but where it leads. We start with small questions like this, then work toward value estimates, policy updates, and complete training loops.
Each chapter starts with an example before introducing equations and code. On a first read, follow the examples; when reviewing, use the tables to compare algorithms and check implementations. The examples are original toy environments, not course assignments or reproduced slides.
If the names blur together, start with Q-learning, PPO, and DPO. For current difficulties, read signal, cost, and stability. For their origins, follow the technology tree into the RL timeline.
A route through the material
| Route | Read | What you should be able to do |
|---|---|---|
| Define the problem | MDPs and Bellman → MC, TD, and boundaries | Compute returns and training targets from a trajectory |
| Change the policy directly | Policy gradients → Actor–Critic and GAE → Natural gradient and TRPO | Explain gradients, baselines, the two masks, and why step size needs control |
| Choose actions through value | Fitted Q and convergence limits → DQN → DDPG / TD3 → SAC | Separate regression loss from policy performance and trace network gradients |
| Think ahead with a model | Model-based RL → Which model to learn → Planning and LQR → How planning and learning feed each other | Separate dynamics learning, planning, and policy learning; say what a model should predict and how search results become training targets |
| Understand data and rewards | Offline RL and OPE → Per-decision weights, doubly robust, and LSTD · Exploration and hierarchy · Imitation and rewards | State the added assumptions, not just the algorithm names |
| Run a meaningful experiment | Experiments and debugging → The LLM bridge | Check the implementation before interpreting reward as progress |
These lessons go beyond algorithm names. We work through Bellman residual bounds, TD targets, gradient directions, PPO clipping, and control trajectories, then examine how CQL, IQL, and FQE handle offline data. Hierarchy, IRL, and multi-agent learning are introductions for now, not comprehensive theoretical treatments.
Different reading goals, not separate sets of notes
| What you want today | How to read | What to take away |
|---|---|---|
| Build intuition | Follow the opening task, diagrams, and calculation tables; skip derivations for now | Explain why a component is needed |
| Understand the mechanism | Revisit equations, conditional expectations, symbols, and assumptions | Calculate an example and locate the limits of a guarantee |
| Implement it | Inspect data sources, targets, gradients, masks, and update timing | Debug in a tiny environment rather than relying only on curves |
Key calculations are visible in the text without moving sliders or opening an interactive panel. A few flows briefly highlight reading order; all content remains visible afterward, and reduced-motion settings are respected. Existing interactive experiments remain optional ways to change parameters, not substitutes for the lesson.
Find these 3 parts in any algorithm
Which policy collects data?→Estimate
What is the target?→Update
Where do gradients go?
| Method | Main learned object | Action selection | An easily missed cost |
|---|---|---|---|
| REINFORCE | Policy | Sample from the policy | High return variance; old data cannot be mixed in naively |
| DQN | Q-value | Discrete argmax with exploration during training | Bootstrapping, approximation, and off-policy data can destabilize learning |
| State-value Actor–Critic | Policy + V | Sample from the policy | Critic errors affect the Actor |
| TD3 / SAC | Policy + two Q-functions | A continuous-action policy | Reusing data saves interaction, not necessarily computation |
| Model-based | Dynamics, sometimes policy / value | Planning or policy | Accurate predictions do not ensure reliable planned behavior |
On-policy / off-policy compares the behavior policy collecting data with the target policy being evaluated or improved. Online / offline describes whether training can obtain new interactions. A method that keeps interacting while learning from older data can be both online and off-policy.
Prerequisites and notation
Conditional expectation, the chain rule, and gradients are enough to start. For a refresher on probability and gradients, see ML mathematics. The PyTorch examples assume familiarity with autograd and tensor shapes.
Throughout, $s_t$ is the state, $a_t$ the action, $r_t$ the reward received after that action, and $\gamma$ the discount. Finite episodes use $t=0,\ldots,T-1$; terminal value is zero. We use $\pi_\theta$ for a stochastic policy and $\phi$ for value parameters. Reward, return, and value are not interchangeable.
These terms recur throughout the notes:
| Term | Meaning here |
|---|---|
| Policy | Chooses an action or assigns action probabilities given a state |
| Reward | Feedback after one step, not the entire trajectory's outcome |
| Return | Future rewards accumulated from a step using the chosen convention |
| Value | Expected future return under a particular policy |
| Training target | The number fitted in this update; it may be an estimate, not ground truth |
| Bootstrapping | Using an existing value estimate for the future not yet observed |
| Trajectory / rollout | A sequence of states, actions, and rewards; a rollout may be only a segment |
Change one assumption at a time
After a worked example, change just one condition: replace termination with a timeout, move $\lambda$ from 0 to 1, or make two Q-networks rank actions differently. The small interactive experiments are for exactly this. They illustrate computations, not complete training runs or invented performance gains.
Run the reference calculations to check the arithmetic, then the PyTorch checks to inspect stop-gradient behavior. Neither requires a GPU.
Start with an experiment if you prefer
| Question | Chapter | What to change |
|---|---|---|
| How does reward propagate backward? | MDPs and Bellman | Adjust the discount and step through backups |
| Which future steps affect this advantage? | Actor–Critic and GAE | Adjust λ and inspect the contribution table |
| Why can better regression still diverge? | Fitted Q | Advance fits and compare value curves with MSE |
| How much does one policy update change? | Natural gradient and TRPO | Adjust the logit step and KL budget |
| How does feedback help a wrong model? | Model-based RL | Adjust model error and horizon; compare real trajectories |
After changing a parameter, try explaining the result before opening the calculation table. The point is not to memorize which way a curve moves, but to see how a change carries through the calculation.
Public references
For terminology and derivations, see Spinning Up: RL foundations; individual chapters link original papers. PPO and RLHF already have a separate sequence. Continue there after these foundations rather than memorizing the same definitions twice. To see these methods applied to multi-step agents and world models, continue to training agents with RL and world models in the frontier series.