Ask, “I'm learning Python for the first time. What could I try today?” One answer gives half a page of language history; another walks you through a few lines of code you can run immediately. Both might be factually correct, but the second is more useful for this request.
It is difficult to capture every such distinction in a reference answer. Comparing two responses is often easier: which addresses the question, which omits something important, and which is padding?
RLHF uses feedback like this to keep adjusting a model's behavior after pretraining and SFT.
How does feedback become a training signal?
The classic approach collects human comparisons and trains a Reward Model to predict those preferences. The language model then generates new answers, receives scores, and updates through RL.
People compare answers → learn preferences → generate new answers → update from scores
The middle step deserves attention: training directly optimizes the reward model's score. If that model mistakes length for quality, answers may get longer rather than more useful. We therefore need a separate check of answer quality after training.
InstructGPT is a public example of this route. Later methods, including GRPO and DPO, change some of these steps; we will examine them separately.
Which names will come up?
The model generating answers—the one we want to improve—is the Actor. The Reward Model scores outcomes. A Critic estimates the return to expect from continuing a prefix, and a Reference retains an anchor policy to help measure how much behavior has changed.
There is no need to memorize every role at once. Follow how one answer is generated and receives feedback, then return to why each component is there.
Where to go next
- Why language models can use RL: how generating a token becomes an action
- Reward, value, and advantage: the result you got versus the result you expected
- Can we train on old answers?: what changes when the model moves beyond its data
- Three stages and four models: demonstrations, preferences, and policy updates
- Reference and Critic: estimating drift versus estimating return
- PPO clipping: why one good result should not keep earning more encouragement
- GRPO, DPO, and RLVR: which part each changes
- Did training help?: separating a score change from a useful improvement
On a first pass, follow that sequence. When implementing a loss or inspecting training logs, turn to Three KL estimators: why a sample estimate can be negative, and why a correct value does not necessarily give the gradient you intended.
For the underlying reinforcement-learning ideas, start with Deep RL foundations. If they are familiar, go straight to the question you want to explore.