RLHF: evaluation and review questions

The four common roles are Actor, Critic, Reward, and Reference; they do not require four full models to be resident simultaneously. Actor and Critic update weights in this setup. Reward is frozen after stage 2; Reference is a frozen SFT copy. Actor and Reference start identical, but only the Actor changes as policy training proceeds.

Preferences learned from limited data may not generalize to later policy outputs. Stronger optimization can find high-scoring answers that people do not actually prefer, such as accommodating a stylistic bias instead of improving the answer.

Reference KL adds a cost for deviating from an anchor. Increasing $\beta$ generally strengthens that cost, but its effect also depends on reward scale. Removing KL does not guarantee collapse, and retaining it does not guarantee freedom from reward hacking. Compare its presence and coefficient with independent evaluation rather than treating a nonzero coefficient as a law. See Reference and Critic for the derivation and numerical example.

The Critic estimates state value $V_t$ to construct advantages such as $\hat A_t=\hat G_t-V_t$. A suitable baseline can reduce variance; TD or GAE also introduces a bootstrapping bias tradeoff. Training without a Critic is still possible; REINFORCE is one example.

GRPO constructs relative signals from answers to the same prompt instead of training a separate Critic. It saves Critic-related costs but requires group sampling; all-correct or all-wrong groups (see DAPO's dynamic sampling), reward scales, and length normalization still matter. Counting fewer models does not establish lower total training cost.

DPO uses a derivation: KL-constrained reward maximization has a closed-form optimum, so the reward can be rewritten in terms of the policy and the preference loss differentiated directly. The reward model and the RL loop both disappear.

Standard DPO's cost is offline preference data. The policy changes while the data does not, so it cannot actively explore its current failures. PPO and other online RL methods resample from the current policy and are more natural for exploration, environment interaction, or verifiable multi-step outcomes, but rollout is expensive and training less stable.

DPO therefore cannot replace PPO everywhere, but PPO is not universally better either. Use DPO when high-quality static preferences cover the task; use online methods when the current policy must keep producing new evidence. Online DPO variants reinforce that the real distinction is the data-and-feedback loop, not only the loss name.

Bradley–Terry directly constrains the gap between chosen and rejected answers to the same prompt. Adding a constant to every score leaves the loss unchanged, so the zero point has no identifiable meaning; a score of $2.4$ is not “2.4 units of satisfaction.”

A fixed Reward Model's raw outputs can of course be used numerically, but comparisons across prompts, domains, or model versions require evidence that calibration and scale are stable. Implementations may also whiten or normalize rewards; that is an optimization choice, not a rule implied by Bradley–Terry.

A verifier only checks the conditions implemented in it. Incomplete tests, faulty answer extraction, or missing boundary cases can reward incorrect outputs. A deterministic program need not express the full task correctly.

RLVR fits tasks or subgoals with automatically checkable outcomes. Open-ended dialogue can also use format or tool-argument checks, but these do not cover all aspects of answer quality. Reserve independent problems, different test families, and human review where needed to distinguish task learning from checker exploitation. Verifiable rewards starts from a worked counterexample: a constant-output “solution” that passes a single sorting test.