Week 7 introduced RLHF as one component of the fine-tuning pipeline. This week we go considerably deeper: deriving the mathematics from first principles, understanding the reward model in detail, working through the PPO update rule as applied to language models, examining failure modes, and surveying the frontier of RLHF alternatives. By the end of this week students should be able to implement every stage of RLHF from scratch on a toy problem.
Supervised fine-tuning (SFT) teaches a model to imitate demonstrations. But human preferences are not always easily captured by demonstrations alone:
RLHF reframes the problem: instead of asking humans to write ideal completions, ask them to compare two completions and say which they prefer. This comparison signal is cheaper to collect and more reliable — humans are better at ranking than generating.
┌─────────────────────────────────────────────────────────────────┐
│ Stage 1: Supervised Fine-tuning (SFT) │
│ Pre-trained LLM → Fine-tune on curated demonstrations │
│ Result: π_SFT (a good starting policy) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 2: Reward Model Training │
│ Collect pairwise comparisons (y_w ≻ y_l | x) │
│ Train RM to assign scalar reward r(x, y) │
│ Result: r_φ (a proxy for human preference) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 3: RL Optimisation (PPO) │
│ Optimise π_θ to maximise E[r_φ(x,y)] - β·KL(π_θ || π_SFT) │
│ Result: π_RLHF (the final aligned model) │
└─────────────────────────────────────────────────────────────────┘
The SFT model is the initialisation for RL. It is trained on a dataset of high-quality (prompt, response) pairs:
L_SFT = -∑_t log π_θ(y_t | x, y_{<t})
The SFT model serves two roles:
Human annotators are shown a prompt x and two completions y_w (winner / preferred) and y_l (loser / dispreferred). They choose which they prefer. This produces a dataset:
D = {(x^(i), y_w^(i), y_l^(i))}_{i=1}^{N}
Annotators may be given a rubric covering:
We model human preference as arising from an underlying latent quality score r*(x, y):
P(y_w ≻ y_l | x) = σ(r*(x, y_w) - r*(x, y_l))
where σ is the sigmoid function. This is the Bradley-Terry model of paired comparisons.
We initialise the reward model from the SFT model and add a linear head that maps the final hidden state to a scalar:
r_φ(x, y) = W^T h_{final}(x, y)
The reward model is trained to minimise the negative log-likelihood of the observed preferences:
L_RM = -E_{(x,y_w,y_l) ~ D} [ log σ(r_φ(x, y_w) - r_φ(x, y_l)) ]
This is equivalent to binary cross-entropy on the preference pair, where the label is always “y_w is better.”
We treat the language model as a policy π_θ that maps a prompt x to a completion y. The goal is to maximise:
J(θ) = E_{x ~ D, y ~ π_θ(·|x)} [ r_φ(x, y) ] - β · KL(π_θ(·|x) || π_ref(·|x))
The two terms:
| **β · KL(π_θ | π_ref)**: a KL divergence penalty that keeps the policy close to the SFT reference policy. This prevents the policy from collapsing to reward-hacking completions that score well on r_φ but are degenerate. |
The KL penalty can be absorbed into a per-token reward:
r̃(x, y) = r_φ(x, y) - β · ∑_t log [π_θ(y_t | x, y_{<t}) / π_ref(y_t | x, y_{<t})]
PPO (Schulman et al., 2017) is the standard RL algorithm used in RLHF. It is an on-policy actor-critic method.
Actor (the LLM being trained): π_θ — generates completions.
Critic (value function): V_ψ(x, y_{<t}) — estimates the expected future reward from state (x, y_{<t}).
Advantage estimate: how much better is action y_t than the average?
A_t = r̃_t + γ V_ψ(s_{t+1}) - V_ψ(s_t)
In practice for LLMs, with γ = 1 and using GAE (Generalised Advantage Estimation):
A_t^{GAE} = ∑_{k=0}^{T-t} (γλ)^k δ_{t+k} where δ_t = r̃_t + V_ψ(s_{t+1}) - V_ψ(s_t)
Clipped surrogate objective:
L_PPO = E_t [ min(ρ_t A_t, clip(ρ_t, 1-ε, 1+ε) A_t) ]
| where ρ_t = π_θ(y_t | x, y_{<t}) / π_old(y_t | x, y_{<t}) is the importance sampling ratio. |
The clip prevents excessively large updates when π_θ deviates far from π_old.
for each iteration:
1. Sample prompts x ~ D
2. Generate completions y ~ π_θ(·|x) [rollout]
3. Score completions: r = r_φ(x, y) [reward]
4. Compute KL penalty against π_ref [regularise]
5. Compute advantages using critic V_ψ [estimate]
6. Update π_θ with clipped PPO objective [actor update]
7. Update V_ψ with MSE loss on returns [critic update]
This is computationally expensive: at each step we need four model forward passes — the actor, critic, reference policy, and reward model.
Reward hacking (Amodei et al., 2016; Skalse et al., 2022) occurs when the policy finds behaviours that achieve high reward model scores without actually satisfying the underlying human preference.
| Behaviour | Why it scores highly | Why it is bad |
|---|---|---|
| Very long responses | RMs often reward thoroughness | Wastes tokens; buries the answer |
| Excessive hedging | RMs reward safety | Becomes unhelpful (“I can’t be sure but…”) |
| Sycophantic agreement | RMs reward positively-toned responses | Confirms user misconceptions |
| Repetition | Padding to seem comprehensive | Reduces information density |
| Confident incorrect answers | Confidence may be rewarded | Hallucination |
Monitor the KL divergence from π_ref throughout training. Rapid KL growth without corresponding human evaluation improvement signals reward hacking.
The reward-KL frontier: as β decreases (weaker KL penalty), reward increases but so does the gap between RM score and true human preference. Optimal β is task-dependent.
Rafailov et al. (2023) show that the RLHF objective with a KL constraint has a closed-form optimal policy:
π*(y|x) = π_ref(y|x) exp(r*(x,y)/β) / Z(x)
Substituting back and rearranging, the reward can be expressed in terms of the policy:
r*(x,y) = β log [π*(y|x) / π_ref(y|x)] + β log Z(x)
Plugging this into the Bradley-Terry preference model and optimising directly over π_θ gives the DPO loss:
L_DPO(θ) = -E_{(x,y_w,y_l)} [ log σ( β log[π_θ(y_w|x)/π_ref(y_w|x)] - β log[π_θ(y_l|x)/π_ref(y_l|x)] ) ]
DPO advantages:
DPO limitations:
A simpler RL algorithm than PPO for language models: for each prompt, sample K completions and use the mean reward of the other K-1 as a baseline for each:
∇J ≈ ∑_k (r_k - mean_{j≠k} r_j) ∇ log π_θ(y_k | x)
Lower variance than vanilla REINFORCE; no separate value network required.
DeepSeek-R1 (2025): samples a group of completions per prompt, normalises rewards within the group, and uses the normalised advantage for the PPO-style update. No critic needed; trains efficiently on mathematical reasoning tasks.
Generate new completions with the current policy, score them with the RM, construct new preference pairs, and run DPO. Bridges the gap between offline DPO and online PPO — better coverage of the current policy’s distribution.
Generate K completions per prompt, keep only those with reward above a threshold, and fine-tune on them with SFT. Simple and effective; used by LLaMA-2 (Touvron et al., 2023) as a first pass before PPO.
| Method | RM needed | RL loop | Stability | Data efficiency |
|---|---|---|---|---|
| SFT | No | No | High | Moderate |
| Rejection sampling | Yes | No | High | Low |
| DPO | No (π_ref needed) | No | High | Moderate |
| RLOO | Yes | Yes | Medium | Moderate |
| PPO-RLHF | Yes | Yes | Low | High |
| Iterative DPO | Yes | Partial | Medium | High |
| GRPO | Yes | Yes | Medium | High |
Bai et al. (2022, Anthropic) replace human preference labels with AI-generated critiques.
Stage 1 — Supervised Learning from AI Feedback (SL-CAI):
Stage 2 — Reinforcement Learning from AI Feedback (RLAIF):
Benefits: scalable (no human raters needed for RM training), more consistent than crowd-sourced annotation, allows explicit statement of values.
Standard reward models score the final output. For multi-step reasoning tasks (mathematics, code), this is a weak signal — the model may reach the right answer via wrong reasoning, or wrong answer via mostly correct reasoning.
Process reward models assign a reward to each reasoning step:
r_total = ∑_t r_t(x, y_{≤t})
Benefits:
Drawbacks:
OpenAI’s Let’s Verify Step by Step (Lightman et al., 2023) shows PRMs outperform outcome reward models on MATH.
Human annotation is noisy:
Implications:
Compare the RLHF model against the SFT baseline using human evaluation or LLM-as-judge:
Win Rate = (# prompts where RLHF is preferred) / (# total prompts)
A win rate > 50% indicates improvement.
Plot RM score vs human preference rating on a held-out set. The correlation should be high early in training; degradation in correlation signals reward hacking.
See practicals/week13_practical.py: