To teach these concepts effectively, you do not need to master the entire reinforcement learning canon. You can bypass action-value methods (like Q-learning) entirely and focus strictly on Policy Gradients—the specific mathematical family that underpins both PPO and the derivations of DPO.
Since you are ultimately aiming to teach how these algorithms enforce alignment and safety guardrails, this learning plan strips away the standard RL textbook fluff (like navigating grid-worlds) and focuses strictly on mathematical optimization and policy bounds.
Here is a focused, sequential curriculum to get you to mastery.
| Key concept: The Policy ($\pi_\theta(a | s)$), which is the probability distribution over actions given a state, parameterized by a neural network. |
When teaching this, students often struggle to grasp why PPO is notoriously finicky to train compared to DPO. The interactive dashboard below allows you to simulate the training dynamics of both approaches.
Pedagogical tip: When designing assessment descriptors for this module, require students to explain the failure modes of both algorithms. PPO typically fails through reward hacking or catastrophic forgetting during the RL loop, while DPO typically fails due to out-of-distribution prompts since it is bound tightly to its static preference dataset.
Explaining Direct Preference Optimization (DPO) to students who aren’t familiar with Reinforcement Learning (RL) is highly achievable because DPO’s core innovation is that it completely bypasses the complex machinery of RL.
An intuitive framework, an educational analogy, and a curated list of technical resources can help you master the material.
When teaching students without an RL background, avoid terms like “policies,” “rewards,” “states,” or “actions.” Instead, frame the problem as a classic binary classification task (sorting things into two categories), which they likely already understand.
Here is a step-by-step breakdown you can use for your lecture:
Start by explaining that after a Large Language Model (LLM) learns how to predict the next word (Pre-training) and learns how to follow instructions (Supervised Fine-Tuning, or SFT), it still doesn’t know what humans prefer. It might give an answer that is technically correct but incredibly rude, or it might give a response that is too wordy. We need a way to teach it human style, safety, and helpfulness.
Use an analogy of an essay-writing student and a teacher to contrast traditional methods with DPO.
The mathematical breakthrough of DPO (introduced by Rafailov et al.) is proving that an LLM can calculate its own “preference score” implicitly.
Instead of training a separate model to output a reward number (e.g., “+5 points for politeness”), DPO looks at the mathematical probability of the model generating the text.
The training objective can be explained to students as a balancing act using a simple math intuition:
\[\text{Maximize } \log \left( \frac{\text{Probability of Chosen}}{\text{Probability of Rejected}} \right)\]You are simply forcing the network to widen the gap between the good response and the bad response. To prevent the model from breaking or forgetting its baseline knowledge, a constraint (using a reference model) keeps the updates within reasonable boundaries. It behaves exactly like standard binary cross-entropy loss—the exact same loss function used in basic logistic regression or image classification.
To master the underlying mathematics, derivations, and implementation details before you teach, these video resources are highly recommended:
| Title: *Direct Preference Optimization (DPO) and Friends | RLHF & Post-training Course, Lecture 6* (by Nathan Lambert) |
For a visually engaging and clear breakdown of the core concepts, the video Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learning provides an excellent explanation of how DPO eliminates the need for complex reinforcement learning when training language models.
Week 7 introduced DPO as an alternative to PPO-RLHF, and Week 13 derived the DPO loss from first principles. This week we go much deeper: the full mathematical story from first principles, the relationship to reward models, the failure modes specific to DPO, the many variants that have emerged (IPO, KTO, SimPO, ORPO, and others), and how to implement DPO end-to-end on a real preference dataset. By the end of this week students should be able to implement DPO from scratch, explain every term in the loss, choose between DPO variants for a given setting, and diagnose training failures.
PPO-RLHF requires:
DPO sidesteps all of this. It shows that the same alignment objective can be optimised directly from preference pairs, without ever training an explicit reward model or running RL. The result is simpler code, more stable training, and competitive performance.
Recall the RLHF objective (from Week 13):
max_{π_θ} E_{x~D, y~π_θ(·|x)} [r(x, y)] − β · KL(π_θ(·|x) || π_ref(·|x))
This asks: find the policy π_θ that maximises expected reward while staying close to the reference policy π_ref (measured by KL divergence). The KL penalty prevents reward hacking.
This problem has an analytical solution (Ziebart et al., 2008; see also the energy-based models literature):
π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp(r(x,y) / β)
where Z(x) = Σ_y π_ref(y|x) exp(r(x,y)/β) is the partition function (normalising constant).
This says: the optimal policy upweights completions that have high reward and downweights completions that have low reward, relative to the reference policy.
Taking logs of the optimal policy expression and rearranging:
log π*(y|x) = log π_ref(y|x) + r(x,y)/β − log Z(x)
⟹ r(x,y) = β · log[π*(y|x) / π_ref(y|x)] + β · log Z(x)
The Bradley-Terry model says human preference for y_w over y_l is:
P(y_w ≻ y_l | x) = σ(r(x, y_w) − r(x, y_l))
Substituting our expression for r:
r(x, y_w) − r(x, y_l)
= β · log[π*(y_w|x) / π_ref(y_w|x)]
− β · log[π*(y_l|x) / π_ref(y_l|x)]
Note: the β log Z(x) terms cancel because they are the same for both completions given the same prompt x.
Replace π* with our trainable policy π_θ and minimise the negative log-likelihood of the observed preferences:
L_DPO(θ) = −E_{(x, y_w, y_l) ~ D}
[log σ( β · log[π_θ(y_w|x)/π_ref(y_w|x)]
− β · log[π_θ(y_l|x)/π_ref(y_l|x)] )]
This is the DPO loss. It is a binary cross-entropy loss over the log-ratio advantage of preferred over dispreferred completions.
Define the implicit reward:
r̂_θ(x, y) ≡ β · log[π_θ(y|x) / π_ref(y|x)]
Then DPO is simply:
L_DPO(θ) = −E [log σ( r̂_θ(x, y_w) − r̂_θ(x, y_l) )]
The model’s own log-probability ratio relative to the reference policy is the reward. There is no separate reward model — the policy and the reward model are unified into a single parameterisation.
Taking the gradient with respect to θ:
∇_θ L_DPO = −β · E [(1 − σ(ĥ_θ)) ·
(∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x))]
where ĥ_θ = β(log[π_θ(y_w|x)/π_ref(y_w|x)] − log[π_θ(y_l|x)/π_ref(y_l|x)]).
The factor (1 − σ(ĥ_θ)) is the residual — how far the model is from correctly ranking the pair. When the model already ranks y_w above y_l by a large margin, this factor approaches 0 and the gradient is small (the pair is easy, not contributing much to learning). When the model has the ranking wrong, the factor is large and the gradient pushes hard to correct it.
In practice this means DPO naturally down-weights pairs the model already gets right and focuses learning on difficult pairs — a form of implicit curriculum.
import torch
import torch.nn.functional as F
def dpo_loss(
model,
ref_model,
prompt_ids, # (B, T_p)
chosen_ids, # (B, T_c) preferred completion
rejected_ids, # (B, T_r) dispreferred completion
beta: float = 0.1,
):
"""
Compute the DPO loss for a batch of preference pairs.
All sequences are (batch, seq_len) token-id tensors.
"""
def sequence_log_prob(lm, input_ids, completion_ids):
"""
Compute sum of log P(completion | prompt) under model lm.
input_ids: (B, T_p + T_c) — prompt + completion concatenated
completion_ids: used to select which positions to score
Returns: (B,) tensor of sum log probs over completion tokens
"""
with torch.no_grad() if lm is ref_model else torch.enable_grad():
logits = lm(input_ids).logits # (B, T, V)
# Score only the completion tokens (shift by 1 for causal LM)
log_probs = F.log_softmax(logits[:, :-1, :], dim=-1) # (B, T-1, V)
# Gather log probs of the actual completion tokens
T_p = prompt_ids.shape[1]
comp_logits = log_probs[:, T_p - 1:, :] # (B, T_c, V)
token_lp = comp_logits.gather(
-1, completion_ids.unsqueeze(-1)
).squeeze(-1) # (B, T_c)
return token_lp.sum(dim=-1) # (B,)
chosen_input = torch.cat([prompt_ids, chosen_ids], dim=1)
rejected_input = torch.cat([prompt_ids, rejected_ids], dim=1)
# Policy log probs
pi_chosen_lp = sequence_log_prob(model, chosen_input, chosen_ids)
pi_rejected_lp = sequence_log_prob(model, rejected_input, rejected_ids)
# Reference log probs (frozen)
ref_chosen_lp = sequence_log_prob(ref_model, chosen_input, chosen_ids)
ref_rejected_lp= sequence_log_prob(ref_model, rejected_input, rejected_ids)
# Log-ratio advantages
log_ratio_chosen = pi_chosen_lp - ref_chosen_lp
log_ratio_rejected = pi_rejected_lp - ref_rejected_lp
# DPO loss
loss = -F.logsigmoid(beta * (log_ratio_chosen - log_ratio_rejected)).mean()
# Useful diagnostics
with torch.no_grad():
chosen_rewards = beta * log_ratio_chosen.mean().item()
rejected_rewards = beta * log_ratio_rejected.mean().item()
reward_margin = chosen_rewards - rejected_rewards
accuracy = (log_ratio_chosen > log_ratio_rejected).float().mean().item()
return loss, {
"chosen_reward": chosen_rewards,
"rejected_reward": rejected_rewards,
"reward_margin": reward_margin,
"accuracy": accuracy,
}
β controls the strength of the KL penalty:
Typical values in practice: β = 0.1 (Rafailov et al., 2023), β = 0.01 (SimPO).
The reference policy π_ref should be the SFT model — the model before alignment fine-tuning. Using a weaker reference (e.g. the base model pre-SFT) degrades performance because the reference is too far from useful behaviour.
DPO is sensitive to preference data quality:
Unlike cross-entropy training where loss directly measures quality, DPO loss alone is not very interpretable. Monitor these additional metrics:
| Metric | Formula | Healthy sign | ||
|---|---|---|---|---|
| Reward accuracy | P(r̂(y_w) > r̂(y_l)) | Should increase toward 1.0 | ||
| Chosen reward | β · mean(log π_θ(y_w)/π_ref(y_w)) | Should increase | ||
| Rejected reward | β · mean(log π_θ(y_l)/π_ref(y_l)) | Should decrease | ||
| Reward margin | chosen_reward − rejected_reward | Should widen | ||
| KL from reference | KL(π_θ | π_ref) | Should stay bounded |
If the chosen reward increases but the rejected reward also increases, the model may be drifting uniformly away from the reference (reward hacking a proxy).
Azar et al. (2023) and Rafailov et al. (2024) identify a critical failure mode: DPO can decrease the probability of both chosen and rejected completions — even as the margin between them increases.
Why? The DPO gradient has two components:
In practice, the second term often dominates for out-of-distribution rejected completions. The model decreases both probabilities but decreases y_l faster, so the margin increases but absolute probabilities of the preferred completion drop.
Consequence: the model becomes less likely to generate the preferred completion entirely, even though it “prefers” it relative to the rejected one.
Mitigation: use a language modelling loss on the chosen completions alongside DPO (SFT + DPO combined training).
When trained for too many steps or with too small β, the policy can diverge far from the reference in ways that look good on DPO metrics but degrade generation quality.
Detection: monitor KL divergence from reference and generation quality (perplexity on a held-out set, or win rate against the reference in human eval).
DPO is an offline algorithm: it trains on a fixed preference dataset collected from a specific policy (typically the SFT model). As the policy π_θ drifts away from the data-collection policy, the preferences become off-distribution and the implicit reward signal becomes less reliable.
Mitigation: iterative DPO — generate new completions with the current policy, collect new preferences, retrain.
If annotators disagree (e.g. 60%/40% win rate), the preference signal is noisy. DPO treats all preferences as hard labels, which can cause instability.
Mitigation: weight pairs by confidence (e.g. annotator agreement score); use IPO (see below) which treats preferences as soft labels.
Since the original DPO paper (2023), many variants have been proposed to address its limitations.
DPO implicitly assumes the Bradley-Terry model is correctly specified. IPO removes this assumption by directly regularising the policy preference:
L_IPO(θ) = E [(log[π_θ(y_w|x)/π_ref(y_w|x)]
− log[π_θ(y_l|x)/π_ref(y_l|x)] − 1/(2β))²]
The target margin is 1/(2β) instead of infinity. This prevents the log-ratio from growing unboundedly and addresses likelihood displacement.
DPO requires paired (y_w, y_l) data for the same prompt. KTO relaxes this: it works with unpaired positive and negative examples — data of the form (x, y, label) where label ∈ {desirable, undesirable}, without needing them to be paired for the same x.
L_KTO(θ) = E_D [ w(y) · (1 − v_KTO(x, y, β)) ]
where v_KTO encodes prospect-theory value (humans weight losses more heavily than gains — Kahneman & Tversky, 1979).
KTO is useful when you have large datasets of labelled completions that are not paired.
SimPO makes two changes to DPO:
L_SimPO(θ) = −E [log σ( (β/|y_w|) log π_θ(y_w|x)
− (β/|y_l|) log π_θ(y_l|x) − γ )]
Benefits: no reference model needed (halves memory); length normalisation avoids length bias; target margin γ improves calibration.
ORPO merges the SFT loss and the preference loss into a single objective, eliminating the need for a separate SFT phase:
L_ORPO(θ) = L_SFT + λ · L_OR
where:
L_OR = −E [log σ( log (odds_θ(y_w|x) / odds_θ(y_l|x)) )]
odds_θ(y|x) = π_θ(y|x) / (1 − π_θ(y|x))
Benefits: one-stage training; no reference model; avoids likelihood displacement by maintaining the SFT objective.
Replace human preference labels with AI-generated labels (Constitutional AI / RLAIF style), then apply DPO on the resulting pairs. Scalable; avoids human annotation bottleneck. Quality depends on the strength of the AI labeller.
Repeatedly:
This addresses distribution shift. Each iteration uses data from the current policy, making the preference signal more on-distribution.
| Method | Ref model needed | Paired data needed | Key advantage | Key limitation |
|---|---|---|---|---|
| DPO | Yes (π_ref) | Yes | Simple derivation, stable | Likelihood displacement; offline |
| IPO | Yes | Yes | No BT assumption; bounded margin | More hyperparameters |
| KTO | Yes | No (unpaired) | Works with labelled data | Weaker signal than pairs |
| SimPO | No | Yes | Memory efficient; length-normalised | No KL anchor |
| ORPO | No | Yes | One-stage; no reference | SFT and alignment must balance |
| Iterative DPO | Yes | Yes (generated) | On-policy; fixes distribution shift | More expensive |
| Factor | Prefer DPO | Prefer PPO |
|---|---|---|
| Engineering complexity | DPO (simpler) | PPO (4 models) |
| Training stability | DPO | PPO (with tuning) |
| Reward model available | Either | PPO (can use RM directly) |
| Online exploration needed | PPO | PPO |
| Compute budget | DPO | PPO |
| Reasoning/math tasks | PPO / GRPO | PPO / GRPO |
| Preference data quality | High quality → DPO | Lower quality → PPO |
| Iterative alignment | Iterative DPO | PPO (naturally online) |
In practice: DPO (or a variant) is the default starting point for most instruction-following alignment tasks. PPO is preferred for tasks requiring exploration, multi-step reasoning rewards, or when a high-quality RM is available.
# Typical HuggingFace TRL format
dataset = [
{
"prompt": "What is the capital of France?",
"chosen": "The capital of France is Paris.",
"rejected": "The capital of France is Lyon.",
},
...
]
Ensure:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import DPOTrainer, DPOConfig
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2",
load_in_4bit=True)
ref_model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2",
load_in_4bit=True)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
lora_cfg = LoraConfig(r=16, lora_alpha=32,
target_modules=["q_proj", "v_proj"],
task_type="CAUSAL_LM")
training_args = DPOConfig(
beta=0.1,
max_length=512,
max_prompt_length=256,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=5e-7,
num_train_epochs=1,
output_dir="./dpo_output",
)
trainer = DPOTrainer(
model=model,
ref_model=ref_model,
args=training_args,
train_dataset=dataset,
tokenizer=tokenizer,
peft_config=lora_cfg,
)
trainer.train()
Log the following at each eval step:
rewards/chosen and rewards/rejectedrewards/accuracies (fraction of pairs correctly ranked)rewards/margins (mean chosen − rejected reward)logps/chosen and logps/rejected (absolute log probs, to detect likelihood displacement)kl (approximate KL from reference)The RLHF objective with KL regularisation is a special case of reward maximisation with an f-divergence constraint. Different divergences give different variants:
DPO is structurally similar to contrastive loss functions in representation learning (InfoNCE, SimCLR). The key difference: DPO contrastively adjusts probability ratios rather than embedding distances.
The trained DPO policy π_θ implicitly defines a reward model r̂(x,y) = β log[π_θ(y|x)/π_ref(y|x)]. This implicit reward can be extracted and used directly for ranking completions or training a separate RM.
See practicals/week16_practical.py: