Pre-trained LLMs are powerful but raw: they continue text rather than follow instructions, and they may produce harmful or unhelpful outputs. This week we cover the techniques used to adapt pre-trained models: supervised fine-tuning (SFT), parameter-efficient fine-tuning (PEFT), LoRA, and reinforcement learning from human feedback (RLHF).
A pre-trained model trained on next-token prediction will, given the prompt "Translate this to French: The cat sat on the mat", continue the text in a way that looks like training data — perhaps generating more English examples — rather than actually translating.
Fine-tuning teaches the model to:
SFT continues training on a curated dataset of (prompt, completion) pairs:
Prompt: "Summarise the following article in two sentences: ..."
Completion: "The article discusses ... The key finding is ..."
The training objective is the same (cross-entropy on the completion tokens). The model learns the mapping from instructions to helpful responses.
Fine-tuning on a small dataset can cause the model to “forget” capabilities learned during pre-training. Mitigated by:
Fine-tuning all parameters of a 7B or 70B model requires enormous GPU memory and is expensive. PEFT methods update only a small fraction of parameters.
Hu et al. (2022) propose representing weight updates as low-rank matrices.
For a weight matrix W ∈ R^{d × k}, instead of learning ΔW ∈ R^{d × k} (dk parameters), learn:
ΔW = B A
where B ∈ R^{d × r}, A ∈ R^{r × k}, r << min(d, k)
The number of trainable parameters is r(d + k) instead of dk.
With r=8, a 4096×4096 matrix goes from 16.7M to 65k trainable parameters — a 256× reduction.
The adapted forward pass:
h = W x + (B @ A) x * scaling_factor
The original weights W are frozen; only A and B are updated.
Dettmers et al. (2023): quantise the frozen base model to 4-bit (NF4 quantisation), then apply LoRA on top.
This allows fine-tuning a 65B model on a single 48GB GPU — previously impossible.
| Method | Idea |
|---|---|
| Adapters | Insert small bottleneck layers between transformer layers |
| Prefix tuning | Prepend learned “soft tokens” to the input |
| Prompt tuning | Only tune the embedding of a short learned prefix |
| IA³ | Scale activations with learned vectors |
Instruction tuning is SFT on a diverse mixture of tasks formatted as instructions. The key insight (Wei et al., 2022; FLAN): fine-tuning on many instruction types improves generalisation to unseen instructions.
FLAN-T5 and FLAN-PaLM were trained on 1,800+ tasks. They significantly outperform the base model on zero-shot benchmarks.
SFT alone is not sufficient to produce consistently helpful, harmless, and honest outputs. RLHF (Christiano et al., 2017; Ouyang et al., 2022) adds a human preference signal.
Stage 1 — SFT: Fine-tune the base model on a curated dataset of demonstrations.
Stage 2 — Reward Model Training:
RM(prompt, completion) → scalar reward
Stage 3 — RL Optimisation (PPO):
reward = RM(prompt, completion) - β * KL(π_θ || π_SFT)
The RM is an imperfect proxy for human preferences. The LLM may learn to produce outputs that score highly on the RM without being genuinely helpful — for example, very long or very confident responses.
An alternative to RLHF: use the LLM itself to critique and revise its responses based on a set of principles (a “constitution”), reducing reliance on human labels.
Rafailov et al. (2023): sidestep explicit RL by directly optimising the policy on preference pairs with a simple classification loss. Simpler to implement and often performs as well as PPO.
L_DPO = -log σ(β log(π_θ(y_w|x)/π_ref(y_w|x)) - β log(π_θ(y_l|x)/π_ref(y_l|x)))
where y_w is the preferred completion and y_l is the dispreferred completion.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", load_in_4bit=True)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
lora_config = LoraConfig(r=8, lora_alpha=32, target_modules=["q_proj", "v_proj"],
lora_dropout=0.05, bias="none", task_type="CAUSAL_LM")
model = get_peft_model(model, lora_config)
model.print_trainable_parameters() # e.g. "trainable params: 4,194,304 || all params: 6,742,609,920"
trainer = SFTTrainer(model=model, tokenizer=tokenizer, train_dataset=dataset,
dataset_text_field="text", max_seq_length=512)
trainer.train()
See practicals/week07_practical.py:
facebook/opt-125m or gpt2) on a custom instruction dataset using LoRA.fine_tune_llm.ipynb, PEFT.md, LLaMA_lora_int8.ipynb