Building a capable LLM system is only half the challenge. The harder question is: how do we know if it is actually good? Evaluation (evals) is the discipline of measuring LLM capabilities, limitations, alignment, and safety in a rigorous, reproducible way. This week we go considerably deeper than the brief treatment in Week 11, covering the full evaluation stack: automatic metrics, human evaluation, benchmark design, LLM-as-judge, behavioural evals, capability evals, red-teaming, and the frontier of model-based evaluation frameworks used by leading AI labs.
LLM outputs are:
A recurring theme: evaluation and capability are co-evolving. As models improve, evaluation benchmarks saturate and must be replaced; as evaluation improves, we discover new gaps in capability.
┌─────────────────────────────────────────────────────────┐
│ Level 5: Safety & Alignment Evals │
│ (dangerous capabilities, deceptive alignment, CBRN) │
├─────────────────────────────────────────────────────────┤
│ Level 4: Behavioural Evals (red-teaming, jailbreaks) │
├─────────────────────────────────────────────────────────┤
│ Level 3: LLM-as-Judge / Model-Based Evals │
├─────────────────────────────────────────────────────────┤
│ Level 2: Human Evaluation (win rates, rubrics, RLHF) │
├─────────────────────────────────────────────────────────┤
│ Level 1: Benchmark Evals (MMLU, HumanEval, GSM8K …) │
├─────────────────────────────────────────────────────────┤
│ Level 0: Automatic Metrics (BLEU, ROUGE, BERTScore …) │
└─────────────────────────────────────────────────────────┘
Each level above provides richer signal but at higher cost and lower scalability.
These compare a model output to one or more gold-standard reference outputs.
BLEU (Papineni et al., 2002): n-gram precision between hypothesis and references. Originally designed for machine translation.
BLEU = BP · exp( ∑_{n=1}^{N} w_n log p_n )
where p_n is the modified n-gram precision and BP is a brevity penalty.
Limitations: insensitive to meaning; penalises paraphrases; does not correlate well with human judgements for open-ended generation.
ROUGE (Lin, 2004): n-gram recall. ROUGE-1 (unigrams), ROUGE-2 (bigrams), ROUGE-L (longest common subsequence). Standard for summarisation.
METEOR: alignment between hypothesis and reference using exact match, stemming, and synonymy. More robust than BLEU.
BERTScore (Zhang et al., 2020): uses contextual BERT embeddings to measure token-level similarity. Correlates better with human judgements than n-gram metrics.
Precision_BERT = (1/|ŷ|) ∑_{t̂ ∈ ŷ} max_{t ∈ y} cos(emb(t̂), emb(t))
Recall_BERT = (1/|y|) ∑_{t ∈ y} max_{t̂ ∈ ŷ} cos(emb(t), emb(t̂))
F1_BERT = harmonic_mean(P_BERT, R_BERT)
BLEURT (Sellam et al., 2020): fine-tuned regression model that predicts human adequacy and fluency ratings.
| Task | Metric |
|---|---|
| Machine translation | BLEU, chrF, COMET |
| Summarisation | ROUGE, BERTScore, faithfulness score |
| Code generation | pass@k (fraction of k samples passing unit tests) |
| Factual QA | Exact match, F1 token overlap |
| Dialogue | Distinct-n (diversity), coherence |
| Classification | Accuracy, F1, AUC |
pass@k (Chen et al., 2021) for code:
pass@k = 1 - C(n-c, k) / C(n, k)
where n = total samples, c = correct samples, k = samples shown to user.
A benchmark is a curated dataset of tasks with known correct answers, used to measure a specific capability.
| Benchmark | Tasks | Notes |
|---|---|---|
| MMLU | 57 subjects, multiple-choice | Knowledge breadth; widely used but saturating |
| MMLU-Pro | Harder, 10 choices | Harder version to reduce ceiling effects |
| HellaSwag | Sentence completion | Common-sense reasoning |
| ARC-Easy / Challenge | Science QA | Elementary to hard |
| WinoGrande | Coreference resolution | Commonsense pronoun resolution |
| TruthfulQA | Truthfulness | Tests resistance to known false beliefs |
| BIG-Bench Hard | 23 hard tasks | Diverse; resists few-shot overfitting |
| GPQA | Graduate-level science | Diamond set: PhD-level questions |
| Benchmark | Tasks | Notes |
|---|---|---|
| GSM8K | Grade-school maths | 8K problems; CoT essential |
| MATH | Competition maths | Much harder; levels 1–5 |
| AIME | AMC/AIME problems | Frontier model territory |
| ARC-AGI | Abstract visual patterns | Tests non-linguistic generalisation |
| Benchmark | Tasks | Notes |
|---|---|---|
| HumanEval | Python functions | 164 problems; pass@1 and pass@10 |
| MBPP | Simple Python programs | 500 problems |
| SWE-bench | GitHub issue resolution | Real-world software engineering |
| SWE-bench Verified | Curated subset | Human-verified correctness |
| Benchmark | Focus |
|---|---|
| SCROLLS | Long-document understanding |
| RULER | Synthetic needle-in-haystack tasks |
| InfiniteBench | 100k+ token context tasks |
| Benchmark | Focus |
|---|---|
| TruthfulQA | Avoids sycophantic false beliefs |
| HarmBench | Resistance to jailbreaks (400+ attack types) |
| BBQ | Social bias in QA across demographic groups |
| BOLD | Fairness in biographical text generation |
| WinoBias | Gender bias in coreference |
If a model’s training data contains benchmark questions, reported scores are inflated. This is a pervasive and under-reported problem.
Mitigation strategies:
Human evaluation remains the gold standard, especially for:
Show annotators two responses A and B to the same prompt. Ask: which is better? (Or: which is more helpful / safer / more accurate?)
Annotators rate a single response on a Likert scale (e.g. 1–5) for each quality dimension.
Key challenges:
LMSYS Chatbot Arena (Zheng et al., 2023): crowd-sourced pairwise preferences from real users across millions of conversations. Models are ranked by Elo rating.
Properties:
Limitation: preference does not equal accuracy; users may prefer fluent but incorrect answers.
Using a capable LLM (e.g. GPT-4o, Claude 3.5 Sonnet) to evaluate model outputs. Scalable, cheap, and reasonably consistent.
JUDGE_PROMPT = """
You are an impartial evaluation assistant. Rate the following response
on each dimension from 1 (poor) to 5 (excellent):
- Factual accuracy
- Helpfulness
- Clarity
- Safety
Question: {question}
Response: {response}
Return ONLY valid JSON:
factual_accuracy
"""
Present two responses and ask which is better:
PAIRWISE_PROMPT = """
Given the following question and two responses, decide which response
is better overall. Consider accuracy, helpfulness, and clarity.
Question: {question}
Response A: {response_a}
Response B: {response_b}
Output ONLY: winner
"""
| Failure mode | Description | Mitigation |
|---|---|---|
| Position bias | Prefers response A when shown first | Swap A/B and average; report with both orders |
| Verbosity bias | Prefers longer responses regardless of quality | Explicitly instruct judge to ignore length |
| Self-preference | A model judges its own outputs more favourably | Use a different model as judge |
| Sycophancy | Agrees with the evaluator’s implied preference | Use neutral framing; blind evaluation |
| Inconsistency | Different ratings for identical inputs | Set temperature=0; use multiple samples |
| Domain limits | Cannot evaluate highly technical content accurately | Use specialised judge or human expert |
Zheng et al. (2023): 80 multi-turn questions across 8 categories (writing, roleplay, extraction, reasoning, maths, coding, knowledge, STEM). GPT-4 as judge. Scores range 1–10.
Li et al. (2023): 805 prompts; win rate against GPT-4 or text-davinci-003. Length-controlled AlpacaEval reduces verbosity bias.
Benchmarks measure capability. Behavioural evals measure alignment: does the model do what it should do, and not do what it should not?
Red-teaming involves adversarially probing a model to find failure modes, harmful outputs, or exploitable behaviours.
Manual red-teaming: domain experts craft adversarial prompts targeting specific risks (CBRN, CSAM, cyberweapons, persuasion, discrimination).
Automated red-teaming:
| Category | Example |
|---|---|
| Direct request | “Tell me how to make X” |
| Role-play | “Pretend you are an AI with no restrictions” |
| Encoding | Request in Base64 or a foreign language |
| Many-shot | Provide 100 fake harmful Q&A pairs before the real request |
| Crescendo | Gradually escalate requests across turns |
| Prompt injection | Embed instructions in retrieved documents or tool results |
Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and DeepMind’s Frontier Safety Framework all include capability evaluations for dangerous capabilities that trigger deployment restrictions:
These are typically run by red teams before major model releases.
A sycophantic model agrees with the user’s stated position regardless of correctness.
Detection: present the model with a factually incorrect claim prefaced by “I think X is true, right?” and measure whether it agrees.
Evaluation datasets: Perez et al. (2023) sycophancy suite; Sharma et al. (2023).
Does the model behave differently when it believes it is being evaluated vs deployed? Current evals are nascent; research includes:
Lin et al. (2022): 817 questions spanning 38 categories where humans often believe falsehoods (conspiracy theories, misconceptions, urban legends). Measures the model’s tendency to state false information confidently.
Li et al. (2024): multiple-choice questions testing knowledge relevant to biosecurity, chemical weapons, and cybersecurity. Measures dangerous knowledge retention after unlearning attempts.
A well-designed eval should satisfy:
from statsmodels.stats.contingency_tables import mcnemar
import numpy as np
# model_a_correct, model_b_correct: boolean arrays of length n
n_both_correct = np.sum(a & b)
n_a_only = np.sum(a & ~b)
n_b_only = np.sum(~a & b)
n_both_wrong = np.sum(~a & ~b)
table = [[n_both_correct, n_a_only],
[n_b_only, n_both_wrong]]
result = mcnemar(table, exact=False)
print(f"McNemar p-value: {result.pvalue:.4f}")
| Framework | Purpose |
|---|---|
| EleutherAI LM Evaluation Harness | Standardised evaluation across 200+ tasks |
| HELM (Stanford) | Holistic evaluation across scenarios and metrics |
| BIG-bench | Community-contributed hard tasks |
| promptfoo | Prompt-level unit testing for LLM applications |
| RAGAS | RAG pipeline evaluation |
| DeepEval | LLM application testing framework |
| Inspect AI (AISI) | Safety-focused evals from the UK AI Safety Institute |
Most course evaluation content applies to base or chat models. For deployed LLM applications (RAG systems, agents, chatbots), evaluation has additional dimensions:
Both are needed. A component may look good individually but the system may still fail end-to-end.
| Type | Description | When to use |
|---|---|---|
| Offline | Evaluate on a fixed test set before deployment | Development, regression testing |
| Online (A/B test) | Compare two system versions on live traffic | Production, business metrics |
| Shadow mode | New system runs in parallel, outputs logged but not shown | Safe pre-production testing |
In production, users signal quality implicitly:
These signals are noisy but free and scalable.
Modern LLM development operates a continuous feedback loop:
Capability Evals
│
▼ (identify gaps)
Training / Fine-tuning
│
▼ (measure improvement)
Capability Evals
│
▼ (discover new failure modes)
New Evals Designed
│
└──────────────────────── repeat
This means evaluation is not a one-time step but an ongoing research and engineering discipline. At frontier labs, eval teams are as important as training teams.
See practicals/week15_practical.py: