An 18-week masters-level course on the internals of large language models and their real-world applications. The course combines rigorous theory with hands-on Python practicals, culminating in a group project.
This course takes students from first principles through to deploying LLM-powered applications. The first half focuses on internals (tokenisation, embeddings, attention, transformers, pre-training, fine-tuning). The second half focuses on applications (RAG, agents, tool use, evaluation, safety, healthcare AI, multimodal models). Week 13 provides a dedicated deep dive into Reinforcement Learning from Human Feedback (RLHF), covering the full mathematical derivation, PPO, DPO, reward hacking, Constitutional AI, and process reward models. Week 14 covers mechanistic interpretability (superposition, sparse autoencoders, circuits, the logit lens, grokking, knowledge editing). Week 15 is a dedicated deep dive into evaluation (evals): the full evaluation stack from automatic metrics through human evaluation, LLM-as-judge, benchmark design, red-teaming, safety evals, and evaluation for deployed LLM applications. Week 16 provides a deep dive into Direct Preference Optimisation (DPO): the full mathematical derivation from the RLHF objective, the gradient interpretation, practical implementation, failure modes (likelihood displacement, over-optimisation, distribution shift), and a survey of variants including IPO, KTO, SimPO, and ORPO. Week 17 covers efficiency in attention and alternative architectures: the quadratic cost of full attention, sparse attention patterns (sliding window, BigBird), FlashAttention (tiling and online softmax), linear attention approximations (Performer, RWKV), state space models (S4, Mamba), grouped-query attention, and hybrid Mamba/transformer architectures. Week 18 is a short, practical primer on reinforcement learning — just enough RL intuition (policies, rewards, REINFORCE, baselines) to understand why PPO, DPO, and GRPO are shaped the way they are.
Each week has:
materials/)practicals/)teaching_llm_applications/
├── README.md ← this file
├── requirements.txt ← Python dependencies
├── materials/
│ ├── week01_intro_and_history.md
│ ├── week02_tokenisation.md
│ ├── week03_embeddings.md
│ ├── week04_attention.md
│ ├── week05_transformer_architecture.md
│ ├── week06_pretraining_and_scaling.md
│ ├── week07_finetuning_and_rlhf.md
│ ├── week08_prompting_and_context_engineering.md
│ ├── week09_rag_and_retrieval.md
│ ├── week10_agents_and_tool_use.md
│ ├── week11_evaluation_safety_ethics.md
│ ├── week12_applications_and_project.md
│ ├── week13_rlhf_deep_dive.md
│ ├── week14_mechanistic_interpretability.md
│ ├── week15_evals.md
│ ├── week16_dpo.md
│ ├── week17_efficiency_attention.md
│ └── week18_intro_to_rl.md
└── practicals/
├── week01_practical.py
├── week02_practical.py
├── week03_practical.py
├── week04_practical.py
├── week05_practical.py
├── week06_practical.py
├── week07_practical.py
├── week08_practical.py
├── week09_practical.py
├── week10_practical.py
├── week11_practical.py
├── week12_practical.py
├── week13_practical.py
├── week14_practical.py
├── week15_practical.py
├── week16_practical.py
├── week17_practical.py
└── week18_practical.py
Week 1 — Introduction and History What are LLMs? A brief history from n-grams through Word2Vec to the Transformer era. Emergent abilities and in-context learning.
Week 2 — Tokenisation Byte-pair encoding (BPE), WordPiece, SentencePiece, tiktoken. Special tokens, vocabulary size trade-offs, tokenisation artefacts.
Week 3 — Embeddings and Representations Token embeddings, contextual vs static embeddings, sentence embeddings, cosine similarity, PCA / t-SNE / UMAP, sinusoidal and RoPE positional encodings.
Week 4 — The Attention Mechanism Scaled dot-product attention (Q, K, V), causal masking, multi-head attention, cross-attention, FlashAttention, visualising attention weights.
Week 5 — Transformer Architecture Full decoder-only architecture. LayerNorm, residual connections, feed-forward networks, weight tying, KV cache, encoder-only and encoder-decoder variants. Scaling model size.
Week 6 — Pre-training and Scaling Laws Pre-training data pipelines, next-token prediction objective, training infrastructure. Kaplan and Chinchilla scaling laws. Emergent abilities. Training instabilities.
Week 7 — Fine-tuning and RLHF (overview) Supervised fine-tuning (SFT), instruction tuning, LoRA, QLoRA, RLHF (reward model, PPO), Constitutional AI, Direct Preference Optimisation (DPO). See Week 13 for the full mathematical treatment of RLHF.
Week 8 — Prompting and Context Engineering Zero-shot, few-shot, chain-of-thought, self-consistency. Prompt patterns (role, format, decomposition, step-back). Structured output. Prompt injection and security.
Week 9 — Retrieval-Augmented Generation (RAG) RAG architecture (indexing, retrieval, generation). Chunking strategies. Embedding models for retrieval. Vector stores (FAISS, ChromaDB). Hybrid retrieval and re-ranking. Advanced patterns: HyDE, multi-query, Self-RAG. Evaluation with RAGAS.
Week 10 — Agents and Tool Use LLM agents, the ReAct framework, tool/function calling, agent memory and planning. Multi-agent systems (LangGraph, AutoGen, CrewAI, smolagents). Agent evaluation and safety.
Week 11 — Evaluation, Safety, and Ethics Automatic metrics (BLEU, ROUGE, BERTScore). Benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA). LLM-as-judge. Hallucination types and mitigations. Bias and fairness. AI safety and alignment (RLHF, Constitutional AI). Responsible deployment checklist.
Week 12 — Applications and Final Project LLMs in healthcare, science, education, and software engineering. Multimodal models. Production architecture. Final project guidelines, assessment rubric, and project ideas.
Week 13 — RLHF: Deep Dive Full mathematical derivation of RLHF from first principles. Bradley-Terry preference model. Reward model architecture and training. PPO actor-critic loop applied to language models: clipped surrogate objective, advantage estimation, KL controller, value clipping. Reward hacking: definition, examples, measurement, and mitigations. DPO derivation from the KL-constrained RLHF objective. Variants and alternatives: RLOO, GRPO, iterative DPO, rejection sampling fine-tuning. Constitutional AI and RLAIF. Process reward models (PRMs) for multi-step reasoning. Annotator agreement and reward model quality. Evaluation: win rate, MT-Bench, AlpacaEval.
Week 14 — Mechanistic Interpretability The superposition hypothesis. Sparse autoencoders (SAEs) for feature disentanglement. Circuits: induction heads, the IOI circuit, the greater-than circuit. Activation patching and causal scrubbing. The residual stream as a communication bus. The logit lens. MLP layers as key-value memories. Knowledge editing (ROME, MEMIT). Linear probing and its causal limitations. Grokking and the modular arithmetic circuit. Automated interpretability (ACDC, Bills et al.). Universality. Open problems and the TransformerLens toolkit.
Week 15 — Evals: Evaluating LLMs The evaluation stack from automatic metrics to safety evals. BLEU, ROUGE, BERTScore, MAUVE, pass@k. Benchmark design, contamination, and saturation (MMLU, GSM8K, HumanEval, SWE-bench, GPQA). Human evaluation: pairwise preference, Chatbot Arena, ELO ratings, annotator agreement. LLM-as-judge: setup, position bias, verbosity bias, self-preference, MT-Bench, AlpacaEval. Behavioural evals and red-teaming: jailbreak taxonomy, automated red-teaming, HarmBench. Safety and alignment evals: sycophancy, deceptive alignment, TruthfulQA, WMDP, dangerous capability evaluations. Eval design: the eval checklist, statistical significance (McNemar), eval frameworks (LM Eval Harness, HELM, Inspect AI). Evaluation for deployed LLM applications: component vs end-to-end, offline vs online, implicit feedback.
Week 16 — Direct Preference Optimisation (DPO) Full mathematical derivation of DPO from the KL-constrained RLHF objective: the analytical optimal policy, rearranging for the implicit reward, cancellation of the partition function, the Bradley-Terry substitution, and the final DPO loss. Gradient interpretation and implicit curriculum. PyTorch implementation from scratch. Key hyperparameters (β, reference policy, data quality). Training diagnostics: chosen/rejected rewards, reward margin, preference accuracy, likelihood displacement. Failure modes: likelihood displacement mechanism, over-optimisation, distribution shift, preference noise. DPO variants: IPO (bounded margin), KTO (unpaired data), SimPO (no reference model, length normalisation), ORPO (one-stage SFT + alignment). Iterative and online DPO. DPO vs PPO decision guide. Practical tips with TRL’s DPOTrainer.
Week 17 — Efficiency: Attention Variants and Alternative Architectures The quadratic O(n²) cost of full self-attention: time, memory, and practical limits. Sparse attention patterns: local/window attention, strided/dilated attention, global+local (Longformer, BigBird), sliding window attention in Mistral. FlashAttention: HBM vs SRAM bottleneck, tiling, online softmax algorithm, FlashAttention-2 and -3, PyTorch integration. Linear attention approximations: the kernel trick for attention, Performer (random Fourier features), RWKV (linear recurrent form), limitations. State space models (SSMs): continuous-time formulation, ZOH discretisation, S4 (HiPPO matrix, convolutional form, O(n log n) training), Mamba (selective/input-dependent B, C, Δ), Mamba vs transformer comparison, hybrid models (Jamba, Griffin). Grouped-query attention (GQA) and multi-query attention (MQA): KV cache reduction. Efficient inference architecture: combining FlashAttention + GQA + sliding window + speculative decoding.
Week 18 — Introduction to Reinforcement Learning A short, practical RL primer: the agent-environment loop, policies, rewards, and returns. The core REINFORCE update rule and the intuition behind it (push up actions that worked, push down ones that didn’t). Variance reduction via baselines and a one-paragraph sketch of actor-critic. A one-line summary of the conceptual path from REINFORCE to PPO and GRPO. Why single-turn RLHF behaves like a bandit problem rather than full multi-step RL.
Practitioner and research perspectives complement the core weekly material. See materials/guest_lectures.md for full details on each speaker, talk overviews, suggested background reading, and guidance on how to prepare questions.
| Speaker | Topic | Relevant weeks |
|---|---|---|
| Cole Robertson | Speech, language, and LLMs in industry — real-time spoken dialogue, ASR + LLM pipelines, startup deployment | 1, 8, 12 |
| Glasgow Startup (TBC) | LLMs for real-world applications — prototyping, fine-tuning vs RAG vs prompting, evaluation in the wild | 7, 8, 9, 11, 12 |
All practicals are self-contained Python scripts with inline comments. Run with:
python practicals/weekNN_practical.py
| Practical | What you build |
|---|---|
| Week 1 | Query GPT-2, inspect next-token probabilities, visualise top-k predictions, compare temperatures, compute perplexity |
| Week 2 | Compare tokenisers (tiktoken, BERT, GPT-2 HF), train BPE from scratch, visualise token boundaries, estimate API cost |
| Week 3 | Sentence embeddings, cosine similarity matrix, semantic search, PCA / t-SNE / UMAP visualisation, sinusoidal positional encoding |
| Week 4 | Scaled dot-product attention (NumPy), causal masked attention, multi-head attention (PyTorch), extract real BERT attention weights |
| Week 5 | Complete decoder-only Transformer from scratch, train on Shakespeare, plot loss curves, generate text at multiple temperatures |
| Week 6 | Scaling law experiment: train models of 5 different sizes, plot loss vs parameters and vs FLOPs, Chinchilla analysis |
| Week 7 | LoRA from scratch, fine-tune a toy LM, Bradley-Terry reward model, DPO loss demonstration |
| Week 8 | Zero-shot vs few-shot vs CoT comparison, self-consistency decoding, structured extraction, prompt injection demo |
| Week 9 | Full RAG pipeline (chunk → embed → FAISS index → retrieve → generate), BM25 hybrid retrieval, faithfulness evaluation |
| Week 10 | ReAct agent from scratch, tool calling loop, two-agent orchestrator-specialist system, success rate evaluation |
| Week 11 | ROUGE / BERTScore / LLM-as-judge evaluation, hallucination rate measurement, gender bias probing, jailbreak attempts |
| Week 12 | End-to-end LLM application (RAG + agent + Streamlit UI), systematic evaluation on 20 queries |
| Week 13 | Full RLHF pipeline from scratch: synthetic preference dataset, Bradley-Terry reward model, PPO actor-critic update loop, DPO training, reward hacking detection; RM score / KL divergence / win rate plots |
| Week 14 | Mechanistic interpretability: logit lens across GPT-2 layers; activation patching heatmap (IOI heads); linear probe for verb detection per layer; sparse autoencoder trained on MLP activations with feature inspection; grokking on modular arithmetic |
| Week 15 | Evals: BLEU/ROUGE/BERTScore/MAUVE comparison; LLM-as-judge pipeline with position-bias mitigation (swap A/B); position bias measurement; sycophancy probing; mini red-team across three jailbreak categories; factual QA eval with McNemar significance testing |
| Week 16 | DPO from scratch: implement DPO, IPO, and SimPO losses; train a toy LM on synthetic preferences; monitor chosen/rejected rewards, margin, accuracy, and KL; demonstrate likelihood displacement; two-iteration online DPO; comparison plots across all three methods |
| Week 17 | Efficiency: time/memory scaling of naive vs tiled (FlashAttention-style) vs sliding-window attention; S4-style SSM from scratch — verify recurrent == convolutional form via FFT; Mamba selective SSM with input-dependent Δ; KV-cache memory comparison (MHA vs GQA vs SSM) across sequence lengths up to 262k tokens; output quality analysis of sliding window at varying window sizes |
| Week 18 | REINFORCE basics: a minimal policy-gradient agent on a toy bandit, trained with and without a baseline, showing the variance reduction directly in a plot |
The objectives are organised by theme, expressed using measurable action verbs at masters level, and grounded in Bloom’s revised taxonomy. They map directly to the weekly topics and assessments.
By the end of Week 2, students will be able to:
By the end of Week 3, students will be able to:
By the end of Week 4, students will be able to:
nn.MultiheadAttention.By the end of Week 5, students will be able to:
By the end of Week 6, students will be able to:
By the end of Week 7, students will be able to:
By the end of Week 8, students will be able to:
By the end of Week 9, students will be able to:
By the end of Week 10, students will be able to:
By the end of Week 11, students will be able to:
By the end of Week 12, students will be able to:
By the end of Week 13, students will be able to:
By the end of Week 16, students will be able to:
By the end of Week 18, students will be able to:
The objectives above span all six cognitive levels:
| Level | Verbs used | Examples from this course |
|---|---|---|
| Remember | Define, identify, list, name | Define token, perplexity, LoRA rank, KV cache, Bradley-Terry model |
| Understand | Explain, describe, distinguish, summarise | Explain causal masking, Chinchilla scaling, reward hacking, KL penalty |
| Apply | Implement, apply, calculate, build | Implement attention, LoRA, RAG pipeline, PPO loop, DPO loss |
| Analyse | Analyse, compare, interpret, diagnose | Interpret attention weights, diagnose reward hacking, compare PPO vs DPO |
| Evaluate | Evaluate, assess, justify, critique | Assess alignment strategies, critique deployment proposals, score with LLM-as-judge |
| Create | Design, construct, assemble, produce | Design multi-agent system, build evaluated LLM application, implement full RLHF pipeline |
| Component | Weight | Details |
|---|---|---|
| Weekly practicals (submitted as scripts/notebooks) | 30% | 18 short submissions, one per week |
| Mid-term written assignment (Week 6) | 20% | 1500-word essay on scaling laws or fine-tuning |
| Final group project | 50% | Working LLM application + 10-page report + live demo |
No prior NLP experience required.
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
Create a .env file with your API keys:
OPENAI_API_KEY=<your-openai-key>
ANTHROPIC_API_KEY=<your-anthropic-key>
HF_TOKEN=<your-huggingface-token>
Run any practical:
python practicals/week01_practical.py
| Paper | Link |
|---|---|
| Vaswani et al. (2017) — Attention is All You Need | https://arxiv.org/abs/1706.03762 |
| Brown et al. (2020) — GPT-3 | https://arxiv.org/abs/2005.14165 |
| Kaplan et al. (2020) — Scaling Laws | https://arxiv.org/abs/2001.08361 |
| Hoffmann et al. (2022) — Chinchilla | https://arxiv.org/abs/2203.15556 |
| Wei et al. (2022) — Emergent Abilities | https://arxiv.org/abs/2206.07682 |
| Wei et al. (2022) — Chain-of-Thought | https://arxiv.org/abs/2201.11903 |
| Hu et al. (2022) — LoRA | https://arxiv.org/abs/2106.09685 |
| Christiano et al. (2017) — Deep RL from Human Preferences | https://arxiv.org/abs/1706.03741 |
| Ouyang et al. (2022) — InstructGPT / RLHF | https://arxiv.org/abs/2203.02155 |
| Schulman et al. (2017) — Proximal Policy Optimisation | https://arxiv.org/abs/1707.06347 |
| Bai et al. (2022) — Constitutional AI | https://arxiv.org/abs/2212.08073 |
| Rafailov et al. (2023) — DPO | https://arxiv.org/abs/2305.18290 |
| Lightman et al. (2023) — Let’s Verify Step by Step (PRMs) | https://arxiv.org/abs/2305.20050 |
| Skalse et al. (2022) — Defining and Characterizing Reward Hacking | https://arxiv.org/abs/2209.13085 |
| Lewis et al. (2020) — RAG | https://arxiv.org/abs/2005.11401 |
| Yao et al. (2022) — ReAct | https://arxiv.org/abs/2210.03629 |
| Dettmers et al. (2023) — QLoRA | https://arxiv.org/abs/2305.14314 |
| DeepSeek-R1 (2025) — Incentivising Reasoning via RL (GRPO) | https://arxiv.org/abs/2501.12948 |
| Azar et al. (2023) — IPO | https://arxiv.org/abs/2310.12036 |
| Ethayarajh et al. (2024) — KTO | https://arxiv.org/abs/2402.01306 |
| Meng et al. (2024) — SimPO | https://arxiv.org/abs/2405.14734 |
| Hong et al. (2024) — ORPO | https://arxiv.org/abs/2403.07691 |
| Rafailov et al. (2024) — Scaling Laws for DPO Overoptimisation | https://arxiv.org/abs/2406.02900 |
| Dao et al. (2022) — FlashAttention | https://arxiv.org/abs/2205.14135 |
| Dao (2023) — FlashAttention-2 | https://arxiv.org/abs/2307.08691 |
| Beltagy et al. (2020) — Longformer | https://arxiv.org/abs/2004.05150 |
| Zaheer et al. (2020) — BigBird | https://arxiv.org/abs/2007.14062 |
| Choromanski et al. (2021) — Performers | https://arxiv.org/abs/2009.14794 |
| Peng et al. (2023) — RWKV | https://arxiv.org/abs/2305.13048 |
| Gu et al. (2022) — S4 | https://arxiv.org/abs/2111.00396 |
| Gu and Dao (2023) — Mamba | https://arxiv.org/abs/2312.00752 |
| Ainslie et al. (2023) — GQA | https://arxiv.org/abs/2305.13245 |
| Team AI21 (2024) — Jamba | https://arxiv.org/abs/2403.19887 |
| Williams (1992) — REINFORCE | https://link.springer.com/article/10.1007/BF00992696 |
Stanford CS336 Language Modelling from Scratch and course webpage
deeplearning.ai course on transformers by Jay Alammar and Maarten Grootendorst
Ilya Sutskever and Andrej Karpathy video on how to code transformers from scratch
NanoGPT
Vizuara videos coding transformers from scratch
DAMTP and CAISH courses math department cambridge 2025
Bluedot impact courses
Stanford CS324: Understanding and Developing Large Language Models — this is probably the best single Stanford reference for your purpose because it explicitly combines modeling, theory, ethics, and systems, and it is designed to give hands-on experience with massive language models. (Stanford CRFM)
Stanford CS224N: Natural Language Processing with Deep Learning — a very strong backbone course. The current offering explicitly covers deep learning for NLP and LLMs, and its assessment structure is especially useful for inspiration: one assignment each on word vectors, neural-network foundations, self-attention and Transformers, and LLM benchmarking/evaluation. (Stanford University)
Stanford CS25: Transformers United V6 — this is less of a core methods course and more of a transformer seminar, but it is valuable if you want to see how Stanford frames cutting-edge transformer topics and guest talks from major researchers. It is especially useful for a reading-seminar or guest-lecture component. (Stanford University)
UC Berkeley CS 194/294-267: Understanding Large Language Models: Foundations and Safety — very useful if you want to balance internals with interpretability, scaling laws, robustness, alignment, privacy, watermarking, agency, reasoning, and evaluation. It is a strong model for a course that treats LLMs as both a technical and safety-relevant system. (rdi.berkeley.edu)
UC Berkeley INFO 290: Applied Generative AI and Large Language Models — this is a good reference for the applied side of the syllabus: transformer architectures, prompt engineering, API integration, RAG, open-source models, fine-tuning, graph enhancements, and agentic technologies. (UC Berkeley School of Information)
MIT OCW 6.7960 Deep Learning, Lecture 8: Architectures: Transformers — a clean, compact lecture resource for explaining the core internals: tokens, attention, and positional codes, with a nice framing that connects transformers to other architectures. (MIT OpenCourseWare)
MIT OCW 15.773, Lecture 10: Adapting LLMs with Parameter-Efficient Fine-Tuning — useful for the post-pretraining part of the course, especially instruction tuning and adapting base models. (MIT OpenCourseWare)
deeplearning.ai search for transformers.
A sensible design pattern would be to use CS224N for the course spine, CS324 for the “LLM systems + theory” lens, MIT’s transformer lecture for the internals exposition, and Berkeley’s courses for safety, interpretability, and application modules. (Stanford University)
Soumya Banerjee