teaching_llm_applications

CS 336 Stanford Language modelling from scratch

• 🎥 course

Week 1

• memory vs. compute

• DGX B200 8 GPUs connected

• Concept 🧩 🚀 GPU architecture

• Kernel is a function that runs on GPU. When using PyTorch, each primitive operation launches a kernel

• ⚠️ moving data from memory is expensive

• HBM: high bandwidth memory

• naive, fused

• operator fusion, tiling (FlashAttention)

• sharding (parameters, activations, gradients) across GPUs

• prefill phase

• The Prefill Phase in Large Language Models (LLMs)

When a Large Language Model (LLM) generates a response, the process (called inference) doesn’t happen all at once. It is split into two distinct steps: the Prefill Phase and the Decode Phase.

The Prefill Phase is the very first step where the model reads and processes your entire input prompt.

What Happens During the Prefill Phase?

1. Parallel Processing of the Prompt

Unlike generating text (which happens one word at a time), the model reads your entire prompt all at once. Because the input tokens are already known, the GPU processes them in parallel. This step is highly efficient and relies heavily on the sheer computational power of the GPU (it is “compute-bound”).

2. Building the KV Cache

As the model processes your prompt, it calculates and stores mathematical representations of every word to understand their context and relationships. It saves these calculations in what is known as the Key-Value (KV) Cache. This is a crucial step because it acts as the model’s short-term memory for your prompt.

3. Generating the First Token

The culmination of the prefill phase is the model predicting and outputting the very first token (word or sub-word) of its response.

---

Contrast: The Decode Phase

Once the first token is generated, the prefill phase ends and the Decode Phase begins.

• Sequential Generation: The model generates the rest of its response strictly one token at a time.
• Using the Cache: To do this efficiently, it doesn’t re-read your original prompt. Instead, it looks back at the KV Cache it built during the prefill phase.
• Speed: Because it has to generate words one by one and constantly fetch data from the cache, the decode phase is much slower per token and is limited by how fast the GPU can move data around (it is “memory-bandwidth bound”).

---

A Simple Analogy

Imagine you are handed a piece of paper with a complex math word problem on it.

• The Prefill Phase: You read the entire paragraph at once, understand the context, set up the equation in your head (building the cache), and write down the very first number of your answer.
• The Decode Phase: You write down the rest of the numbers one by one to finish the equation, relying on your memory of the problem rather than re-reading the paragraph for every single digit.

Remedies

• quantize, distill

Week 1 assignment/practical

• week 1

• FLOP budget -> hyperparameters

• Scaling laws Flops budget = 6 * N * D: use a bigger model (N) or more tokens (D)? Hoffman2022 (Chinchilla caling laws)

• 🤔 D = 20 * N scaling law: 70B parameter model should be trained on 1.4T tokens

• 🎮 scaling laws practical VERY GOOD using nanoGPT by Andrew Karpathy

• this can be used to calculate a token budget

• 💡 also used to ask for more money to investors!

• 🤔❓ data: what capabilities you want your model to have? Conversation? Agentic coding?

• diverse set of evaluations since LLMs are general purpose.

• Perplexity within token budget

• licesing issues: lot of github code does not have a license. So assume permissive?

• efficiency: hardware, data, compute, memory, communication bandwidth. How do you build the best model?

• Faster inference

Divine Benevolence

This site is open source. Improve this page.