teaching_llm_applications
GPUs and Flash Attention
-
what will you learn by the end of this?
What does a GPU look like?
Resources
🤔 Question for class
- Do you think deep learning/LLMs can beat humans at everything?
- 📝 Rich Sutton’s Bitter Lesson
- Can just stacking more layers help?
🤔 Question for class
-
🤔 why would increasing the size of a matrix make matmul faster?
Animation of how GPU/TPU works
- from the amazing book on how to scale your model TPU book
- What is a TPU? See link
- 💡 TPUs are very, very fast at matrix multiplication.
Introduction
- Denard scaling, clock scaling
- cannot make clocks go faster
- have more things execute in parallel
- Concept 🧩 🚀 CPUs optimize for a few fast threads while GPUs optimize for many many many threads
- Each SM has many SP (streaming processors)
-
Each SM has can execute in parallel
-
GA100 has 128 SMs
-
Concept 🧩 🚀 The closer the memory to SM, the faster it is:
L1and shared memory is inside the SM.L2cache is on die, global memory are on memory chips next to GPU -
L1 and L2 cache is shared memory (SRAM): more expensive and more power hungry
- SRAM is 8x faster than HBM
Software model
- SIMT (Single Instruction, Multiple Threads): threads work in parallel but same instruction, but different inputs
- Blocks of 32 threads are called warps
- Threads execute in groups
- decreases overhead on scheduling
TPU
- TPU (Tensor Processing Unit) by Google
- optimized for matrix multiplication
Back to GPUs
-
Matmulis faster than floating point operations (additions, multiplications) -
compute is scaling faster than memory
-
memory bandwidth is the bottleneck
Summary
Recent trends
-
Prefill phase is memory bound and is one chip
- Decoding is compute bound and is another chip
- attention can go to one chip; MLP will go to another chip
TODO 📚: Questions Assignment
- TODO 📚: question written assignment on this (theory)
TODO Practical
-
TODO Practical see Stanford CS365 practical on GPUs.
-
Practical on counting FLOPS using
PyTorchand here usingTorchDispatchMode