teaching_llm_applications

Paged attention and vLLM (virtual LLM)

• Video by Vizuara on vLLM

• Video by Vizuara on one GPU serving multiple requests

• vLLM (virtual LLM)

• Vizuara video on how KV cache gets fragmented

• 🤔 Solution: paging (from the 1960s)

• an engine, not a model

• same model served to thousands of people on same GPU

• KV cache

• every user gets their own KV cache

• on a busy GPU, KV cache gets bigger

• you can reserve a certain amount of KV cache for each user (for the longest possible answer), but you potentially waste space

• 💡 OS trick - operating systems faced and solved a similar problem (virtual memory)

• Concept 🧩 🚀 Paged Attention

Break up the KV cache into small pieces

• Virtual memory for the KV cache

• PagedAttention blocks

• Video by Vizuara on one GPU serving multiple requests

This site is open source. Improve this page.