10 free resources to learn LLM inference engineering
Covering topics such as:
- KV cache and batching
- Serving engines
- GPU kernels and benchmarks
Start here and go in order:
1) LLM Inference Handbook (Modular)
↳ Learn the words first: prefill, decode, KV cache, latency, throughput.
↳ A reference you keep open while you read the rest.
https://handbook.modular.com/
2) Continuous batching from first principles (Hugging Face)
↳ Starts at attention and KV caching, ends at continuous batching.
↳ Shows why mixing prefill and decode in one batch helps throughput.
3) Stanford CS336, Lecture 10: Inference (Spring 2025)
↳ Percy Liang explains why inference is memory limited.
↳ Then the fixes: quantization, speculative decoding and paged attention.
4) All About Transformer Inference (How To Scale Your Model, Google DeepMind)
↳ The math behind latency, throughput, memory and KV caches.
↳ Harder than the rest. Worth the extra effort.
5) MIT 6.5940 EfficientML (Fall 2024, Song Han)
↳ Watch the quantization lectures and Lecture 13 on LLM deployment.
↳ Slides and videos sit on the course page.
6) Inside vLLM (Aleksa Gordić)
↳ Scheduling, paged attention, prefix caching, speculative decoding, multi-GPU serving.
↳ Based on the vLLM V1 engine at one August 2025 commit, so some code has moved since.
7) Mini-SGLang (SGLang project)
↳ A compact serving engine with radix cache, chunked prefill and overlap scheduling.
↳ Reading the code is free. Running it needs Linux and an NVIDIA GPU with CUDA.
sgl-project / mini-sglang
A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.
Mini-SGLang
A lightweight yet high-performance inference framework for Large Language Models.
Mini-SGLang is a compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems. With a compact codebase of ~5,000 lines of Python, it serves as both a capable inference engine and a transparent reference for researchers and developers.
✨ Key Features
- High Performance: Achieves state-of-the-art throughput and latency with advanced optimizations.
- Lightweight & Readable: A clean, modular, and fully type-annotated codebase that is easy to understand and modify.
- Advanced Optimizations
- Radix Cache: Reuses KV cache for shared prefixes across requests.
- Chunked Prefill: Reduces peak memory usage for long-context serving.
- Overlap Scheduling: Hides CPU scheduling overhead with GPU computation.
- Tensor Parallelism: Scales inference across multiple GPUs.
- Optimized Kernels: Integrates FlashAttention and FlashInfer for maximum efficiency.
- ...
🚀 Quick Start
⚠️ Platform Support: Mini-SGLang currently supports…
8) GPU MODE Lectures
↳ Start with Lectures 1, 3 and 4: profiling, getting started with CUDA, compute and memory architecture.
↳ The repo has the code and slides. The recordings are on their YouTube channel.
Supplementary Material for Lectures
The PMPP Book: Programming Massively Parallel Processors: A Hands-on Approach (Amazon link)
Lecture 1: Profiling and Integrating CUDA kernels in PyTorch
- Speaker: Mark Saroufim
- Notebook and slides in lecture_001 folder
Lecture 2: Recap Ch. 1-3 from the PMPP book
- Speaker: Andreas Koepf
- Slides: The powerpoint file lecture_002/cuda_mode_lecture2.pptx can be found in the root directory of this repository. Alternatively here as Google docs presentation.
Lecture 3: Getting Started With CUDA
- Speaker: Jeremy Howard
- Notebook: See the lecture_003 folder, or run the Colab version
Lecture 4: Intro to Compute and Memory Architecture
- Speaker: Thomas Viehmann
- Notebook and slides in the lecture_004 folder.
Lecture 5: Going Further with CUDA for Python Programmers
- Speaker: Jeremy Howard
- Notebook in the lecture_005 folder.
Lecture 6: Optimizing PyTorch Optimizers
Lecture 7: Advanced Quantization
- Speaker: Charles Hernandez
- Slides
Lecture 8: CUDA Performance Checklist
- Speaker: Mark Saroufim
- Code in…
9) Triton Tutorials (official docs)
↳ Go from vector addition to fused softmax, matrix multiplication and fused attention.
↳ Each tutorial is runnable code you can download.
10) AIPerf (NVIDIA)
↳ Measure time to first token, inter token latency, throughput and goodput.
↳ Its guides cover trace replay, so you can test with real traffic patterns.
ai-dynamo / aiperf
AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution.
AIPerf
AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution. It provides detailed metrics using a command line display as well as extensive benchmark performance reports.
Quick Start
This quick start guide leverages Ollama via Docker Desktop.
Setting up a Local Server
In order to set up an Ollama server, run granite4:350m using the following commands:
docker run -d \ --name ollama \ -p 11434:11434 \ -v ollama-data:/root/.ollama \ ollama/ollama:latest docker exec -it ollama ollama pull granite4:350m Basic Usage
Create a virtual environment and install AIPerf:
python3 -m venv venv source venv/bin/activate pip install aiperf Note
The optional crick dependency has no prebuilt wheel for some platforms including Linux aarch64 (arm64), and falls back to an sdist that needs a C compiler at install time. crick backs the non-default tdigest list-metric backend, so a plain pip…
Basic Python, PyTorch and Transformer knowledge will help.
All 10 are free to read or watch. Running the code examples may need a paid GPU.
