Efficient Inference: Quantisation, Distillation and Batching

Make language models faster, smaller and cheaper to serve: find where time and memory go, batch continuously, quantise weights and activations with the maths behind it, and distil, prune and speculate.

Advanced · AI Engineering · about 18 hours · 5 modules · certificate on completion

Taught by Wei-Chéng Liú, Mouhamadouù Bâ, AI Learning Companions.

About this course

A language model that answers well is only useful if you can afford to serve it. This course is about making inference fast, small and cheap, and about the measurements that tell you whether it worked. It starts with where the time and memory go: prefill against decode, why decode is limited by memory bandwidth rather than arithmetic, and the KV cache that grows with every token. You'll add a KV cache to a small GPT and time it, then simulate static, dynamic and continuous batching and paged KV-cache allocation on the same workload to see where the throughput comes from.

The middle of the course is quantisation with its maths: number formats from bf16 to FP8, int8 and 4-bit, scales and zero points, granularity and outliers, GPTQ, AWQ and GGUF, and W8A8 with SmoothQuant and mixed precision. You'll quantise a real open model and measure its size, perplexity and next-token agreement, then tame activation outliers in a W8A8 layer. The last module makes the model itself cheaper with distillation, pruning and speculative decoding, and compares vLLM, llama.cpp and ONNX Runtime. Every lab runs on a laptop CPU with exact expected output, and the capstone takes a model of your choice from baseline to a measured, recommended configuration. Operationalising LLMs on Kubernetes and Fine-Tuning Open Language Models with LoRA pair well with it.

What you'll learn

  • Explain where inference time and memory go, from prefill and decode to the KV cache, and estimate both from a model's configuration
  • Compare static, dynamic and continuous batching, and size a paged KV cache for a workload
  • Quantise weights to int8 and 4-bit with scales and zero points, and measure size, error and perplexity
  • Handle activation outliers in W8A8 quantisation with SmoothQuant, mixed precision or FP8
  • Apply distillation, pruning and speculative decoding, and choose a serving engine for the hardware and workload

Skills you'll gain

LLM inference optimisation, KV-cache management, Continuous batching, Quantisation, Knowledge distillation, Pruning and sparsity, Speculative decoding, Performance measurement, PyTorch

Tools

Python 3, PyTorch, Hugging Face Transformers, vLLM, llama.cpp, ONNX Runtime

Before you start

Transformers and Large Language Models and Deep Learning with PyTorch, or the same skills. Engineering with Foundation Models introduces the KV cache and quantisation that this course takes much further. Install Python 3.10 or later with PyTorch 2 and transformers (tested with PyTorch 2.14 and transformers 5.20). Every lab runs on a laptop CPU, and two download small Hugging Face checkpoints, about 50 MB in all.

Modules

  1. Where inference time and memory go

    Split generation into prefill and decode, see why decode is limited by memory bandwidth rather than arithmetic, and budget the KV cache that grows with every token.

    • Prefill and decode (12 min)
    • Arithmetic intensity and the roofline (12 min)
    • The KV cache as a memory budget (12 min)
    • Hands-on lab: Measure prefill, decode and the KV cache

  2. Batching and scheduling

    Compare static, dynamic and continuous batching on a simulated workload, and see how paged KV-cache allocation and preemption let a server run more sequences at once.

    • Static and dynamic batching, and why they waste (10 min)
    • Continuous batching (12 min)
    • Paged KV cache, admission and preemption (12 min)
    • Hands-on lab: Simulate static, dynamic and continuous batching

  3. Quantisation: the numbers

    Learn the number formats inference uses, the maths of scales and zero points, and why granularity and outliers decide how much accuracy low-bit weights lose.

    • Number formats for inference (12 min)
    • Scales and zero points (12 min)
    • Granularity, outliers and 4-bit groups (12 min)
    • Hands-on lab: Quantise a small model's weights: size, error and perplexity

  4. Quantisation in practice: weights, activations and the KV cache

    Go beyond rounding: error-compensating weight quantisation, activation outliers and SmoothQuant, FP8, quantised KV caches and the formats engines load, judged by an evaluation you can trust.

    • Weight-only quantisation: GPTQ, AWQ and GGUF (12 min)
    • Quantising activations: outliers, SmoothQuant and FP8 (12 min)
    • Quantised KV caches, and judging quality (12 min)
    • Hands-on lab: Activation outliers, SmoothQuant and FP8 in a W8A8 layer

  5. Distillation, pruning, speculative decoding and serving engines

    Make the model itself smaller or faster: distil a student, prune into patterns hardware can use, draft tokens with speculative decoding, and choose an engine that brings it all together.

    • Knowledge distillation (12 min)
    • Pruning and sparsity (12 min)
    • Speculative decoding (12 min)
    • Serving engines, and choosing a stack (12 min)
    • Hands-on lab: Distil, prune and speculate on a CPU

Project: Make a model cheaper to serve, and prove it's still good

Take a small open model you can run, such as pythia-70m or pythia-160m on a laptop CPU or a model of one to three billion parameters on a GPU, and a realistic workload of your choice: prompt and output lengths and a target number of concurrent users. Measure the baseline first: memory, time to first token, time per output token and throughput at one and at several concurrent requests, and quality as perplexity on held-out text plus a task metric. Then apply and measure at least three optimisations from the course: weight quantisation at two granularities (your own code from module 3 or an engine's format such as GGUF); a KV-cache budget and batching plan for your target concurrency, using the module 2 simulator or an engine's metrics; and speculative decoding with a smaller model of the same family (the Pythia models share a tokeniser, so pythia-14m can draft for pythia-160m through generate's assistant_model). Distillation or pruning are optional extras. Finish with the configuration you'd deploy, on which engine and hardware, with its measured costs and quality.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free