Operationalising LLMs on Kubernetes

Deploy, scale, observe and upgrade open large language models on Kubernetes: inference servers, model storage, GPU scheduling, queue-based autoscaling and safe rollouts.

Advanced · AI Engineering · about 24 hours · 5 modules · certificate on completion

Taught by Abderrahmàn Bensoussàn, Carlòs Ramirèz, AI Learning Companions.

About this course

Running a large language model in production is mostly an operations problem: weights of many gigabytes to deliver, scarce and expensive GPUs to schedule, load that comes in bursts, and users who notice every extra second before the first token. This advanced course teaches you to run open models on Kubernetes the way platform teams do. You'll compare inference servers (vLLM, TGI and Ollama), containerise them, and pre-load weights onto persistent volumes so start-ups are fast and predictable; request GPUs through the NVIDIA device plugin with node selectors, taints and tolerations, and share them with time-slicing, MPS or MIG.

Then you'll autoscale on what matters for LLMs, the queue of waiting requests, with KEDA and custom metrics, and see where a cluster autoscaler and cost planning come in; build dashboards and unit-tested alerts for tokens per second, time to first token and KV cache use; trace requests and aggregate logs; and keep models serving through drains, node failures and version upgrades with topology spread, disruption budgets, rolling updates and canaries. Every lab runs on a laptop: a real small model served by Ollama, and llm-sim, a lightweight stand-in that speaks vLLM's API and metrics, on a kind cluster with a simulated GPU pool. Where you have a GPU cluster, each lab also has steps for vLLM on real GPUs. The capstone puts a fine-tuned open model into production on Kubernetes, with autoscaling on queue length.

What you'll learn

  • Deploy open LLMs on Kubernetes with vLLM or Ollama, with model weights pre-loaded onto persistent volumes
  • Configure GPU requests, node selection, taints and GPU sharing for inference workloads
  • Autoscale LLM servers on queue length with KEDA and custom metrics, and plan GPU node capacity and cost
  • Monitor tokens per second, time to first token and saturation with Prometheus, Grafana and tested alerts
  • Use affinity, topology spread and disruption budgets to survive drains and failures, and upgrade models safely

Skills you'll gain

LLM serving, vLLM, GPU scheduling, Kubernetes autoscaling, KEDA, LLM observability, Model storage, Canary releases, Capacity and cost planning

Tools

Kubernetes, kind, Helm, vLLM, Ollama, KEDA, Prometheus, Grafana

Before you start

Kubernetes experience as taught in Containerisation and Orchestration with Docker and Kubernetes (Deployments, Services, probes, kubectl); Generative AI Design Patterns helps. You need Docker with 8 GB of memory or more for it (16 GB in the computer), kind, kubectl, Helm and Python 3.10 or later. A GPU cluster is optional: every lab has a CPU path.

Modules

  1. Inference Servers and Model Data

    What an LLM inference server does and how vLLM, TGI and Ollama compare, how to containerise one, and how to store, deliver and load model weights in Kubernetes without slow or fragile start-ups.

    • What an inference server does: vLLM, TGI and Ollama (13 min)
    • Containerising an LLM server (12 min)
    • Model weights: PVCs, object storage and images (13 min)
    • Loading models faster (11 min)
    • Hands-on lab: Serve a small open model on kind

  2. GPU Management in Kubernetes

    How GPUs become schedulable resources through drivers, the container toolkit and the NVIDIA device plugin, how Pods request them, how one GPU can be shared, and how to watch GPU use.

    • How Kubernetes sees GPUs: drivers, the toolkit and the device plugin (13 min)
    • Requesting GPUs: limits, labels and taints (12 min)
    • Sharing GPUs: time-slicing, MPS and MIG (13 min)
    • Monitoring GPU utilisation (11 min)
    • Hands-on lab: Schedule onto a simulated GPU pool

  3. Production Autoscaling

    Scale LLM servers on signals that reflect their load, such as queue length, with the Horizontal Pod Autoscaler, custom metrics and KEDA, and add GPU nodes with a cluster autoscaler while keeping cost in check.

    • What to scale LLM servers on (12 min)
    • The Horizontal Pod Autoscaler with custom metrics (13 min)
    • Event-driven autoscaling with KEDA (13 min)
    • Cluster autoscaling and cost (13 min)
    • Hands-on lab: Autoscale on queue length with KEDA

  4. Observability and Monitoring

    Collect and use the signals that matter for LLM serving: Prometheus and Grafana for tokens per second, latency and saturation, alerting rules you can test, distributed tracing, and aggregated logs.

    • Prometheus and Grafana for model servers (12 min)
    • LLM metrics: tokens per second, latency and saturation (13 min)
    • Distributed tracing for LLM requests (12 min)
    • Log aggregation (11 min)
    • Hands-on lab: Dashboards, alerts and logs for LLM serving

  5. Advanced Scheduling and Operations

    Place LLM servers precisely with affinity, taints and topology spread, keep them serving through drains and node failures, and roll out new model versions safely with rolling updates and canaries.

    • Node affinity, taints and tolerations for GPU pools (12 min)
    • Topology spread constraints (11 min)
    • Handling node failures (12 min)
    • Upgrading model versions safely (13 min)
    • Hands-on lab: Spread, drain, fail and upgrade

Project: Put a fine-tuned open model into production on Kubernetes

Your team has a fine-tuned open model (your own, for example from Fine-Tuning Open Language Models with LoRA, or a public fine-tuned instruct model) to serve on Kubernetes. On a GPU cluster, serve it with vLLM; on a laptop, serve a small model with Ollama for real answers and use llm-sim wherever you need many replicas. Pre-load the weights, with the revision pinned, onto a PersistentVolumeClaim with a Job; run the server with GPU requests (real or simulated), node selection, tolerations, probes and /dev/shm; autoscale it on inference queue length with KEDA; give it a Grafana dashboard and promtool-tested alerts; make it resilient with topology spread, a PodDisruptionBudget and failover tolerations; plan a canary upgrade; and estimate its cost from your own load tests and your provider's GPU price. Prove each part with outputs.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free