Fine-Tuning Open Language Models with LoRA
Decide when fine-tuning pays, prepare chat data properly, and fine-tune open models with LoRA and QLoRA, from the maths and a LoRA layer of your own to transformers, peft, evaluation and serving.
Advanced · Generative AI · about 19 hours · 5 modules · certificate on completion
Taught by Mei-Líng Zhāng, Chloé Tremblàŷ, AI Learning Companions.
About this course
Fine-tuning is the most powerful way to change what an open language model does, and the easiest to waste money on. This course starts with the decision: when prompting or retrieval is enough, what fine-tuning reliably teaches (behaviour and format, more than facts), and how to set a target and a baseline before training. Then it gets the data right: JSON Lines formats, chat templates and their surprises, completion-only labels with the prompt masked, sequence lengths, de-duplication and group splits that keep validation honest. You'll compare full fine-tuning with partial and parameter-efficient methods on a tiny GPT you pretrain yourself, measuring the memory each needs and the catastrophic forgetting each causes.
The core is LoRA: its low-rank update, parameter count, initialisation, merging and scaling, rank and alpha, target modules, rsLoRA and DoRA. You'll implement a LoRA layer in PyTorch, then fine-tune a real open model with Hugging Face transformers and peft on a CPU: configure and inspect the adapter, train it, save it, reload it and merge it, with QLoRA's 4-bit recipe for when a GPU is available. The last module evaluates fine-tuned models against fair baselines, catches overfitting with early stopping, and serves several adapters from one base model. Every lab prints exact, checkable output, and the capstone takes a task of your choice from data to a documented, served adapter. Efficient Inference: Quantisation, Distillation and Batching and Operationalising LLMs on Kubernetes are the natural next steps.
What you'll learn
- Decide when fine-tuning beats prompting or retrieval, and set a metric, a target and a baseline first
- Prepare clean, de-duplicated, leak-free chat data with the model's template and completion-only labels
- Explain LoRA's update, parameter count and scaling, and implement, train and merge a LoRA layer in PyTorch
- Fine-tune open models with transformers and peft, and configure QLoRA's 4-bit loading
- Evaluate fine-tuned models against baselines, catch overfitting and forgetting, and serve adapters merged or side by side
Skills you'll gain
Fine-tuning language models, LoRA, QLoRA, Parameter-efficient fine-tuning, Training data preparation, Chat templates, Hugging Face Transformers, PEFT, Model evaluation, Serving adapters
Tools
Python 3, PyTorch, Hugging Face Transformers, PEFT, Hugging Face Hub, bitsandbytes, vLLM
Before you start
Transformers and Large Language Models and Deep Learning with PyTorch, or the same skills: PyTorch training loops and optimisers, and the basics of Hugging Face transformers; Engineering with Foundation Models helps with evaluation. Install Python 3.10 or later with PyTorch 2, transformers and peft (tested with PyTorch 2.14, transformers 5.20 and peft 0.21). Every lab runs on a laptop CPU and downloads two tiny checkpoints, about 50 MB; only the optional QLoRA and vLLM steps need a GPU.
Modules
Decide, then prepare the data
Decide whether fine-tuning is the right tool, then turn raw examples into clean, de-duplicated, leak-free training data in the model's own chat format, with the prompt masked out of the loss.
- When fine-tuning is the right tool (12 min)
- Data formats and chat templates (12 min)
- Labels, masking and sequence length (12 min)
- Data quality, leakage and how much you need (10 min)
- Hands-on lab: Prepare a chat dataset for supervised fine-tuning
Full fine-tuning and parameter-efficient methods
Work out what full fine-tuning costs, compare it with partial and parameter-efficient methods, and measure learning and forgetting on a tiny GPT you pretrain yourself.
- What full fine-tuning costs (12 min)
- The parameter-efficient family (12 min)
- Forgetting, and choosing a method (10 min)
- Hands-on lab: Full fine-tuning against freezing: cost, learning and forgetting
LoRA from the inside
Derive LoRA's update, parameter count and scaling, choose rank, alpha and target modules for good reasons, and implement a LoRA layer in PyTorch that trains, merges and matches.
- The low-rank update (12 min)
- Rank, alpha and the learning rate (12 min)
- Target modules and LoRA's variants (12 min)
- Hands-on lab: Implement a LoRA layer and fine-tune a tiny model
LoRA and QLoRA with Hugging Face
Fine-tune a real open model with transformers and peft: configure and inspect the adapter, train with sound settings, save, reload and merge it, and see how QLoRA fits a 4-bit base on one GPU.
- The peft workflow (12 min)
- Training settings that matter (12 min)
- QLoRA: fine-tuning on a 4-bit base (12 min)
- Hands-on lab: Fine-tune a small open model with transformers and peft
Evaluate, avoid overfitting and serve adapters
Judge a fine-tuned model against the right baselines, catch overfitting and keep the best checkpoint, then serve adapters merged or side by side from one base model.
- Evaluating a fine-tuned model (12 min)
- Spotting and preventing overfitting (10 min)
- Serving and managing adapters (12 min)
- Hands-on lab: Catch overfitting, keep the best checkpoint and serve two adapters
Project: Fine-tune, evaluate and ship a task adapter
Pick a narrow task where a small open model should beat its prompted self, such as turning support emails into JSON fields, tagging product reviews or rewriting notes in a house style, and take it from data to a served adapter. Build a dataset of at least 300 examples from data you are allowed to use (your own, a public dataset whose licence permits it, or carefully generated examples, labelled as such), clean, de-duplicate and split it by group as in module 1, and freeze a test set. Measure a baseline first: an instruct model with your best prompt. Then fine-tune with LoRA using transformers and peft. On a laptop CPU, a model of up to a few hundred million parameters is practical (for example a small Pythia or SmolLM2 checkpoint); with a GPU, try QLoRA on a model of a few billion. Compare at least two ranks or learning rates using validation curves, keep the best checkpoint, check for overfitting and for regressions on a small general suite, and finish with a merged-versus-adapter parity check, a serving recommendation and a model card.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.