Engineering with Foundation Models

Choose, understand, evaluate and steer foundation models like an engineer: scale and serving arithmetic, transformer internals, rigorous model and system evaluation, LLM judges in CI, and prompt optimisation.

Advanced · Generative AI · about 20 hours · 5 modules · certificate on completion

Taught by Dariusz Wojciechowskì, Chloé Tremblàŷ, AI Learning Companions.

About this course

Foundation models are general-purpose engines: trained once on broad data, then adapted to thousands of tasks. Engineering with them means making decisions that hold up under scrutiny: which model to use and where to run it, what its context window and memory footprint really cost, how to tell whether it's good enough, and how to keep it good as prompts, models and data change. This course works through those decisions. You'll start with what scale does and doesn't buy, and how to shortlist models against requirements; go inside the transformer to see what attention, tokenisers, the KV cache and quantisation mean for cost and quality; then spend two modules on evaluation, from benchmarks, BLEU and ROUGE and properly run human ratings to AI-as-a-judge pipelines that are validated against people and run as quality gates in CI/CD. The last module treats prompt engineering as experimentation, and sets it beside retrieval and fine-tuning.

Every module has a hands-on Python lab that runs on an ordinary laptop without an API key: scaling arithmetic and a model shortlist, attention, byte-pair encoding and quantisation in NumPy, metrics against human ratings, judge validation with a pytest quality gate, and prompt selection with paired statistical tests. The capstone is an automated evaluation pipeline for a customer support chatbot, with your own rubric, test set and judge.

What you'll learn

  • Explain what makes a foundation model, how scale drives its abilities, and estimate training, inference and memory costs
  • Explain self-attention, tokenisation, context windows and the KV cache, and what quantisation trades away
  • Compare and select foundation models against an application's requirements
  • Evaluate models with benchmarks, reference metrics and human ratings, with agreement and confidence intervals
  • Build and validate AI-as-a-judge pipelines and run them as quality gates in CI/CD
  • Optimise prompts with held-out evaluation, and choose between prompting, retrieval and fine-tuning

Skills you'll gain

Foundation model selection, Scaling laws, Transformer internals, Quantisation, LLM evaluation, LLM-as-a-judge, Evaluation in CI/CD, Prompt optimisation, Statistical testing

Tools

Python 3, NumPy, scikit-learn, SciPy, statsmodels, NLTK, pytest, Anthropic Python SDK

Before you start

Applied Machine Learning with TensorFlow, or experience training and evaluating models in Python; Transformers and Large Language Models helps with module 2, and Prompting and LLM Application Design with modules 4 and 5. Install Python 3.10+ and pip install numpy scipy scikit-learn statsmodels nltk pydantic pytest anthropic. Labs run without an API key; the capstone needs a Claude Console API key with credit and CLAUDE_MODEL set to a current model ID.

Modules

  1. Introduction to Foundation Models

    What foundation models are, how they're trained and adapted, what scaling laws and emergent abilities tell an engineer, and how to shortlist models for an application.

    • What makes a model a foundation model (12 min)
    • Scale: parameters, data and compute (14 min)
    • Emergent abilities, and the debate about them (10 min)
    • Choosing a foundation model (14 min)
    • Hands-on lab: Scaling arithmetic and a model shortlist

  2. Inside Foundation Models

    The parts of the transformer that decide what models cost and how they behave: attention, tokenisation, pretraining and context windows, the KV cache, and the basics of serving and quantisation.

    • Self-attention for engineers (14 min)
    • Tokenisation and what it costs (12 min)
    • Pretraining, context windows and the KV cache (12 min)
    • Serving and quantisation basics (14 min)
    • Hands-on lab: Attention, tokens and quantisation by hand

  3. Model Evaluation

    Why evaluating generation is hard, what benchmarks can and can't tell you, where BLEU and ROUGE mislead, and how to run human evaluation with agreement measures and confidence intervals.

    • Why evaluating generation is hard (12 min)
    • Benchmarks: what MMLU and others tell you (12 min)
    • Reference-based metrics: BLEU, ROUGE and their limits (12 min)
    • Human evaluation done properly (12 min)
    • Hands-on lab: Metrics against human ratings

  4. System Evaluation

    Evaluate the whole application, not just the model: realistic test sets, component checks, AI-as-a-judge pipelines validated against human labels and tested for bias, and evaluation as an automated gate in CI/CD.

    • From model evaluation to system evaluation (12 min)
    • AI-as-a-judge: design (12 min)
    • Validating and de-biasing the judge (12 min)
    • Automating evaluation in CI/CD (12 min)
    • Hands-on lab: Validate an LLM judge and gate a release

  5. Advanced Prompt Engineering

    Prompting as engineering: reasoning on demand, examples and system prompts at scale, designs that reduce hallucination, prompt optimisation that survives held-out testing, and when to retrieve or fine-tune instead.

    • Reasoning on demand: chain-of-thought and thinking modes (12 min)
    • Examples and system prompts at scale (12 min)
    • Reducing hallucinations by design (12 min)
    • Prompt optimisation, and when to fine-tune instead (14 min)
    • Hands-on lab: Choose a prompt version without fooling yourself

Project: Automated evaluation pipeline for a customer support chatbot

Bluebell Broadband, a fictional provider, has a retrieval-augmented support chatbot (chatbot.py) built on eight knowledge-base articles. Before it goes live, build the pipeline that decides whether it's good enough and catches regressions afterwards. Define a rubric for accuracy, tone and helpfulness with pass and fail examples for each; grow the eight seed queries into a test set of at least 30 that covers every article, edge cases, questions the articles don't answer, and awkward or upset customers; and write an AI-as-a-judge that uses a strong Claude model with structured output, one criterion per call, given the same articles as the chatbot. Validate the judge against your own labels for at least 20 answers, with agreement and Cohen's kappa, and test it for position or verbosity bias where it applies. Then run the chatbot over the test set, produce a report with pass rates, bootstrap confidence intervals, results by category and the failures that matter, make one improvement to the chatbot, and show its effect with a paired comparison. Finally, wrap the pipeline in a pytest quality gate that could run in CI. eval_pipeline.py --demo shows the report format on recorded results.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free