Prompting and LLM Application Design
Design LLM features that hold up in production: structured outputs, reasoning, retrieval, tools, guardrails and evaluation, built in Python on the Claude API.
Intermediate · Generative AI · about 20 hours · 5 modules · certificate on completion
Taught by Vladimìr Novàk, Fernandà Oliveirà, AI Learning Companions.
About this course
Calling a large language model is easy; building a feature on one that behaves well on the thousandth request is not. This course takes you from prompts typed into a chat window to LLM features designed in code. You'll review how chat models generate text and why instruction tuning and RLHF make them behave as they do, then assemble prompts from templates, examples and data, and get structured output that validates. You'll improve multi-step reasoning with chain-of-thought and self-consistency, with Vietnamese as well as English examples, ground answers in documents with retrieval and checked citations, and give the model tools through a ReAct-style loop. The last module turns to production: workflows, guardrails, cost and latency budgets, offline evaluation with code checks and LLM judges, and online A/B tests.
Every module has a Python lab built around the Anthropic Python SDK. The labs run without an API key, on replies recorded for them, so their results are exact and checkable; optional live steps send the same requests to the Claude API. The course ends with a capstone: a bilingual English and Vietnamese policy assistant that uses retrieval, a search tool and an LLM judge, evaluated on a test set.
What you'll learn
- Explain how chat models generate text and were tuned to follow instructions, and control length, stopping and sampling in API calls
- Assemble prompts from templates, examples and data, and get structured output that validates, with repair when it doesn't
- Improve multi-step answers with chain-of-thought and self-consistency, in English and Vietnamese
- Build grounded answers with retrieval and checked citations, and a tool-using assistant with a ReAct-style loop
- Design workflows with guardrails and cost and latency budgets
- Evaluate LLM features offline with code checks and an LLM judge, and read online A/B tests correctly
Skills you'll gain
LLM application design, Prompt design, Structured outputs, Chain-of-thought prompting, Retrieval-augmented generation, Tool use, LLM evaluation, A/B testing, Guardrails
Tools
Python 3, Anthropic Python SDK, Claude API, Pydantic, scikit-learn, statsmodels, NumPy
Before you start
Transformers and Large Language Models (or a working knowledge of how LLMs predict tokens), Prompt Engineering: From Basics to Advanced Techniques, and confident Python, as in Python Programming with PyCharm. Install Python 3.10+ and pip install anthropic pydantic scikit-learn numpy statsmodels. Labs run without an API key; optional live steps and the capstone need a Claude Console API key with credit (billed per token) and CLAUDE_MODEL set to a current model.
Modules
LLM foundations and Chat Models
Review how a chat model produces text one token at a time and how instruction tuning and RLHF shape its behaviour, then use the Messages API's system prompt, roles, length limits, stop reasons and sampling settings with care.
- From next-token prediction to a chat reply (12 min)
- How instruction tuning and RLHF shape chat models (12 min)
- System prompts, roles and the Messages API (12 min)
- Sampling settings: temperature, top-p and what to use instead (12 min)
- Hands-on lab: Sampling, stop reasons and token budgets
Assembling prompts and content
Build prompts in code from templates, examples and data; choose and order few-shot examples; get structured output that validates; and write role and policy instructions that make an assistant behave consistently.
- Assembling prompts in code (10 min)
- Few-shot examples that generalise (12 min)
- Structured outputs: schemas, validation and repair (14 min)
- Roles and policies for a product assistant (12 min)
- Hands-on lab: Extract support tickets that validate
Reasoning and Chain-of-Thought
Get more reliable answers to multi-step problems: chain-of-thought with and without examples, built-in reasoning modes, self-consistency voting, and what changes when the problems and the reasoning are in Vietnamese.
- Chain-of-thought prompting (12 min)
- Zero-shot chain-of-thought and built-in reasoning (10 min)
- Self-consistency: sample, then vote (12 min)
- Reasoning in Vietnamese and other languages (10 min)
- Hands-on lab: Self-consistency on English and Vietnamese problems
RAG and Tool Use (ReAct)
Ground answers in your own documents with retrieval and citations, give the model tools described with JSON schemas, run the reason-act-observe loop, and catch answers that still go beyond their sources.
- Retrieval-augmented generation, the basics (12 min)
- Tools and JSON schemas (12 min)
- The ReAct pattern: reason, act, observe (12 min)
- Handling hallucination in grounded answers (12 min)
- Hands-on lab: Grounded answers and a tool-using loop
LLM workflows and evaluation
Combine model calls into workflows with guardrails and cost and latency budgets, then evaluate them: code checks and LLM judges offline, and online metrics and A/B tests once real users arrive.
- Orchestrating multi-step workflows (12 min)
- Guardrails, cost and latency budgets (12 min)
- Offline evaluation and LLM-as-a-judge (14 min)
- Online metrics and A/B testing (12 min)
- Hands-on lab: Evaluate two prompt versions and read an A/B test
Project: Bilingual policy assistant with RAG, a search tool and an LLM judge
Lotus Lane Co-working, a fictional co-working space whose members write in English and Vietnamese, wants an assistant that answers their questions from its policies. Build it in Python on the Claude API. The starter files give you the Vietnamese policies (the English ones are policies.json from the module 4 lab), a made-up snapshot of local web pages, 16 evaluation questions, and tools.py, which detects the language, retrieves policies in the question's language and searches the snapshot. Your assistant must answer in the user's language; search the policies first and cite them like [P:guests]; call web_search, through a ReAct-style tool loop, only when the policies don't cover a question, and cite the page; say plainly when neither source has the answer, without borrowing facts about other places; and keep its model call in one function. tools.py retrieves poorly for at least two questions (run python tools.py to see): find them and fix retrieval, for example with query rewriting, more results, better Vietnamese tokenisation or a multilingual embedding model. Then evaluate it: run all 16 questions, apply code checks (facts, citations, language, abstention) and an LLM judge that scores accuracy against the reference answer and flags unsupported claims, with structured verdicts. Label at least 8 answers yourself to check the judge, and write a short report on what failed and what you changed.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.