Securing LLM Applications

Protect your own LLM features: threat-model them with the OWASP Top 10, contain prompt injection in layers, handle output and tools safely, keep data private, and prove it with an automated test suite.

Intermediate · Cybersecurity · about 22 hours · 5 modules · certificate on completion

Taught by Cataliná Ionescù, Amèlia Whitè, AI Learning Companions.

About this course

LLM features bring new ways for things to go wrong: a document that gives the assistant orders, a reply that runs script in a browser, a tool that does more than it should, a log full of personal data. This intermediate course teaches developers to protect their own LLM applications, with the OWASP Top 10 for LLM Applications as the framework. You'll threat-model a small FastAPI assistant for a fictional lettings agency and then secure it lab by lab: layered defences against direct and indirect prompt injection (instructions kept apart from data, untrusted content screened, allow-listed tools, human approval for risky actions and checked replies), safe output handling, least-privilege tools and credentials, permission-aware retrieval, redaction and clean logs, a verified supply chain, rate limits and token budgets, and plans for monitoring and incidents.

Every defence is proved by tests, and the course ends with an automated pytest suite that runs known-bad and known-good inputs against the app's guards on every change. The labs run offline with a deliberately naive stand-in model that obeys any instruction it sees, so the guards, validators and tests need no API key; optional live runs call Claude with the model read from CLAUDE_MODEL. The course is defensive throughout, and every test input is a harmless placeholder in your own app. It builds on Generative AI Design Patterns, which measures guardrails on a golden set, and Building RAG and Agentic Systems, which puts approval gates on agents. Good next steps are LLMOps: Evaluating, Tracing and Monitoring LLM Applications for tracing and guardrail metrics, and Security Operations: Monitoring, Detection and Alert Triage and Incident Response Fundamentals for running detection and response.

What you'll learn

  • Threat-model an LLM feature with data flows, trust boundaries and a risk register mapped to the OWASP Top 10 for LLM Applications
  • Contain direct and indirect prompt injection in layers: separated instructions and data, screened content, allow-listed tools, approvals and output checks
  • Handle model output safely by escaping it and validating it against strict schemas, and give tools and credentials least privilege
  • Prevent data leaks with permission-aware retrieval, redaction and clean logs, and verify the packages, models and data you depend on
  • Limit cost abuse with rate limits and token budgets, and prepare monitoring, alerts and an incident runbook for LLM features
  • Build an automated pytest suite of known-bad and known-good inputs that tests your guards without an API key on every change

Skills you'll gain

LLM threat modelling, OWASP Top 10 for LLM Applications, Prompt injection defence, Secure output handling, Least-privilege tool design, Personal data protection, Supply chain security, Rate limiting and token budgets, Security monitoring, Security regression testing

Tools

Python, FastAPI, Pydantic, pytest, SQLite, pip-tools, pip-audit, Claude API

Before you start

Confident Python and experience calling an LLM API, as in Prompting and LLM Application Design; Introduction to Cybersecurity covers the security basics, and Introduction to FastAPI and Modern Web Services the web framework. Install Python 3.10 or later, then pip install fastapi uvicorn httpx pytest; anthropic and an API key are optional, as every lab runs offline.

Modules

  1. LLM threats and the OWASP Top 10

    Why an LLM feature needs its own threat model, the OWASP Top 10 for LLM Applications as the shared framework, and a threat model for the Wrenfold assistant you'll secure throughout the course.

    • What changes when an application calls an LLM (10 min)
    • The OWASP Top 10 for LLM Applications (12 min)
    • Threat modelling an LLM feature (12 min)
    • Hands-on lab: Set up the Wrenfold assistant, run its baseline and write a threat register

  2. Prompt injection: layered defences

    How direct and indirect prompt injection work, why no single control stops them, and the layers that contain them: instructions kept apart from data, untrusted content screened, tools allow-listed, risky actions approved and replies checked.

    • Direct and indirect prompt injection (12 min)
    • Keeping instructions and data apart (10 min)
    • Treating retrieved and tool content as untrusted (12 min)
    • Allow-listed tools, human approval and output checks (12 min)
    • Hands-on lab: Layer injection defences into the assistant

  3. Safe output handling and least-privilege tools

    Treat model output as untrusted input: encode it for the page, validate structured output against strict schemas and never execute it unchecked. Then give every tool and credential only the access its task needs.

    • Model output is untrusted input (12 min)
    • Validating structured output and tool arguments (10 min)
    • Least-privilege tools and credentials (12 min)
    • Hands-on lab: Safe output and least-privilege tools

  4. Protecting data and the supply chain

    Stop the assistant revealing what it shouldn't: filter documents by permission before the model sees them, minimise and redact personal data, log without leaking, and verify the packages, models and data the app is built from.

    • How LLM applications leak data (10 min)
    • Minimising and redacting personal data (10 min)
    • Logging without leaking (10 min)
    • The supply chain: models, packages and data (12 min)
    • Hands-on lab: Permission-aware retrieval, clean logs and a verified knowledge base

  5. Abuse limits, monitoring, incidents and security testing

    Keep the assistant affordable and observable under abuse with size limits, rate limits and token budgets; turn guard decisions into security events and alerts; prepare for an LLM incident; and prove every defence with an automated pytest suite.

    • Cost abuse, rate limits and budgets (12 min)
    • Logging, monitoring and alerting for LLM features (10 min)
    • Handling an LLM security incident (10 min)
    • An automated security test suite with pytest (12 min)
    • Hands-on lab: Rate limits, budgets and an automated security suite

Project: Secure and test a new LLM feature end to end

Wrenfold wants a second feature: a staff assistant that reads contractors' emails about repairs, summarises them on a web page and drafts replies, which are sent only after a member of staff approves them. Build it into your Wrenfold app (or do the same work on an LLM application of your own, with the same deliverables). Contractors' emails are untrusted content that can carry instructions; drafts go to people outside the company; summaries are shown in a browser; and the emails contain names, phone numbers and prices. Threat-model the feature against the OWASP Top 10 for LLM Applications, build its defences in layers, protect its data, limit its cost, plan its monitoring and incident response, and prove it all with an automated security suite that runs without an API key in CI. Use only harmless placeholder test inputs. Code that calls Claude must read the model from CLAUDE_MODEL, and every test must pass with the stand-in model.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free