Reinforcement Learning: From Bandits to Policy Gradients

Learn how agents learn from reward, from bandits and dynamic programming through Monte Carlo, TD learning, SARSA and Q-learning to deep Q-networks and policy gradients in PyTorch.

Advanced · Machine Learning · about 16 hours · 5 modules · certificate on completion

Taught by Jae-Wòn Parkk, Kaimànà Kahananùi, AI Learning Companions.

About this course

Reinforcement learning is learning by trial and error: an agent acts, sees what happens and collects reward, and has to work out which of its actions led to the reward and which exploring is worth the cost. It's behind game-playing agents, robot controllers, recommendation and experimentation systems, and the fine-tuning of large language models from human feedback. This course builds it up in the standard order, each idea on the one before: bandits, Markov decision processes, dynamic programming, Monte Carlo and temporal-difference learning, SARSA and Q-learning, and then deep Q-networks and policy gradients.

Every module has a lab that reproduces a classic experiment in code you complete yourself: exploration strategies on the 10-armed testbed, policy and value iteration on a slippery grid world, Monte Carlo against TD on the random walk, SARSA against Q-learning on the cliff, and REINFORCE on Gymnasium's CartPole in PyTorch. The capstone is a stock-control problem where dynamic programming gives the optimal policy, so you can measure exactly how close your learned agents get.

What you'll learn

  • Balance exploration and exploitation in bandit problems with ε-greedy, UCB and Thompson sampling, and measure regret
  • Formulate problems as Markov decision processes and solve them with policy iteration and value iteration
  • Estimate values from experience with Monte Carlo and temporal-difference learning
  • Learn control policies with SARSA and Q-learning, and explain how on-policy and off-policy methods differ
  • Explain how deep Q-networks stabilise learning, and train a policy-gradient agent with REINFORCE in PyTorch

Skills you'll gain

Reinforcement learning, Multi-armed bandits, Markov decision processes, Dynamic programming, Monte Carlo and TD learning, SARSA and Q-learning, Deep Q-networks, Policy gradients, PyTorch, Gymnasium

Tools

Python 3, NumPy, PyTorch, Gymnasium

Before you start

Deep Learning with PyTorch (for the last module) and probability at the level of Statistics for Data Science and AI or Foundations of Probability and Statistics: expectations, conditional probability and sampling. Comfortable Python and NumPy. Install Python 3.10 or later with NumPy, PyTorch (pip install torch; the CPU build is enough) and Gymnasium (pip install gymnasium, for CartPole). Everything runs on a laptop CPU.

Modules

  1. Bandits: exploration and exploitation

    Meet the reinforcement learning problem in its simplest form: choosing between options with uncertain rewards, and deciding when to try something new.

    • The reinforcement learning problem (10 min)
    • Multi-armed bandits and action values (12 min)
    • Exploration strategies (12 min)
    • Hands-on lab: Exploration strategies on the 10-armed testbed

  2. Markov decision processes and dynamic programming

    Describe sequential decision problems as Markov decision processes, write their Bellman equations, and solve them exactly when the dynamics are known.

    • Markov decision processes (12 min)
    • Optimal policies and the Bellman optimality equation (10 min)
    • Policy iteration and value iteration (12 min)
    • Hands-on lab: Solve a slippery grid world with dynamic programming

  3. Learning from experience: Monte Carlo and TD

    Estimate values without a model, from sampled episodes with Monte Carlo methods and from single steps with temporal-difference learning.

    • Monte Carlo prediction (12 min)
    • Temporal-difference learning (12 min)
    • Hands-on lab: Monte Carlo against TD(0) on the random walk

  4. Control: SARSA and Q-learning

    Turn value estimates into better behaviour, learning action values on-policy with SARSA and off-policy with Q-learning, and see where each one shines.

    • From prediction to control (10 min)
    • SARSA and Expected SARSA (12 min)
    • Q-learning (12 min)
    • Hands-on lab: SARSA and Q-learning on the cliff

  5. Deep reinforcement learning

    Scale up with neural networks: deep Q-networks and the tricks that keep them stable, policy gradients with REINFORCE, and actor-critic methods, using PyTorch and Gymnasium.

    • Function approximation and deep Q-networks (12 min)
    • Policy gradients and REINFORCE (12 min)
    • Actor-critic methods and reinforcement learning in practice (12 min)
    • Hands-on lab: REINFORCE on CartPole

Project: Stock control: from dynamic programming to learned agents

inventory.py models a small shop's stock: each evening the agent orders 0 to 10 units, Poisson demand with a mean of 3 arrives the next day, and the day's profit counts sales, ordering costs (including a fixed cost for any order), holding costs and unmet demand. Episodes last 100 days. Because transition_model() gives the exact dynamics, dynamic programming can find the optimal policy, and every learned agent can be measured against it. Solve the problem with policy or value iteration (γ = 0.99), learn it from simulated experience with Q-learning and SARSA, then train an agent with function approximation in PyTorch, a DQN or REINFORCE, and compare them all with evaluate(), which runs 200 fixed episodes. For calibration, on the course's machine never ordering lost 2.68 a day, ordering 3 every day lost 2.08, the rule 'below 4 in stock, order up to 8' earned 3.86, and the optimal policy earned 4.76. Finally, test each policy when mean demand rises to 4, and explain what a shop owner should take from your results.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free