Practical Statistics for Data Scientists
Apply statistics to real decisions: robust summaries, the bootstrap, A/B tests done properly, regression with diagnostics, and logistic models judged by the costs of their errors.
Intermediate · Data Science · about 15 hours · 5 modules · certificate on completion
Taught by Ferencz Nagŷ, Tatiànna Morrisòn, AI Learning Companions.
About this course
Knowing what a p-value is doesn't tell you how to run an A/B test, and knowing the formula for a regression doesn't tell you whether to trust one. This course is about statistics as data scientists use it, on the kind of data they meet: skewed, messy and gathered for some other purpose. You'll start by profiling an online shop's orders with robust statistics, the median, trimmed means and the MAD, see why z-scores miss outliers, and meet Simpson's paradox. Then you'll learn where uncertainty comes from: sampling and its biases, standard errors, and the bootstrap, which gives intervals for medians, differences and ratios that have no tidy formula.
The middle of the course is experiments: designing an A/B test with a primary metric and guardrails, checking the randomisation, analysing it with permutation tests, z-tests and Welch's t-test, working out sample sizes, and seeing by simulation why peeking and multiple comparisons produce false positives. Then you'll build regression models with statsmodels and check them properly, with residual plots, the Breusch-Pagan test, Cook's distance and variance inflation factors, and model which clinic appointments will be missed with logistic regression, reading its odds ratios and choosing a threshold from a confusion matrix. All the data are made up for the course and come with the labs, whose results you can check exactly, and the capstone is a complete A/B test case study: design a test of a free-delivery threshold, analyse its results and recommend what the shop should do.
What you'll learn
- Summarise skewed, messy data with robust statistics, and recognise confounding and Simpson's paradox
- Estimate standard errors and confidence intervals for almost any statistic with the bootstrap
- Design, check and analyse A/B tests, with permutation tests, power calculations and corrections for multiple testing
- Build multiple linear regression models, interpret their coefficients and check them with residual and influence diagnostics
- Fit and interpret logistic regression, and choose thresholds with confusion matrices, precision and recall
Skills you'll gain
Exploratory data analysis, Robust statistics, Bootstrap and resampling, A/B testing, Permutation tests, Power and sample size, Multiple testing, Linear regression, Regression diagnostics, Logistic regression
Tools
Python 3, pandas, SciPy, statsmodels, matplotlib, JupyterLab
Before you start
Foundations of Probability and Statistics, or the same grounding in distributions, confidence intervals and hypothesis tests, and pandas at the level of Data Wrangling with pandas and NumPy. Statistics for Data Science and AI covers first tests and regression; this course takes them into practice. Install Python 3 with pandas, SciPy and statsmodels (0.14 or later), plus matplotlib for the plots; JupyterLab is optional.
Modules
Exploratory Data Analysis Refresher
Look at data the way a practising statistician does: robust measures of centre and spread, outlier rules that don't fool themselves, plots that show a distribution honestly, and the confounding that makes correlations mislead.
- Robust estimates of location (10 min)
- Spread, outliers and robust scales (12 min)
- Seeing distributions: histograms, boxplots and ECDFs (10 min)
- Correlation, confounding and Simpson's paradox (12 min)
- Hands-on lab: Profile an online shop's orders
Data and Sampling Distributions
Understand where the uncertainty in a statistic comes from: how samples are drawn and biased, sampling distributions and standard errors, and the bootstrap, which estimates them for almost any statistic.
- Sampling, bias and the sampling distribution (12 min)
- The bootstrap (12 min)
- Confidence intervals in practice (12 min)
- Hands-on lab: Bootstrap the shop's numbers
Statistical Experiments and Significance Testing
Design and analyse A/B tests the way experienced teams do: a primary metric and guardrails, a randomisation check, permutation and classical tests, power and sample size, and the traps of peeking and multiple comparisons.
- Designing an A/B test (12 min)
- Permutation tests and classical tests (12 min)
- Power, sample size and the traps of peeking and multiple tests (12 min)
- Hands-on lab: Analyse a checkout A/B test end to end
Regression and Prediction
Build and evaluate linear regression models with statsmodels: simple and multiple regression, categorical predictors and transformations, fit and prediction error, and the residual, influence and collinearity diagnostics that say whether to trust them.
- Simple and multiple linear regression (12 min)
- Fit, prediction and their intervals (10 min)
- Residual diagnostics (12 min)
- Hands-on lab: Model delivery times, then check the model
Classification and Logistic Regression
Model a yes-or-no outcome with logistic regression: probabilities, odds and the logit, coefficients as odds ratios, likelihood-ratio tests and calibration, and decisions with confusion matrices, precision and recall.
- Logistic regression: probabilities, odds and the logit (12 min)
- Interpreting and checking a logistic model (12 min)
- Confusion matrices, thresholds and imbalanced classes (12 min)
- Hands-on lab: Predict appointment no-shows
Project: Comprehensive A/B Testing Case Study: a free-delivery threshold
A made-up online shop wants to know whether lowering its free-delivery threshold, from orders of 60 to orders of 40, will help. Before the test it converted about 3.2% of its roughly 2,000 daily visitors. First, design the test as if it hadn't started: the hypothesis; the primary metric and guardrail metrics (at least revenue per visitor and average order value); the randomisation unit; a minimum detectable effect of 0.5 points of conversion; the sample size for 80% power at the 5% level; the duration in whole weeks; and the stopping rule. Then analyse the 21 days of results the shop collected: experiment_daily.csv (visitors, orders and revenue by day, variant and device) and experiment_orders.csv (the value of every order). Check the split for a sample ratio mismatch; test conversion with a two-proportion z-test and a permutation test, with an interval for the difference; compare revenue per visitor, counting every visitor, and average order value with Welch's t-tests and bootstrap intervals; look at mobile and desktop separately, correcting for multiple tests; and check whether the effect changed from week to week. Finish with a recommendation for the shop's managers that weighs the extra orders against the smaller baskets and the cost of the free deliveries, and proposes the next test. Check yourself: the design needs about 20,900 visitors per group; A had 21,047 visitors and 663 orders and B 20,917 and 754 (SRM p = 0.53); conversion rose from 3.15% to 3.60% (+0.45 points, 95% CI +0.11 to +0.80, p = 0.0099); and the average order value fell from 63.71 to 57.68.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.