Statistics for Data Science and AI
Describe data, reason with probability, and test ideas with confidence intervals, hypothesis tests, regression and A/B tests, in Python with pandas and SciPy.
Beginner · Data Science · about 10 hours · 4 modules · certificate on completion
Taught by Ferencz Nagŷ, Idrissà Ouédraogò, AI Learning Companions.
About this course
Data science and AI are built on statistics: every model is fitted to a sample, every evaluation is an estimate, and every claim that one version beats another is a test. In this course you learn the statistics behind those decisions by doing it in Python. You'll describe data with pandas, from means and medians to quartiles and outliers; reason about chance with the binomial, Poisson and normal distributions in SciPy; see the central limit theorem happen in a simulation; and build confidence intervals and hypothesis tests that say how sure you can be. You'll also learn what p-values don't tell you, measure relationships with correlation and regression, and analyse an A/B test, including how many users it needs before it starts. Every lesson is shown in PyCharm, with the Python Console, the debugger and the docs at your fingertips, every module ends with a lab on made-up sample data that comes with the course, and the course ends with a project in which you analyse a field trial and write up what it does and doesn't show.
What you'll learn
- Summarise a dataset's centre, spread and shape with pandas, and investigate its outliers
- Calculate probabilities with the binomial, Poisson and normal distributions in SciPy
- Explain sampling distributions and the central limit theorem, and build confidence intervals
- Run and interpret t-tests and chi-square tests, and avoid the common traps with p-values
- Measure relationships with correlation and simple linear regression
- Analyse an A/B test and work out how many users it needs
Skills you'll gain
Descriptive statistics, Probability, Probability distributions, Central limit theorem, Confidence intervals, Hypothesis testing, Correlation and regression, A/B testing, pandas, SciPy
Tools
PyCharm, Python, pandas, SciPy
Before you start
Basic Python: variables, functions, lists and running a script (Python Programming with PyCharm covers it), and school maths. Install PyCharm (the free version is enough) and Python 3; you'll install pandas and SciPy into a project's virtual environment in the first lesson.
Modules
Describing data
Summarise a dataset before you model it: the kinds of data, the centre, the spread and the shape, outliers, and how groups differ, with pandas in PyCharm.
- Kinds of data, and where the middle is (10 min)
- Spread, quartiles and outliers (12 min)
- Comparing groups (10 min)
- Hands-on lab: Describe a month of deliveries
Probability and distributions
Reason about chance: the rules of probability, conditional probability, and the binomial, Poisson and normal distributions, calculated with SciPy and checked by simulation.
- Probability rules and conditional probability (12 min)
- Counting successes: the binomial and Poisson distributions (12 min)
- The normal distribution (12 min)
- Hands-on lab: Probability with SciPy
From samples to conclusions
Go from a sample to the population it came from: the central limit theorem, confidence intervals, hypothesis tests, and the traps that make p-values mislead.
- Samples, sampling distributions and the central limit theorem (12 min)
- Confidence intervals (12 min)
- Hypothesis tests and p-values (12 min)
- The traps in p-values (12 min)
- Hands-on lab: Intervals and tests on real questions
Relationships and experiments
Measure how two variables move together with correlation and simple linear regression, then run the experiment that data science uses most: the A/B test.
- Correlation (10 min)
- Simple linear regression (12 min)
- A/B tests (12 min)
- Hands-on lab: Rainfall, yields and an A/B test
Project: Analyse a field trial of SMS weather advice
A farmers' cooperative piloted a service that texts weather advice to farmers (the data are made up). Of 120 farmers across three regions, 60 were chosen at random to get the texts for a season, and the other 60 carried on as before; sms_trial.csv records each farmer's group, region, the season's rainfall on their plot and their yield. The cooperative wants to know whether to pay for the service for everyone. Analyse the trial in PyCharm with pandas and SciPy: describe the data, check that randomisation balanced the groups, estimate the difference in yield with a confidence interval and a test, look at how rainfall relates to yield, and write a short report for people who aren't statisticians that says what the trial shows, what it doesn't, and what you recommend.
Part of
PyCharm is a trademark of JetBrains s.r.o. This course is independent: JetBrains hasn't made or endorsed it.