Machine Learning from Scratch
Build gradient descent, k-NN, k-means, Naive Bayes, linear and logistic regression and decision trees in NumPy, and prove each one matches scikit-learn.
Intermediate · Machine Learning · about 26 hours · 5 modules · certificate on completion
Taught by Nataliyà Kovalenkò, Chukwuemekà Obiechè, AI Learning Companions.
About this course
Machine learning libraries make models easy to use and easy to misunderstand. In this course you build the classic algorithms yourself, in Python with NumPy, so you know exactly what fit and predict do. You start with the maths as code: vectors, matrices and broadcasting, then gradients and gradient descent, checked numerically and tuned through the learning rate, feature scaling and mini-batches. Then you build k-nearest neighbours and k-means clustering on fast distance computations, Naive Bayes for word counts and for continuous measurements with Laplace smoothing and log probabilities, linear and logistic regression trained by gradient descent with an L2 penalty, and a decision tree grown with entropy and information gain.
scikit-learn is used only to load data, split it and check you: every lab ends with a script that compares your model with scikit-learn's on its built-in datasets, so you see your predictions, coefficients and metrics agree, and learn why they sometimes differ (tied splits, 32-bit thresholds, regularisation conventions). Every lesson has a JupyterLab walkthrough that runs its ideas on real numbers. In the capstone you gather everything into a small, tested library with one consistent API and benchmark it against scikit-learn.
What you'll learn
- Write machine learning maths as vectorised NumPy code, and check every gradient numerically
- Train models with batch, stochastic and mini-batch gradient descent, choosing learning rates and scaling features
- Implement k-nearest neighbours, k-means and Naive Bayes from scratch, with smoothing and log probabilities
- Implement linear and logistic regression with MSE, log-loss and L2 penalties, and decision trees with entropy and information gain
- Evaluate models with metrics you write yourself, and verify every implementation against scikit-learn
- Package your algorithms as a tested library with one consistent API, and benchmark it on standard datasets
Skills you'll gain
NumPy, Vectorisation, Gradient descent, k-nearest neighbours, k-means clustering, Naive Bayes, Linear regression, Logistic regression, Decision trees, Model evaluation
Tools
JupyterLab, Python 3, NumPy, scikit-learn, matplotlib, pytest
Before you start
Python Foundations for Data Science (or comfortable Python with functions, classes and NumPy arrays) and Foundations of Probability and Statistics (conditional probability, the normal distribution, means and variances). Machine Learning Foundations with Python helps: this course rebuilds the models it uses. Install Python 3 with NumPy, scikit-learn, matplotlib, pytest and JupyterLab; every dataset comes with scikit-learn or the labs.
Modules
Linear Algebra and Gradient Descent
Write the maths of machine learning as vectorised NumPy code, then minimise a loss with gradient descent, checking every gradient numerically and seeing how the learning rate, feature scaling and batch size change the path to the minimum.
- Vectors, matrices and vectorised NumPy (12 min)
- Loss functions, gradients and gradient descent (12 min)
- Learning rates, feature scaling and stochastic gradient descent (12 min)
- Hands-on lab: Gradient descent from scratch
k-Nearest Neighbours (k-NN)
Build a k-nearest neighbours classifier on distances computed for whole arrays at once, see why scaling and the choice of k matter, and reuse the same distances, without labels, to cluster data with k-means.
- Distances and the k-NN classifier (12 min)
- Choosing k, scaling, and the cost of a lazy learner (12 min)
- k-means: the same distances, without labels (12 min)
- Hands-on lab: k-NN and k-means from scratch
Naive Bayes
Turn Bayes' theorem into a classifier: estimate class priors and per-feature likelihoods from word counts or from normal distributions, smooth them, and work with log probabilities so tiny numbers don't vanish.
- Bayes' theorem as a classifier (12 min)
- Multinomial Naive Bayes for text, and Laplace smoothing (12 min)
- Gaussian Naive Bayes for continuous features (10 min)
- Hands-on lab: Naive Bayes from scratch: a spam filter and a Gaussian model
Linear and Logistic Regression
Build the two workhorse linear models as classes with fit and predict: linear regression trained by gradient descent and checked against the closed-form least-squares answer, and logistic regression with the sigmoid, the log-loss and an L2 penalty, each scored with metrics you write yourself.
- Linear regression: gradient descent and the closed form (12 min)
- Logistic regression: the sigmoid and the log-loss (12 min)
- Metrics you write yourself (12 min)
- Hands-on lab: Linear and logistic regression from scratch
Decision Trees
Grow a classification tree recursively: measure impurity with entropy or Gini, choose the split with the highest information gain, stop at a depth limit, and compare the tree you built with scikit-learn's, split by split.
- Entropy, Gini impurity and information gain (12 min)
- Growing a tree recursively (12 min)
- Overfitting, depth limits and pruning (10 min)
- Hands-on lab: Grow a decision tree
Project: Build your own machine learning library
Turn the code from the five labs into one small Python package with a consistent, scikit-learn-like API, then prove it works. Name it what you like; the starter files call it mylearn. Every model has fit(X, y), which returns the model itself, predict(X) and score(X, y), plus predict_proba(X) where it makes sense; hyperparameters are set in __init__ and learned attributes end in an underscore, as in scikit-learn. Include KNNClassifier, KMeans, MultinomialNB, GaussianNB, LinearRegressionGD, LogisticRegressionGD and DecisionTreeClassifier, a metrics module with the regression and classification metrics you wrote, and a preprocessing module with StandardScaler and a train_test_split of your own. Share code through small base classes, such as one that gives every classifier the same score method. Test the library with pytest against scikit-learn on its built-in datasets, and benchmark it: a table of dataset, model, your score, scikit-learn's score and both fit times. Finish with a README and a short report on where your library matches scikit-learn, where it differs and why (tied splits, 32-bit thresholds, stopping tolerances), and how its speed compares. For extra polish: multiclass logistic regression with softmax, a k-NN regressor, or a faster best_split.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.