Feature Engineering and Model Evaluation
Prepare real, messy data for machine learning, evaluate models reliably with cross-validation, tune them honestly, and wrap it all in leak-proof scikit-learn pipelines.
Intermediate · Machine Learning · about 15 hours · 5 modules · certificate on completion
Taught by Marisolò Garcíà, Zawadì Mutembò, AI Learning Companions.
About this course
Models are only as good as the data they're given and the way they're judged. This course covers both halves, in scikit-learn. On the data side you scale features and learn when it matters, tame skewed values and outliers, fill in missing values while keeping the signal that missingness itself carries, and turn categories into numbers: one-hot and ordinal encoding, grouping rare levels, target encoding done without leaking the answer, and features from dates, text and interactions. On the evaluation side you use k-fold, stratified, group and time-series cross-validation, read several metrics and training scores at once, tune hyperparameters with grid, randomised and halving searches, and measure the tuning itself with nested cross-validation.
Everything comes together in pipelines: ColumnTransformer for mixed columns, custom steps, tuning preprocessing and model together, and saving the result as one object, with experiments that show how leakage inflates scores when a step is fitted in the wrong place. The labs work on a made-up club's season-ticket renewals, where every technique makes a measurable difference, and every lesson has a JupyterLab walkthrough with real outputs. The capstone is a robust, tuned pipeline for a new, messy dataset, evaluated honestly.
What you'll learn
- Apply and compare scaling and normalisation techniques, and tame skewed features and outliers
- Impute missing values while keeping the information that missingness carries
- Encode categorical variables appropriately, including high-cardinality features with leak-free target encoding
- Implement k-fold, stratified, group and time-series cross-validation for robust evaluation
- Optimise hyperparameters with grid, randomised and halving searches, and evaluate tuning with nested cross-validation
- Construct scikit-learn pipelines that prevent data leakage, from raw columns to a saved model
Skills you'll gain
Feature scaling, Missing-value imputation, Categorical encoding, Target encoding, Cross-validation, Hyperparameter tuning, Nested cross-validation, scikit-learn pipelines, Data leakage prevention
Tools
JupyterLab, Python 3, scikit-learn, pandas, NumPy, SciPy, joblib
Before you start
Classical Classification and Clustering (or Machine Learning Foundations with Python): train/test splits, scikit-learn models and cross_val_score. Comfortable pandas. Install Python 3 with pandas, scikit-learn (1.3 or later for TargetEncoder; the outputs shown come from 1.9) and JupyterLab; SciPy and joblib come with scikit-learn. Every dataset comes with the labs.
Modules
Data Scaling & Preprocessing
Put numeric features on comparable scales, tame skew and outliers, and fill in missing values in ways that keep the information missingness carries, with every step fitted on training data only.
- Why scale, and which scaler (12 min)
- Skewed features and outliers (10 min)
- Missing values: imputation and indicators (12 min)
- Hands-on lab: Scale and impute the season-ticket data
Categorical Encoding
Turn categories and text into numbers: one-hot encoding for unordered categories, ordinal encoding for ordered ones, grouping or target encoding for many categories, and simple features from dates, text and interactions.
- One-hot and ordinal encoding (12 min)
- High-cardinality features and target encoding (12 min)
- Dates, text and interactions (10 min)
- Hands-on lab: Encode the categories
Cross-Validation Techniques
Estimate performance reliably without overfitting to a single test split: k-fold and stratified k-fold, groups and time order, repeated runs, several metrics at once, and what all those fits cost.
- k-fold and stratified k-fold (12 min)
- Groups, time and repeated cross-validation (12 min)
- Metrics, training scores and learning curves (10 min)
- Hands-on lab: Validate it properly
Hyperparameter Tuning
Search systematically for the best model settings: exhaustive grid search, randomised search over distributions, successive halving, and nested cross-validation to measure the whole tuning process honestly.
- Hyperparameters and grid search (12 min)
- Randomised search and successive halving (12 min)
- Nested cross-validation (10 min)
- Hands-on lab: Tune a gradient boosting pipeline
scikit-learn Pipelines
Chain preprocessing and modelling so that every step runs in the right order on the right data: ColumnTransformer for mixed columns, leak-proof cross-validation, custom steps, tuning preprocessing and model together, and saving the whole workflow as one object.
- Pipelines and ColumnTransformer for mixed data (12 min)
- Leakage, and how pipelines prevent it (12 min)
- Custom steps, tuning everything together, and saving (12 min)
- Hands-on lab: Build a leak-proof pipeline
Project: Robust pipeline construction
Build and tune a complete scikit-learn pipeline that predicts whether a learner completes an online course, from enrolments.csv: 360 made-up learners with their weekly study hours, earlier courses, age, country, device, sign-up month, whether they paid, their average quiz score and their forum posts. The data is messy on purpose: missing ages and quiz scores (and a missing quiz score may itself be telling), skewed hours and post counts, 24 countries of very different sizes, and a cyclical month. Hold out a stratified test set first. Build one Pipeline whose ColumnTransformer handles every column, with a reason for each choice: imputation, with indicators where missingness may matter; transforms for skew; scaling; one-hot encoding for device and paid; target encoding or grouping for country; and sine and cosine features for the month. Compare at least two model families, tune their hyperparameters and at least one preprocessing choice with GridSearchCV or RandomizedSearchCV, and report the tuned procedure's performance with nested cross-validation on the training set. Only then fit the final model, score it once on the test set and save it with joblib. Finally, show with one deliberate experiment how a leak, such as target encoding or imputation fitted before the split, would have inflated your scores.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.