Training Deep Neural Networks
Diagnose and fix what stops deep networks learning: vanishing and exploding gradients, slow or unstable optimisation, overfitting and badly scaled activations, in TensorFlow and Keras.
Advanced · Machine Learning · about 14 hours · 5 modules · certificate on completion
Taught by Jae-Wòn Parkk, Mei-Líng Zhāng, AI Learning Companions.
About this course
A network that's deep enough to be useful is also deep enough to be hard to train. Its gradients can fade to nothing or blow up on the way back through its layers, plain gradient descent can crawl or bounce, a large model can memorise its training data instead of learning from it, and the scale of its activations can drift as it learns. This course is about the techniques that fix those problems, and about knowing which one a failing network needs.
You'll measure gradients layer by layer and fix them with non-saturating activations, Glorot and He initialisation and gradient clipping; write momentum, RMSprop and Adam yourself and check them against Keras; schedule learning rates with decay and warm-up; regularise with L1 and L2, weight decay, dropout, augmentation and early stopping; and implement batch normalisation by hand before using it in Keras. The last module brings everything together in a training recipe and a debugging routine, and the capstone hands you a deep network that fails on Fashion-MNIST for you to diagnose and rescue, one measured change at a time. Every lab runs on a laptop CPU with fixed seeds and comes with starter code and the output to expect.
What you'll learn
- Diagnose vanishing and exploding gradients from per-layer gradient norms, and fix them with activations, initialisation and clipping
- Explain and implement momentum, RMSprop and Adam, and choose and schedule learning rates
- Control overfitting with L1 and L2 penalties, weight decay, dropout, data augmentation and early stopping
- Use batch normalisation correctly in training and inference, and know when layer normalisation suits better
- Debug a training run from its first loss and learning curves, and optimise a deep network for a hold-out test set
Skills you'll gain
Deep learning, Weight initialisation, Gradient clipping, Momentum, RMSprop and Adam, Learning-rate schedules, Regularisation, Dropout, Batch normalisation, Debugging training, TensorFlow and Keras
Tools
Python 3, TensorFlow, Keras, NumPy, scikit-learn, TensorBoard
Before you start
Introduction to Neural Networks and TensorFlow: dense networks, backpropagation, and building and training Keras models. Comfortable Python, NumPy and basic calculus. Install Python 3.10 or later, then pip install tensorflow scikit-learn matplotlib (TensorFlow 2.16 or later, with Keras 3). The capstone downloads Fashion-MNIST, about 30 MB, once.
Modules
Vanishing and Exploding Gradients
Find out why deep networks are hard to train: measure gradients shrinking or growing layer by layer, see how saturating activations make it worse, and fix it with non-saturating activations, careful initialisation and gradient clipping.
- Why gradients vanish or explode (12 min)
- Saturating and non-saturating activations (12 min)
- Weight initialisation: Glorot, He and LeCun (14 min)
- Hands-on lab: Measure vanishing gradients, then fix them
Advanced Optimisers
Go beyond plain stochastic gradient descent: momentum and Nesterov, the adaptive methods AdaGrad, RMSprop and Adam, decoupled weight decay with AdamW, and learning-rate schedules with warm-up.
- Momentum and Nesterov (12 min)
- Adaptive learning rates: AdaGrad, RMSprop and Adam (14 min)
- Learning-rate schedules and warm-up (12 min)
- Hands-on lab: Optimisers from scratch, then in Keras
Regularisation Strategies
Stop deep networks memorising their training data: penalise large weights with L1 and L2, drop units at random, stop training at the right moment, and create more training data with augmentation.
- Weight penalties: L1, L2 and weight decay (12 min)
- Dropout (12 min)
- Early stopping and data augmentation (12 min)
- Hands-on lab: Fight overfitting on a small training set
Batch Normalisation
Normalise activations inside the network: how batch normalisation computes, learns and switches between training and inference, why it lets networks train faster with higher learning rates, and where it goes wrong.
- How batch normalisation works (12 min)
- Why it helps, and where it goes wrong (12 min)
- Hands-on lab: Batch normalisation by hand and in Keras
Practical Optimisation Project
Bring every technique together: a dependable recipe for building and training a deep network, tuning the learning rate, batch size and schedule, reading learning curves to debug training, and a final evaluation on a hold-out test set.
- A recipe for training a deep network (12 min)
- Tuning the learning rate, batch size and schedule (12 min)
- Reading learning curves and debugging training (12 min)
- Hands-on lab: Rescue a badly trained network
Project: Deep network optimisation
bad_fashion.py trains an eight-layer network on Fashion-MNIST (60,000 training and 10,000 test images of clothing in 10 classes, 28 x 28 greyscale, about 30 MB, downloaded once) and gets stuck at a validation accuracy of about 0.80: its sigmoid layers start with weights drawn far too large, it uses plain SGD at a fixed learning rate, and it has no normalisation or regularisation. Diagnose it first, with its first loss against ln 10, its per-layer gradient norms and a one-batch overfitting test. Then fix it one measured change at a time, using initialisation and activations, batch normalisation, an adaptive optimiser with a tuned learning rate and schedule, and dropout or weight decay with early stopping. Keep the starter's split: 50,000 images for training, 10,000 for validation (chosen with a fixed seed) and the 10,000 test images for one final evaluation. Record every run in an ablation table. A well-tuned MLP reaches about 0.89 test accuracy: the course's reference fix, with four 256-unit ReLU layers, He initialisation, batch normalisation, dropout 0.2, AdamW with cosine decay and early stopping, reached 0.888.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.