Module 3 – Optimization Foundations & Ablation Methodology

Name: Lecture 3 – Optimization Foundations & Ablation Methodology
Description: Before diving into deep multilayer perceptrons, we add regularization (weight decay) and data splits (train/validation). We minimize MSE(y, ŷ) + λ‖θ‖² on the training split and evaluate generalization on the validation split.

What you'll learn

Builds on Modules 1–2

≈5 h · 1.5 h video · 1.5 h reading · 2 h coding

Run an optimizer sweep (SGD/Momentum/Adam) with LR schedules; compare results
Design a 2–3 factor ablation; log all runs with consistent seeds and configs

Lecture 3 – Optimization Foundations & Ablation Methodology

Before diving into deep multilayer perceptrons, we add regularization (weight decay) and data splits (train/validation). We minimize MSE(y, ŷ) + λ‖θ‖² on the training split and evaluate generalization on the validation split.

3.2: Training MLP I

3.3: Training MLP II

📚 Resources & Lecture Code

Required reading · course book

Ch. 4 · Training: Loss and SGD
Ch. 6 · Generalization and Inductive Bias
Interlude · Who Trains the Trainer? Learning by Experiment
Appendix B · Tensors in Practice
Appendix D · Notation
Appendix E · Statistical Learning Contracts

The Colab notebook contains the lecture code for Module 3 (optimization and training MLPs). Run the cells sequentially as demonstrated in the lecture.

Open Colab Notebook (shared for Modules 2–3)

Module 2

Module 4