Name: Lecture 3 – Optimization Foundations & Ablation Methodology
Description: Before diving into deep multilayer perceptrons, we add regularization (weight decay) and data splits (train/validation). We minimize MSE(y, ŷ) + λ‖θ‖² on the training split and evaluate generalization on the validation split.

What you'll learn

Builds on Modules 1–2

≈5 h · 1.5 h video · 1.5 h reading · 2 h coding

Run an optimizer sweep (SGD/Momentum/Adam) with LR schedules; compare results
Design a 2–3 factor ablation; log all runs with consistent seeds and configs

Lecture 3 – Optimization Foundations & Ablation Methodology

Before diving into deep multilayer perceptrons, we add regularization (weight decay) and data splits (train/validation). We minimize MSE(y, ŷ) + λ‖θ‖² on the training split and evaluate generalization on the validation split.

3.2: Training MLP I

3.3: Training MLP II

📚 Resources & Lecture Code

Course book · primary reading

Alternative reading from Dive into Deep Learning (D2L):

The Colab notebook contains the lecture code for Module 3 (optimization and training MLPs). Run the cells sequentially as demonstrated in the lecture.

Open Colab Notebook (shared for Modules 2–3)

🧠 Test your understanding

Try each question before revealing the answer — these mirror the ideas the module quiz checks.

Q1.In a standard autograd training step, what is the correct order of operations?

$\texttt{backward()} \to \texttt{zero\_grad()} \to \texttt{step()}$
$\texttt{zero\_grad()} \to \texttt{forward()} \to \texttt{loss()} \to \texttt{backward()} \to \texttt{step()}$
$\texttt{forward()} \to \texttt{step()} \to \texttt{loss()} \to \texttt{zero\_grad()}$
$\texttt{forward()} \to \texttt{backward()} \to \texttt{zero\_grad()} \to \texttt{step()}$

Show answer ▾

Answer: $\texttt{zero\_grad()} \to \texttt{forward()} \to \texttt{loss()} \to \texttt{backward()} \to \texttt{step()}$

Clear old gradients first, then compute forward pass and loss, backpropagate to compute gradients, and finally update parameters. Placing $\texttt{zero\_grad()}$ after $\texttt{backward()}$ would discard the current gradients before using them to update.

Q2.Which statement about activation functions in deep MLPs is most accurate?

Sigmoid is preferred because its derivative is always $\geq 0.25$
Tanh eliminates vanishing gradients in deep networks
ReLU (or variants) is generally preferred; sigmoid/tanh can saturate and vanish
All activations are equivalent if the learning rate is small

Show answer ▾

Answer: ReLU (or variants) is generally preferred; sigmoid/tanh can saturate and vanish

ReLU avoids the saturation zones where derivatives approach zero, which is critical for gradient flow in deep networks. Sigmoid and tanh have derivative bounds (like $\sigma'(z) \leq 0.25$ ) and saturate to plateau regions where gradients vanish regardless of learning rate.

Q3.Which change best improves gradient flow in a very deep MLP (same width)?

Replace all ReLUs with sigmoids
Add residual (skip) connections between blocks
Remove bias terms
Use $\texttt{sum()}$ instead of $\texttt{mean()}$ in the loss

Show answer ▾

Answer: Add residual (skip) connections between blocks

For a residual block, the input Jacobian contains an additive identity term as well as the learned branch. That direct route usually improves conditioning and makes the identity function easy to represent. It is not attenuation-proof: the learned Jacobian can reinforce or partly cancel the identity contribution, and successful depth still depends on normalization, initialization, optimization, and architecture.

Q4.Proper initialization for ReLU networks typically uses:

Xavier only, regardless of activation
He/Kaiming init to maintain activation/gradient scale
All zeros for faster symmetry breaking
Random choice; init rarely matters

Show answer ▾

Answer: He/Kaiming init to maintain activation/gradient scale

He initialization sets $\text{Var}(w) = 2/\text{fan\_in}$ to preserve the variance of activations and gradients through ReLU layers. This is critical because ReLU kills half the activations (zeros out negative values), so He init accounts for this by doubling the variance compared to Xavier initialization designed for symmetric activations.

Q5.In an ablation study for neural network training, what is the correct methodology for isolating the effect of a single technique (e.g., batch normalization)?

Apply all techniques simultaneously and compare against a baseline to measure total improvement
Start with a baseline configuration, change one component at a time, and measure performance changes for each modification
Test each technique in isolation on completely separate datasets to avoid any interaction effects
Apply techniques in random order and compute statistical significance with p-values

Show answer ▾

Answer: Start with a baseline configuration, change one component at a time, and measure performance changes for each modification

The ablation study methodology involves a controlled baseline and then systematically adding or modifying one component at a time while keeping everything else fixed. This allows you to isolate and measure the individual contribution of each technique. The lecture demonstrates this by starting with a control configuration (default initialization, no normalization, no dropout), then progressively adding He initialization, batch norm, layer norm, and dropout one at a time to see which components provide the most performance benefit.

Q6.Why is batch normalization usually omitted from the final classification logits?

It makes each example's scores depend on batch statistics and introduces a train/eval statistic shift where stable, independently interpretable logits are preferred
It is too expensive for a small output layer
Cross-entropy cannot differentiate through normalization
Batch normalization only works after convolution

Show answer ▾

Answer: It makes each example's scores depend on batch statistics and introduces a train/eval statistic shift where stable, independently interpretable logits are preferred

Cross-entropy accepts any real logits, so normalization is not mathematically forbidden. The practical concern is that output BN couples one example's scores to the other examples in its batch and changes behavior between batch and running statistics, which can complicate calibration and interpretation. Hidden layers often benefit from that normalization; the final score layer usually does not need it.