Skip to main content
What you'll learn
Builds on Module 1
≈6 h · 2 h video · 2 h reading · 2 h coding
  • Derive gradients for a 1–2 layer network and draw the computation graph
  • Implement gradient checking and validate layer derivatives
  • Use torch.autograd to build a tiny autograd toy and inspect backward
Lecture 2 – Backpropagation
Lecture 2.2 – Coding in PyTorch & Linear Regression with Autograd
📚 Resources & Lecture Code

Course book · primary reading

Alternative reading from Dive into Deep Learning (D2L):

The Colab notebook contains the lecture code for Module 2 (backpropagation and PyTorch basics). Follow along by running the cells in order.

Open Colab Notebook (shared for Modules 2–3)
🧠 Test your understanding

Try each question before revealing the answer — these mirror the ideas the module quiz checks.

Q1.For mean squared error loss L=12n∑i=1n(y^i−yi)2L = \frac{1}{2n}\sum_{i=1}^n (\hat{y}_i - y_i)^2, what is ∂L∂y^i\frac{\partial L}{\partial \hat{y}_i}?Show answer ▾

Answer: 1n(y^i−yi)\frac{1}{n}(\hat{y}_i - y_i)

The factor 12\frac{1}{2} in the loss cancels the factor of 2 from differentiating the squared term, leaving 1n(y^i−yi)\frac{1}{n}(\hat{y}_i - y_i). This is the gradient of MSE with respect to predictions, foundational for understanding backpropagation.

Q2.For loss L=12n∥Xw+b1−y∥22L = \frac{1}{2n}\|\mathbf{X}\mathbf{w} + b\mathbf{1} - \mathbf{y}\|_2^2, the gradient with respect to weights w\mathbf{w} is which of the following? (a) 1nX⊤(Xw+b1−y)\frac{1}{n}\mathbf{X}^{\top}(\mathbf{X}\mathbf{w} + b\mathbf{1} - \mathbf{y}), (b) 1nX(Xw+b1−y)\frac{1}{n}\mathbf{X}(\mathbf{X}\mathbf{w} + b\mathbf{1} - \mathbf{y}), (c) 1n(Xw+b1−y)\frac{1}{n}(\mathbf{X}\mathbf{w} + b\mathbf{1} - \mathbf{y}), (d) X⊤Xw\mathbf{X}^{\top}\mathbf{X}\mathbf{w}Show answer ▾

Answer: (a) 1nX⊤(Xw+b1−y)\frac{1}{n}\mathbf{X}^{\top}(\mathbf{X}\mathbf{w} + b\mathbf{1} - \mathbf{y})

The matrix transpose and multiplication by X⊤\mathbf{X}^{\top} are essential when computing the gradient of a quadratic form with respect to w\mathbf{w}. This is the core equation derived in the backpropagation handout and is critical for understanding how gradients flow through linear layers.

Q3.In PyTorch, when using an optimizer in a training loop, what is the correct order of operations each iteration?Show answer ▾

Answer: zero_grad()\texttt{zero\_grad()} → backward()\texttt{backward()} → step()\texttt{step()}

First clear accumulated gradients from the previous iteration, then compute new gradients via backpropagation, then update parameters. Reversing this order accumulates gradients incorrectly or updates with stale information, a common source of training bugs.

Q4.What is the fundamental difference between tensor.detach()\texttt{tensor.detach()} and with torch.no_grad(): \texttt{with torch.no\_grad():\,}?Show answer ▾

Answer: detach()\texttt{detach()} returns a new tensor disconnected from the computation graph, while no_grad()\texttt{no\_grad()} is a context manager that temporarily disables automatic gradient tracking for all operations within its scope.

These are complementary mechanisms: detach()\texttt{detach()} breaks the graph locally for a single tensor (useful when you want to reuse values without backprop), while no_grad()\texttt{no\_grad()} disables recording globally for efficiency during inference or when updating parameters. Understanding both is essential for controlling autograd.

Q5.Why does ReLU mitigate the vanishing gradient problem in deep neural networks, while sigmoid does not?
  • An active ReLU contributes an activation derivative of 1 instead of an additional factor below 1, whereas sigmoid derivatives are at most 0.25 and shrink in saturation
  • ReLU uses a larger learning rate internally, which prevents gradient shrinkage compared to sigmoid
  • ReLU is faster to compute, so the backpropagation pass completes before gradients can vanish
  • ReLU normalizes the input data automatically, whereas sigmoid requires manual normalization to prevent vanishing gradients
Show answer ▾

Answer: An active ReLU contributes an activation derivative of 1 instead of an additional factor below 1, whereas sigmoid derivatives are at most 0.25 and shrink in saturation

Sigmoid's derivative peaks at 0.25 and approaches zero in its saturated tails, so repeated activation factors can strongly attenuate blame. An active ReLU contributes a factor of 1 at that gate. The complete gradient can still shrink, grow, or cancel through weight matrices and other operations; ReLU removes one common source of attenuation rather than guaranteeing unchanged flow through the network.

Q6.In batch normalization, why are learnable scale and shift parameters (gamma and beta) necessary after normalizing pre-activation values to mean zero and unit variance?
  • They allow the network to undo the normalization and recover the original signal distribution when a standard normal is not optimal for the next layer
  • They are required for numerical stability and prevent overflow errors during the forward pass
  • They enable batch normalization to work with non-convex loss functions
  • They reduce the computational cost of the normalization operation
Show answer ▾

Answer: They allow the network to undo the normalization and recover the original signal distribution when a standard normal is not optimal for the next layer

The lecturer notes a counterintuitive point: forcing all activations into a 'rigid structure' of standard normal distribution may limit the network's representation power, since that distribution may not be optimal for the next layer. The scale parameter (gamma) and shift parameter (beta) let the network learn to adjust the normalized values to whatever mean and variance is best for the downstream computation. In the worst case, the network can learn gamma and beta to recover the original distribution entirely.

Module 1
Module 3