- Inspect gradient flow; compare networks with/without BatchNorm
- Implement a ResNet block; reproduce a mini-ResNet training
- Fine-tune a pretrained model; summarize efficiency trade-offs (e.g., MobileNet)
Course book · primary reading
Alternative reading from Dive into Deep Learning (D2L):
The Colab notebook contains the lecture code for Module 5 (advanced CNN architectures and transfer learning). Use it to follow along the in-class demos.
Try each question before revealing the answer — these mirror the ideas the module quiz checks.
Q1.In VGG networks, why are multiple 3×3 convolutions preferred over a single larger kernel?- They have the same receptive field with fewer parameters and more non-linearities
- 3×3 is the optimal kernel size for all vision tasks
- Larger kernels don't work on GPUs
- They eliminate the need for pooling layers
Show answer ▾
Answer: They have the same receptive field with fewer parameters and more non-linearities
Two 3×3 convolutions create a 5×5 receptive field with 18 parameters vs. 25 for one 5×5, plus two activation functions instead of one—this is VGG's key insight for building deeper, more efficient networks.
Q2.What is the fundamental equation for a residual block's output?
Show answer ▾
Answer:
The identity mapping allows networks to learn residuals rather than full functions, making it easy to learn identity mappings and enabling training of extremely deep networks.
Q3.Why do skip connections in ResNet usually make very deep networks easier to optimize?- They reduce the number of parameters
- They add an identity route for activations and gradients and make the identity mapping easy to represent
- They eliminate the need for batch normalization
- They automatically tune learning rates
Show answer ▾
Answer: They add an identity route for activations and gradients and make the identity mapping easy to represent
A residual block has Jacobian , so incoming blame has a direct additive identity term as well as the learned branch. This strongly improves gradient flow and makes doing nothing as simple as learning , although it is not a mathematical guarantee against every cancellation or poorly conditioned network.
Q4.In which small-data regime is transfer learning most likely to beat a well-tuned scratch model?- Whenever the target dataset is small, regardless of task
- When labels are scarce, the task needs rich features, and source coverage and input scale match the target
- Only when the pretrained model has more parameters
- Whenever the target images are grayscale
Show answer ▾
Answer: When labels are scarce, the task needs rich features, and source coverage and input scale match the target
Scarce labels alone are not enough. Transfer pays when the source model learned features the target genuinely needs and those features survive the domain and resolution change. At tiny 28-pixel scale, the course experiment found scratch tied or beat imported features; that honest null result is part of the decision rule.
Q5.In MobileNets, how does depthwise-separable convolution reduce parameters compared to standard convolution?- By applying 1×1 convolutions before spatial convolutions to reduce channel dimensions first
- By performing spatial convolution separately on each input channel, then mixing channels separately using 1×1 convolutions
- By replacing all convolutions with fully connected layers
- By removing batch normalization between convolution layers
Show answer ▾
Answer: By performing spatial convolution separately on each input channel, then mixing channels separately using 1×1 convolutions
Depthwise-separable convolution splits standard convolution into two steps: (1) apply a K×K filter independently to each input channel, then (2) mix channels with 1×1 convolutions. Its per-layer parameter ratio relative to a dense convolution is roughly ; summing those savings across a network can be substantial, but the savings do not grow exponentially with depth.