9 Modern CNNs and Transfer Learning

Chapter 8 ended with a working machine and a report card. LeNet reads garments at 82.5% with 61,706 parameters; graceful under shift where the MLP cliff-dived. But the card has two demerits we wrote down explicitly: 96% of those parameters sit in the dense head rather than in the convolutions that do the seeing, and the shift cliff was softened, not abolished. And one IOU: the live session’s batch normalization, promised for this chapter.

What happened after LeNet is a decade of architecture research, and it can drown you in named networks. The lecture compresses it into four design questions, and we will let them drive:

Why \(5 \times 5\)? Could smaller filters do the same job with fewer parameters?
How do we go deeper without gradients dying on the way down?
Where do the parameters actually live, and how do we evict them?
Can we design reusable blocks: optimized layer-combinations we stack, keeping a bird’s-eye view instead of re-designing what already works?

Each question has a famous answer (VGG, ResNet, NiN’s \(1\times1\) + global pooling, and the block habit itself), and each answer is a constraint-shaped idea we can test on our own data. At the end, the practical superpower the lecture builds to, reusing a pretrained backbone, gets the most honest experiment in this book.

Code: imports, data, and the shared training recipe

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
from collections.abc import Callable

torch.manual_seed(6050)
train = torch.load("../../data/fashion-train.pt")
test = torch.load("../../data/fashion-test.pt")
X_tr = train["X"].float().unsqueeze(1) / 255.0    # (1200, 1, 28, 28)
X_te = test["X"].float().unsqueeze(1) / 255.0     # (600, 1, 28, 28)
y_tr, y_te, classes = train["y"], test["y"], train["classes"]

def train_existing(
    model: nn.Module, opt: torch.optim.Optimizer, epochs: int,
    batch: int = 128,
    data: tuple[torch.Tensor, torch.Tensor] | None = None,
) -> nn.Module:
    X, y = data if data is not None else (X_tr, y_tr)
    n = len(X)
    for _ in range(epochs):
        perm = torch.randperm(n)
        for i in range(0, n, batch):
            idx = perm[i:i + batch]
            model.train()
            loss = F.cross_entropy(model(X[idx]), y[idx])
            opt.zero_grad(); loss.backward(); opt.step()
    model.eval()
    return model

def train_model(model_fn: Callable[[], nn.Module], epochs: int,
                batch: int = 128, seed: int = 6050, lr: float = 1e-3,
                data: tuple[torch.Tensor, torch.Tensor] | None = None) -> nn.Module:
    torch.manual_seed(seed)
    model = model_fn()
    opt = torch.optim.Adam(model.parameters(), lr=lr)
    return train_existing(model, opt, epochs, batch, data)

@torch.no_grad()
def accuracy(model: nn.Module, X: torch.Tensor, y: torch.Tensor) -> float:
    model.eval()
    return (model(X).argmax(1) == y).float().mean().item()

def count(model: nn.Module) -> int:
    return sum(p.numel() for p in model.parameters())

One evaluation label needs an honesty note. Chapter 8 already opened the 600-image holdout, and this chapter queries it repeatedly while comparing architectures. We keep the familiar X_te name, but treat it as the book’s fixed benchmark. These comparisons are descriptive, not an unbiased estimate after model selection; a final claim would require a fresh, untouched test set.

Two small novelties in the recipe, both because this chapter’s networks contain batch normalization: model.train() before each step and model.eval() before each evaluation. Why that matters is the subject of the second section.

9.1 Question 1: why \(5 \times 5\)? Stack small kernels instead

VGG’s answer (Simonyan & Zisserman, 2014) is the cleanest idea in this chapter. Recall the receptive-field ledger of Chapter 8: stacking grows the field. Two stacked \(3 \times 3\) convolutions let the second layer see \(5 \times 5\) of the input, the same receptive field as one \(5 \times 5\) kernel. But compare the bills, for a layer with \(C\) channels in and \(C\) out:

\[ \underbrace{25\,C^2}_{\text{one } 5\times5} \qquad\text{versus}\qquad \underbrace{2 \times 9\,C^2 = 18\,C^2}_{\text{two } 3\times3\text{s}}, \]

a 28% saving. The stacked pair also fires a ReLU twice where the big kernel fires once. Same sight, fewer parameters, more nonlinearity. Three \(3\times3\)s reach a \(7\times7\) field for \(27C^2\) against \(49C^2\): the deeper you take the idea, the better the deal gets.

Code: the parameter bill, C = 32

C = 32
one_5x5 = nn.Conv2d(C, C, 5, padding=2)
two_3x3 = nn.Sequential(nn.Conv2d(C, C, 3, padding=1), nn.ReLU(),
                        nn.Conv2d(C, C, 3, padding=1))
print(f"one 5x5:  {count(one_5x5):,} parameters, 1 ReLU after")
print(f"two 3x3s: {count(two_3x3):,} parameters, 2 ReLUs")

one 5x5:  25,632 parameters, 1 ReLU after
two 3x3s: 18,496 parameters, 2 ReLUs

One honest caveat before you re-derive the field equations of vision from this: the \(18C^2 < 25C^2\) arithmetic assumes the channel width stays \(C\) through the stack. When a layer grows channels (as LeNet’s \(6 \rightarrow 16\) did), splitting it into two growing \(3\times3\)s can cost more, not less. VGG’s design sidesteps this by keeping width constant inside each block and changing it only between blocks, which is exactly the shape our code will take. (Exercise 1 makes you find the break-even point.)

9.2 The stabilizer we owe you: batch normalization

Before we stack deeper than ever, we need the tool Chapter 8’s live session used and we deferred. Here is the problem it solves. Watch what happens to the scale of activations as a signal crosses twelve freshly initialized \(3\times3\) conv layers:

Code: activation scale through 12 conv layers, with and without BN

def stack12(with_bn: bool, ch: int = 16) -> nn.Sequential:
    layers = [nn.Conv2d(1, ch, 3, padding=1)]
    for _ in range(11):
        if with_bn:
            layers.append(nn.BatchNorm2d(ch))
        layers += [nn.ReLU(), nn.Conv2d(ch, ch, 3, padding=1)]
    return nn.Sequential(*layers)

torch.manual_seed(0)
x = X_tr[:256]
plain_stack = stack12(False)
bn_stack = stack12(True)
plain_convs = [m for m in plain_stack if isinstance(m, nn.Conv2d)]
bn_convs = [m for m in bn_stack if isinstance(m, nn.Conv2d)]
for source, target in zip(plain_convs, bn_convs):
    target.load_state_dict(source.state_dict())       # identical convolution weights

for tag, network in [("without BN", plain_stack), ("with BN   ", bn_stack)]:
    h, stds = x, []
    with torch.no_grad():
        for layer in network:
            h = layer(h)
            if isinstance(layer, nn.Conv2d):
                stds.append(h.std().item())
    print(f"{tag} layer stds: " + "  ".join(f"{s:.3f}" for s in stds[::2]))

without BN layer stds: 0.344  0.059  0.043  0.046  0.049  0.061
with BN    layer stds: 0.344  0.404  0.395  0.385  0.432  0.384

Without normalization the signal’s spread collapses by an order of magnitude within a few layers and keeps sagging. Forward activation scales and backward gradients are not the same quantity, but both are shaped by the stack’s weights and nonlinearities; poorly scaled activations are therefore a warning to inspect gradient flow directly, as we do below.

Batch normalization (Ioffe & Szegedy, 2015) re-standardizes at every layer. For each channel of an NCHW tensor, BatchNorm2d computes the mean and variance across the current minibatch and both spatial axes (\(N\), \(H\), and \(W\)), normalizes the activations, then lets the layer undo that transformation through two learnable knobs:

\[ \hat{x} = \frac{x - \mu_{\text{batch}}}{\sqrt{\sigma^2_{\text{batch}} + \epsilon}}, \qquad y = \gamma\,\hat{x} + \beta . \tag{9.1}\]

The \(\gamma, \beta\) pair matters: normalization is a reset to a healthy scale, not a straitjacket. The network can learn to restore any mean and spread that helps, but it starts every layer from sane numbers. In the printout above, the BN column holds near a constant spread through all twelve layers: every layer trains from day one. The standard placement, which the live session uses and we adopt, is conv \(\rightarrow\) BN \(\rightarrow\) ReLU, with bias=False on the convolution, since \(\beta\) already provides the shift.

BN is two different machines: tell PyTorch which one you’re running

At training time BN normalizes by the current batch’s statistics. At evaluation time there may be no batch (one image!), so it uses running averages collected during training. model.train() and model.eval() switch between the two. Forgetting the switch is the classic BN bug: evaluate in train mode and your predictions depend on whatever else happens to be in the batch; train in eval mode and BN never learns its statistics. Our train_model recipe flips the switch in both directions; look for it. Corollary: BN needs real batches to estimate statistics, so it gets unreliable at tiny batch sizes.

Normalize across what? A seed for Part IV

Batch statistics are one choice among several. When we build transformers (Chapter 14), batches of variable-length sequences will make per-batch statistics awkward, and the same standardize-then-rescale idea will reappear computed per token instead: layer normalization. Same equation, different axis. Remember Equation 9.1 when you meet it.

9.3 Building with blocks: a small VGG

Now assemble Question 1’s answer with Question 4’s habit. Define the atom — conv, BN, ReLU — once; a block stacks atoms and pools; a network stacks blocks. This is the lecture’s point about VGG versus its hand-tuned predecessors: you stop re-designing and start repeating.

Code: VGGSmall — two blocks of stacked 3×3 atoms

def atom(c_in: int, c_out: int) -> list[nn.Module]:
    return [nn.Conv2d(c_in, c_out, 3, padding=1, bias=False),
            nn.BatchNorm2d(c_out), nn.ReLU()]

class VGGSmall(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            *atom(1, 16), *atom(16, 16), nn.MaxPool2d(2),    # -> (16, 14, 14)
            *atom(16, 32), *atom(32, 32), nn.MaxPool2d(2),   # -> (32, 7, 7)
        )
        self.head = nn.Sequential(nn.Flatten(),
                                  nn.Linear(32 * 7 * 7, 128), nn.ReLU(),
                                  nn.Linear(128, 10))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.features(x))

vgg = train_model(VGGSmall, epochs=60)
n_head = count(vgg.head)
print(f"VGGSmall: {count(vgg):,} parameters ({n_head:,} in the head, "
      f"{n_head / count(vgg):.0%})")
print(f"test accuracy {accuracy(vgg, X_te, y_te):.1%}")

VGGSmall: 218,586 parameters (202,122 in the head, 92%)
test accuracy 86.7%

Two readings again. The good: this VGG-style recipe reaches 86.7%, four points above LeNet in this run. Because depth, width, normalization, kernel sizes, parameter count, and training duration all changed together, this comparison does not isolate which ingredient earned the gain. The bad: 218,586 parameters, triple LeNet’s total, and the head’s share got worse — 92% of the network is a dense layer reading a flattened grid. The convolutional diet succeeded and the total got fatter anyway. Which forces the lecture’s third question: where do the parameters live, and — his phrasing — where can we do the most damage to the count?

9.4 Question 3: \(1 \times 1\) convolutions, and firing the flatten head

The bloat has one address: nn.Linear(1568, 128). To evict it we need one more tool, odd-looking at first sight: a convolution whose kernel is \(1 \times 1\).

A \(1 \times 1\) kernel does no spatial mixing at all — it looks at a single pixel. But across channels it does exactly what Equation 8.1 says: at each position, take the \(C_{\text{in}}\) channel values and form \(C_{\text{out}}\) weighted combinations plus bias. Pause on what that is: a linear model across the channels, run separately at every pixel — Chapter 1’s machine, miniaturized and stamped across the image. Add a ReLU and each pixel is running a tiny MLP over its own feature vector. The lecture’s summary: we summarize channels, we don’t discard them — compressing 64 feature maps into 32 loses far less than pooling them away, and each compression injects another nonlinearity almost for free.

That tool enables the Network-in-Network design (Lin et al., 2013), and with it the clean head. A NiN block is one \(3\times3\) (some spatial mixing) followed by two \(1\times1\)s (per-pixel channel mixing). And the classifier becomes:

Let the last block produce a stack of feature maps.
Global average pooling (GAP): average each map over all positions — 64 maps in, 64 numbers out. No parameters at all.
One small linear layer to the logits.

Code: NINSmall — architecture and first 75 epochs

def nin_block(c_in: int, c_out: int) -> list[nn.Module]:
    return [nn.Conv2d(c_in, c_out, 3, padding=1, bias=False),
            nn.BatchNorm2d(c_out), nn.ReLU(),
            nn.Conv2d(c_out, c_out, 1), nn.ReLU(),
            nn.Conv2d(c_out, c_out, 1), nn.ReLU()]

class NINSmall(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            *nin_block(1, 16), nn.MaxPool2d(2),
            *nin_block(16, 32), nn.MaxPool2d(2),
            *nin_block(32, 64),
        )
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(64, 10))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.features(x))

torch.manual_seed(6050)
nin = NINSmall()
nin_opt = torch.optim.Adam(nin.parameters(), lr=1e-3)
nin = train_existing(nin, nin_opt, epochs=75)

Code: NINSmall — continue the same optimizer through epoch 150

nin = train_existing(nin, nin_opt, epochs=75)
print(f"NINSmall: {count(nin):,} parameters "
      f"(head: {count(nin.head):,})")
print(f"test accuracy {accuracy(nin, X_te, y_te):.1%}")

NINSmall: 35,034 parameters (head: 650)
test accuracy 76.2%

The parameter story is a rout: 35,034 total, a 650-parameter head, six times smaller than VGGSmall. The accuracy is 76.2%, and that dip is worth more attention than the win, so let us not rush past it. First, though, collect the promised payoff. Chapter 8 said the remaining shift-cliff was the flatten head’s fault: it reads the final grid positionally. GAP averages over positions, so the head no longer assigns a different weight to every location. The rematch of the rematch:

Code: shift curves, flatten head vs GAP head

class LeNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, 5, padding=2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(400, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = F.max_pool2d(F.relu(self.conv1(x)), 2)
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.flatten(1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.fc3(x)

lenet = train_model(LeNet, epochs=150)          # Chapter 8's model, same recipe

def shift_right(X: torch.Tensor, px: int) -> torch.Tensor:
    if px == 0:
        return X.clone()
    out = torch.zeros_like(X)
    out[..., px:] = X[..., :-px]
    return out

shifts = list(range(5))
acc_lenet = [accuracy(lenet, shift_right(X_te, s), y_te) for s in shifts]
acc_nin = [accuracy(nin, shift_right(X_te, s), y_te) for s in shifts]
for s in shifts:
    print(f"shift {s}px:   LeNet {acc_lenet[s]:.1%}   NIN+GAP {acc_nin[s]:.1%}")

plt.figure(figsize=(5.5, 3.2))
plt.plot(shifts, acc_lenet, "s-", color="#232D4B", lw=2, ms=7, label="LeNet (flatten head)")
plt.plot(shifts, acc_nin, "^-", color="#E57200", lw=2, ms=7, label="NINSmall (GAP head)")
plt.axhline(0.1, ls=":", color="#B8B8A8")
plt.xlabel("shift (pixels right)"); plt.ylabel("test accuracy")
plt.ylim(0, 0.9); plt.xticks(shifts); plt.legend()
plt.tight_layout(); plt.show()

shift 0px:   LeNet 82.5%   NIN+GAP 76.2%
shift 1px:   LeNet 74.8%   NIN+GAP 70.3%
shift 2px:   LeNet 62.3%   NIN+GAP 69.3%
shift 3px:   LeNet 45.3%   NIN+GAP 67.5%
shift 4px:   LeNet 26.8%   NIN+GAP 66.5%

Figure 9.1: Chapter 8’s experiment, third round. LeNet’s flatten head still slides below 30% by four pixels of shift. NINSmall with its global-average-pool head barely tilts: per-channel averages change much less when features move. GAP removes position-specific weights from the head, but padding, stride, pooling, and content clipped at the image edge keep the complete network from being exactly shift-invariant.

The curve that fell from 82% to 27% across this book’s Part II is now a gentle slope from 76% to 67%. Position-specific weights are gone from NINSmall’s head, and the result is consistent with the promised robustness benefit. It is not a matched-head ablation: the trunks and training recipes differ too. The complete CNN is only approximately shift-tolerant; boundaries, strides, and pooling can still change its features.

Now the dip. On clean, centered test garments NINSmall trails LeNet by six points. That result is consistent with Chapter 8’s warning: on centered data, position can carry information, and a GAP head cannot assign separate weights to separate locations. But GAP is not the only changed ingredient, so this run does not prove it caused the gap. With the full 60,000-image dataset the lecture’s NiN reaches about 90%, near its LeNet. At our scale the comparison exposes a hypothesis worth testing with a matched-head ablation: shift tolerance may be bought with positional information.

Question 4’s other famous block: Inception, in one breath

GoogLeNet’s Inception block (2014) answers “which kernel size?” with “all of them”: parallel \(1\times1\), \(3\times3\), \(5\times5\), and pooling branches, concatenated. Its enabling trick is the \(1\times1\) bottleneck: compress channels before the expensive spatial kernels. The seed’s arithmetic for one \(5\times5\) branch at 256 channels: direct, \(5^2 \times 256 \times 128 \approx 819\)k parameters; with a \(1\times1\) squeeze to 32 first, \(256 \times 32 + 5^2 \times 32 \times 128 \approx 111\)k — an 86% cut for the same nominal operation. We won’t build Inception here (the principle, channel compression before spatial expense, is the transferable part), but Exercise 2 walks the arithmetic and the course assignment has you build the block itself.

9.5 Question 2: going deeper, and the wall you hit

VGG says depth is the direction. So push it: stack twenty of our two-conv blocks (with BN, per the recipe, on a \(14\times14\) grid so the experiment fits a laptop) and compare against the identical stack with one change we’ll reveal after the numbers.

Code: train the plain 20-block network

class Block(nn.Module):
    def __init__(self, ch: int, residual: bool):
        super().__init__()
        self.c1 = nn.Conv2d(ch, ch, 3, padding=1, bias=False)
        self.b1 = nn.BatchNorm2d(ch)
        self.c2 = nn.Conv2d(ch, ch, 3, padding=1, bias=False)
        self.b2 = nn.BatchNorm2d(ch)
        self.residual = residual
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        y = self.b2(self.c2(F.relu(self.b1(self.c1(x)))))
        return F.relu(y + x) if self.residual else F.relu(y)

class DeepNet(nn.Module):
    def __init__(self, nblocks: int, residual: bool, ch: int = 12):
        super().__init__()
        self.stem = nn.Conv2d(1, ch, 3, padding=1)
        self.pool = nn.MaxPool2d(2)                       # 28 -> 14 early
        self.blocks = nn.Sequential(*[Block(ch, residual)
                                      for _ in range(nblocks)])
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(ch, 10))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.blocks(self.pool(F.relu(self.stem(x)))))

plain = train_model(lambda: DeepNet(20, residual=False), epochs=30)
print(f"plain-20     train {accuracy(plain, X_tr, y_tr):.1%}   "
      f"test {accuracy(plain, X_te, y_te):.1%}")

plain-20     train 60.8%   test 51.0%

Code: train the matched residual 20-block network

resid = train_model(lambda: DeepNet(20, residual=True), epochs=30)
print(f"residual-20  train {accuracy(resid, X_tr, y_tr):.1%}   "
      f"test {accuracy(resid, X_te, y_te):.1%}")

residual-20  train 100.0%   test 77.8%

Look at the training column first, because it carries the whole lesson. The plain 40-conv-layer network cannot even fit the 1,200 images it sees every epoch: 61% train accuracy, while its residual twin memorizes them and generalizes twenty-seven points better. This is an optimization failure in the same family as the degradation problem He et al. reported in 2015, where their 56-layer plain net trained worse than their 20-layer one. Our experiment isolates the residual rescue at one depth; a strict demonstration that adding depth degrades a plain network would also require a shallower plain control.

The one change: each block computes \(F(x)\) and outputs \(F(x) + x\). A residual connection lets the input skip over the block and adds it back.

\[ H(x) = F(x) + x \qquad\Longrightarrow\qquad \frac{\partial L}{\partial x} = \frac{\partial L}{\partial H}\left(\frac{\partial F}{\partial x} + I\right). \tag{9.2}\]

Code: draw the residual stream and identity shortcut

from matplotlib.patches import FancyBboxPatch, Circle

fig, ax = plt.subplots(figsize=(7.2, 2.5))
ax.set_xlim(0, 10); ax.set_ylim(0, 4); ax.axis("off")

def box(x: float, label: str, color: str) -> None:
    patch = FancyBboxPatch((x, 1.25), 1.65, 1.0,
                           boxstyle="round,pad=0.08", facecolor=color,
                           edgecolor="#232D4B", linewidth=1.3)
    ax.add_patch(patch)
    ax.text(x + 0.825, 1.75, label, ha="center", va="center", fontsize=9)

ax.text(0.35, 1.75, "$x$", ha="center", va="center", fontsize=12)
box(1.2, "conv–BN–ReLU", "#E8F0F8")
box(3.55, "conv–BN", "#E8F0F8")
add = Circle((6.45, 1.75), 0.28, facecolor="#F9DCBF",
             edgecolor="#E57200", linewidth=1.5)
ax.add_patch(add); ax.text(6.45, 1.75, "+", ha="center", va="center", fontsize=14)
box(7.25, "ReLU", "#F4F4EF")
ax.text(9.55, 1.75, "$H(x)$", ha="center", va="center", fontsize=12)

for start, end in [((0.55, 1.75), (1.2, 1.75)),
                   ((2.85, 1.75), (3.55, 1.75)),
                   ((5.2, 1.75), (6.15, 1.75)),
                   ((6.73, 1.75), (7.25, 1.75)),
                   ((8.9, 1.75), (9.3, 1.75))]:
    ax.annotate("", xy=end, xytext=start,
                arrowprops=dict(arrowstyle="->", color="#232D4B", lw=1.5))
ax.plot([0.55, 0.8, 0.8, 6.45, 6.45], [1.75, 1.75, 3.2, 3.2, 2.05],
        color="#E57200", lw=2)
ax.annotate("", xy=(6.45, 2.05), xytext=(6.45, 2.55),
            arrowprops=dict(arrowstyle="->", color="#E57200", lw=2))
ax.text(3.65, 3.45, "identity shortcut: $x$", color="#B45309",
        ha="center", fontsize=10)
ax.text(4.35, 0.75, "learned correction $F(x)$", color="#232D4B",
        ha="center", fontsize=10)
plt.tight_layout(); plt.show()

Figure 9.2: A residual block has two routes. The learned branch computes a correction \(F(x)\); the identity branch carries \(x\) unchanged to the addition. Even when the learned branch has a difficult Jacobian, the shortcut leaves a direct route for activations and gradients.

Read the right-hand side with Chapter 5 eyes. Blame arriving at a block’s output reaches its input through two additive terms: the learned branch \(\partial F /\partial x\) and the identity \(I\). The identity contribution gives gradient flow a direct route that does not itself multiply by the learned weights. It is powerful, not magical: the learned Jacobian can still reinforce, distort, or even partly cancel that term. Chapter 5 called ReLU’s open gate a gradient superhighway through a single unit; a skip connection builds a direct lane into every block. The block only needs to learn the residual, the correction on top of “pass it through,” and doing nothing is easy to represent: \(F = 0\).

Here is the highway visible at initialization, extending Chapter 5’s depth experiment from dense layers to conv stacks:

Code: gradient at the stem vs. depth, three designs

class PlainNoBN(nn.Module):
    def __init__(self, nconvs: int, ch: int = 12):
        super().__init__()
        self.stem = nn.Conv2d(1, ch, 3, padding=1)
        self.pool = nn.MaxPool2d(2)
        self.layers = nn.Sequential(*[m for _ in range(nconvs)
                                      for m in (nn.Conv2d(ch, ch, 3, padding=1),
                                                nn.ReLU())])
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(ch, 10))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.layers(self.pool(F.relu(self.stem(x)))))

def stem_grad(make_net: Callable[[], nn.Module]) -> tuple[float, float, int, int]:
    torch.manual_seed(0)
    net = make_net()
    F.cross_entropy(net(X_tr[:128]), y_tr[:128]).backward()
    grad = net.stem.weight.grad
    return (grad.double().norm().item(), grad.abs().max().item(),
            torch.count_nonzero(grad).item(), grad.numel())

depths = [4, 12, 24]                                  # blocks (2 convs each)
diagnostics = {
    "plain, no BN": [stem_grad(lambda d=d: PlainNoBN(2 * d)) for d in depths],
    "plain + BN": [stem_grad(lambda d=d: DeepNet(d, False)) for d in depths],
    "residual + BN": [stem_grad(lambda d=d: DeepNet(d, True)) for d in depths],
}
curves = {name: [row[0] for row in rows]
          for name, rows in diagnostics.items()}
for name, ys in curves.items():
    print(f"{name:14s}: " + "   ".join(f"{v:.1e}" for v in ys))
norm64, maxabs, nonzero, total = diagnostics["plain, no BN"][-1]
print(f"48-layer no-BN check: max |component| {maxabs:.1e}; "
      f"nonzero {nonzero}/{total}; float64 norm {norm64:.1e}")

plt.figure(figsize=(6.2, 3.4))
for (name, ys), color in zip(curves.items(),
                             ["#B8B8A8", "#E57200", "#232D4B"]):
    xs = [2 * d for d in depths]
    plt.semilogy(xs, ys, "o-", ms=5, color=color, label=name)
plt.xlabel("conv layers"); plt.ylabel(r"$\|\partial L / \partial W^{(\mathrm{stem})}\|$")
plt.legend(); plt.tight_layout(); plt.show()

plain, no BN  : 9.0e-06   3.3e-13   2.2e-23
plain + BN    : 7.6e-02   5.0e-01   8.3e+01
residual + BN : 1.6e-01   4.9e-01   5.0e-01
48-layer no-BN check: max |component| 5.2e-24; nonzero 108/108; float64 norm 2.2e-23

Figure 9.3: Gradient magnitude reaching the stem (first layer) at initialization, versus depth, for three designs. In the 48-layer no-BN stack, the true norm is 2.2e-23; a float32 norm calculation reports zero because squaring these nonzero components underflows. BN fixes the vanishing but overshoots: in plain stacks the gradient grows with depth. Residual blocks keep the gradient within one order of magnitude across these depths.

The most important seed this chapter plants

Residual connections are not only a CNN trick. They became a central design device in many deep architectures; when we assemble a transformer in Chapter 14, every attention layer and every feed-forward layer will be wrapped as \(x + F(x)\), and the freshly-met layer normalization will sit beside each skip. The picture to carry forward: a residual stream with direct additive routes through the network, each block reading from it and writing a correction back. Those routes improve conditioning without making the stream immune to learned-branch interactions. Attention will be one kind of correction. You now own both parts of that skeleton.

9.6 The scorecard: what a decade of design bought

The lecture closes its architecture tour with a scatter worth reproducing: every model of Part II on one canvas, accuracy against parameters, each trained to convergence under our shared recipe:

Code: accuracy vs. parameters scatter

models_ = {
    "LeNet": (lenet, "#B8B8A8"),
    "VGGSmall": (vgg, "#E57200"),
    "NIN+GAP": (nin, "#232D4B"),
    "plain-20": (plain, "#DDA0DD"),
    "residual-20": (resid, "#5379AA"),
}
plt.figure(figsize=(6.2, 3.6))
for name, (m, color) in models_.items():
    p, a = count(m), accuracy(m, X_te, y_te)
    plt.scatter(p, a, s=70, color=color, zorder=3)
    plt.annotate(name, (p, a), textcoords="offset points",
                 xytext=(8, -3), fontsize=9)
plt.xscale("log")
plt.xlabel("parameters (log scale)"); plt.ylabel("test accuracy")
plt.ylim(0.45, 0.95)
plt.tight_layout(); plt.show()

Figure 9.4: Part II’s models on one canvas (our 1,200-image subset; each model trained to convergence with the shared Adam recipe). VGG-style depth buys accuracy but hoards parameters in its flatten head; NiN+GAP evicts the head at a visible cost in clean accuracy at this data scale; residual blocks let a 53,050-parameter network go 40 layers deep. The lecture’s full-data frontier, where the models exceed 89%, needs data our subset does not have; the shape of the trade-offs is already visible.

9.7 Transfer learning: the mechanics, and an honest experiment

Everything so far trains from scratch. The lecture’s final move is the one that defines practice today: most image features are shared — edges, textures, parts transfer across tasks — so train one strong backbone once, on enormous data, and reuse it everywhere. The mechanics come in two strengths:

Linear probe: freeze the pretrained trunk entirely (requires_grad = False), extract its features, train only a new linear head on your labels.
Fine-tuning: additionally unfreeze the last block or two, training them at a much smaller learning rate than the fresh head, so the pretrained features adapt gently instead of being trampled.

We will test the idea properly. The task: classify the three shoe classes (sandal, sneaker, ankle boot) from ten labeled examples each. The 30-image regime is exactly where transfer should shine — too few labels to learn features, goes the story, so imported features should dominate. Three contenders:

From scratch: our VGG-style trunk trained on the 30 images alone.
Our own pretrained trunk: the same trunk pretrained on the seven non-shoe classes (863 images), frozen, linear probe.
A real pretrained backbone: SqueezeNet 1.1 trained on ImageNet — 1.2 million photographs, 1,000 classes — loaded from weights committed with this book, frozen, linear probe on its 512-dimensional features. (Fashion images get upsampled to 96×96 and repeated to three channels to fit its expectations. That awkwardness is part of the experiment, and part of the lesson.)

Code: the shoe task, our own trunk, and the frozen probes

from torchvision.models import squeezenet1_1

shoe = torch.tensor([5, 7, 9])
new_label = {5: 0, 7: 1, 9: 2}
is_shoe_tr = (y_tr[:, None] == shoe).any(1)
is_shoe_te = (y_te[:, None] == shoe).any(1)
X_shoe_te = X_te[is_shoe_te]
y_shoe_te = torch.tensor([new_label[int(c)] for c in y_te[is_shoe_te]])

# our own trunk: pretrain on the 7 non-shoe classes
src = [0, 1, 2, 3, 4, 6, 8]
remap = {c: i for i, c in enumerate(src)}
X_src = X_tr[~is_shoe_tr]
y_src = torch.tensor([remap[int(c)] for c in y_tr[~is_shoe_tr]])

class Trunk(nn.Module):
    def __init__(self, n_classes: int):
        super().__init__()
        self.features = nn.Sequential(
            *atom(1, 16), *atom(16, 16), nn.MaxPool2d(2),
            *atom(16, 32), *atom(32, 32), nn.MaxPool2d(2), *atom(32, 64))
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(64, n_classes))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.features(x))

own_trunk = train_model(lambda: Trunk(7), epochs=100, data=(X_src, y_src))
X_src_te = X_te[~is_shoe_te]
y_src_te = torch.tensor([remap[int(c)] for c in y_te[~is_shoe_te]])
print(f"own trunk, source task (7 classes): "
      f"{accuracy(own_trunk, X_src_te, y_src_te):.1%}")

# the real thing: ImageNet SqueezeNet, from committed weights (no download)
sq = squeezenet1_1(weights=None)
sq.load_state_dict(torch.load("../../data/squeezenet1_1-imagenet.pt"))
sq.eval()
for p in sq.parameters():
    p.requires_grad = False

IMNET_MEAN = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
IMNET_STD = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)

@torch.no_grad()
def squeeze_features(X: torch.Tensor) -> torch.Tensor:  # (N,1,28,28) -> (N,512)
    x = X.repeat(1, 3, 1, 1)
    x = F.interpolate(x, size=96, mode="bilinear", align_corners=False)
    f = sq.features((x - IMNET_MEAN) / IMNET_STD)
    return F.adaptive_avg_pool2d(f, 1).flatten(1)

@torch.no_grad()
def own_features(X: torch.Tensor) -> torch.Tensor:      # (N,1,28,28) -> (N,64)
    return F.adaptive_avg_pool2d(own_trunk.features(X), 1).flatten(1)

def linear_probe(Z_tr: torch.Tensor, y_f: torch.Tensor,
                 Z_te: torch.Tensor, seed: int) -> float:
    torch.manual_seed(seed)
    head = nn.Linear(Z_tr.shape[1], 3)
    opt = torch.optim.Adam(head.parameters(), lr=1e-2,
                           weight_decay=1e-3)   # built-in L2 penalty (ch. 1's ridge)
    for _ in range(400):
        loss = F.cross_entropy(head(Z_tr), y_f)
        opt.zero_grad(); loss.backward(); opt.step()
    return (head(Z_te).argmax(1) == y_shoe_te).float().mean().item()

own trunk, source task (7 classes): 79.7%

Code: scratch vs. own trunk vs. ImageNet trunk, three seeds

K = 10
rows = []
for seed in [0, 1, 6050]:
    torch.manual_seed(seed)
    idx = torch.cat([(y_tr == c).nonzero().squeeze(1)
                     [torch.randperm(int((y_tr == c).sum()))[:K]] for c in shoe])
    X_few = X_tr[idx]
    y_few = torch.tensor([new_label[int(c)] for c in y_tr[idx]])

    scratch = train_model(lambda: Trunk(3), epochs=100, seed=seed,
                          data=(X_few, y_few))
    rows.append((accuracy(scratch, X_shoe_te, y_shoe_te),
                 linear_probe(own_features(X_few), y_few,
                              own_features(X_shoe_te), seed),
                 linear_probe(squeeze_features(X_few), y_few,
                              squeeze_features(X_shoe_te), seed)))

print("seed     scratch   own-trunk probe   ImageNet probe")
for seed, r in zip([0, 1, 6050], rows):
    print(f"{seed:4d}     {r[0]:.1%}       {r[1]:.1%}           {r[2]:.1%}")
mean = [sum(c) / 3 for c in zip(*rows)]
print(f"mean     {mean[0]:.1%}       {mean[1]:.1%}           {mean[2]:.1%}")

seed     scratch   own-trunk probe   ImageNet probe
   0     86.4%       68.9%           88.1%
   1     85.3%       65.5%           85.9%
6050     88.1%       77.4%           87.6%
mean     86.6%       70.6%           87.2%

Read that table slowly, because it does not say what the textbook story predicts, and the discrepancy is the best lesson in this chapter. Training from scratch on thirty images fights the mighty ImageNet backbone to a dead heat — the means differ by less than a point, well inside seed noise. And our own pretrained trunk, perfectly competent on its source task per the printout above, transfers worse than nothing. Three reasons, each a general principle:

A trunk can only donate what its data taught it. Our own trunk saw 863 shirts, bags, and trousers. Nothing in that curriculum required foot-shaped detectors, so its 64 features flatten sandals, sneakers, and boots into nearly the same point. Pretraining is not magic; it is curriculum.
Domain and resolution gaps tax the donation. SqueezeNet’s filters expect 224-pixel color photographs; we feed it 28-pixel grayscale icons inflated to 96. Its early layers hunt for detail that simply is not there. (The lecture’s recipe — resize and normalize to match the pretraining pipeline — is doing real work; we complied as far as the data allows.)
Transfer wins when the target is big relative to your labels — not small. This is the quiet one. Three shoe silhouettes at \(28 \times 28\) is a small problem: thirty images genuinely suffice, so the scratch baseline is strong and there is little room for imported knowledge to pay rent. The lecture’s numbers show the other regime: linear-probing an ImageNet ResNet-18 on the full FashionMNIST task reaches about 93%, and fine-tuning about 94% — at 224 pixels, with a task rich enough that features matter more than labels.

When to reach for a pretrained backbone

The honest decision rule our experiment and the lecture’s jointly support: transfer pays when (your labels are scarce) and (the target task is feature-hungry) and (the pretraining data plausibly covers the target’s features at a matched scale). At 224 pixels and 18 landmark classes — the course assignment — all three hold, and transfer is the winning move by a wide margin. At 28 pixels and 3 silhouettes, the conditions fail and scratch fights the superpower to a draw. Knowing which regime you are in is the skill; the mechanics (requires_grad, learning-rate splits) are the easy part.

The mechanics, for completeness, since you will use them constantly from Part V onward — fine-tuning SqueezeNet’s last block with the lecture’s two-learning-rate recipe (fresh head fast, pretrained layers slow):

Code: fine-tuning — unfreeze the last block, two learning rates

def prep(X: torch.Tensor) -> torch.Tensor:
    x = X.repeat(1, 3, 1, 1)
    x = F.interpolate(x, size=96, mode="bilinear", align_corners=False)
    return (x - IMNET_MEAN) / IMNET_STD

torch.manual_seed(6050)
idx = torch.cat([(y_tr == c).nonzero().squeeze(1)
                 [torch.randperm(int((y_tr == c).sum()))[:K]] for c in shoe])
X_few, y_few = prep(X_tr[idx]), torch.tensor([new_label[int(c)]
                                              for c in y_tr[idx]])

ft = squeezenet1_1(weights=None)
ft.load_state_dict(torch.load("../../data/squeezenet1_1-imagenet.pt"))
head = nn.Linear(512, 3)
for p in ft.parameters():
    p.requires_grad = False
for p in ft.features[-1].parameters():         # last Fire block only
    p.requires_grad = True

opt = torch.optim.Adam([
    {"params": head.parameters(), "lr": 1e-3},          # fresh head: normal speed
    {"params": ft.features[-1].parameters(), "lr": 1e-4},  # pretrained: gentle
], weight_decay=1e-3)

ft.train()
for _ in range(60):
    z = F.adaptive_avg_pool2d(ft.features(X_few), 1).flatten(1)
    loss = F.cross_entropy(head(z), y_few)
    opt.zero_grad(); loss.backward(); opt.step()
ft.eval()
with torch.no_grad():
    z = F.adaptive_avg_pool2d(ft.features(prep(X_shoe_te)), 1).flatten(1)
    print(f"fine-tuned (last block + head): "
          f"{(head(z).argmax(1) == y_shoe_te).float().mean():.1%}")

fine-tuned (last block + head): 88.7%

Fine-tuning edges the frozen probe on this seed and lands in the same tie with scratch. This is consistent with the regime analysis above: adaptation helps at the margin, but no amount of it makes a small target task need imported features.

Keep the pattern, though: this loop, scaled to real backbones and feature-hungry tasks, is how most of applied deep learning ships today. It is also this book’s destination: Part V is what happened when the field noticed that “pretrain big, adapt cheaply” was not a vision trick but the recipe for language too. The BERT moment (Chapter 15) is this section’s idea, taken seriously at scale.

9.8 Okay, so —

Small kernels, stacked: two \(3\times3\)s see what a \(5\times5\) sees, for \(18C^2\) against \(25C^2\), with twice the nonlinearity — provided width holds constant through the stack. Design in blocks, repeat the block.
BatchNorm re-standardizes every channel over the batch and spatial axes, then hands back two knobs (\(\gamma, \beta\)); conv→BN→ReLU with bias=False; train and eval are different machines, so flip the switch. Its per-token cousin, LayerNorm, awaits in Chapter 14.
\(1\times1\) convolutions run Chapter 1’s linear model across channels at every pixel: summarize channels, don’t discard them. With global average pooling they fire the flatten head: parameters collapse (218k \(\rightarrow\) 35k) and position-specific weights leave the head, at a clean-accuracy price our small dataset makes visible. Boundaries and downsampling still prevent exact invariance.
Depth can hit an optimization wall: our plain 40-layer net couldn’t fit its own training set. The residual connection \(H(x) = F(x) + x\) adds an identity lane to the Jacobian, Chapter 5’s gradient superhighway as infrastructure, and the same-depth twin trains to 100%. Transformer stacks will reuse this residual pattern around each attention and feed-forward sublayer.
Transfer learning = freeze + probe, or gently fine-tune. Our honest experiment found scratch and the ImageNet probe in a near tie across three seeds at 28-pixel scale. Pretraining is curriculum, gaps are taxed, and small targets may leave little room for imports. The lecture’s full-scale runs (93–94% via ImageNet ResNet-18) illustrate a richer regime where transfer pays. Diagnosing the regime is the skill.

Sources and further reading

Simonyan & Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition: the small-kernel, repeated-block argument behind VGG.
Lin, Chen, & Yan, Network In Network: introduces the \(1\times1\) channel mixer and global-average-pooling classifier.
Ioffe & Szegedy, Batch Normalization: the original normalization method and its train/evaluation distinction.
He et al., Deep Residual Learning for Image Recognition: the degradation experiment and residual-block solution.
Yosinski et al., How Transferable Are Features in Deep Neural Networks?: a careful empirical account of when transferred features remain useful.

Exercises

(Pencil.) Generalize the kernel-stacking arithmetic: \(n\) stacked \(3\times3\) layers versus one \((2n{+}1)\times(2n{+}1)\) kernel, at constant width \(C\). Then redo it for a layer that grows channels \(C \rightarrow 2C\): split into two \(3\times3\)s (\(C \rightarrow C \rightarrow 2C\) and \(C \rightarrow 2C \rightarrow 2C\)) and find when the split stops saving parameters.
(Pencil.) Verify the Inception bottleneck arithmetic from the callout (\(819{,}200\) versus \(110{,}592\)), then design the cheapest \(1\times1\)-plus- \(5\times5\) pipeline from 128 channels to 64 output channels that keeps the squeeze width at least a quarter of the input width.
(Code.) Make VGGSmall’s blocks depthwise separable: replace each \(3\times3\) conv with a per-channel \(3\times3\) (groups=c_in) followed by a \(1\times1\) mixer. Count parameters, retrain, and report the accuracy-per-parameter ratio against the original. (This is MobileNet’s trick for phones, the lecture’s fifth principle.)
(Code.) Ablate BatchNorm: retrain VGGSmall with the BN layers removed, same budget. Compare final accuracy and the first ten epochs’ training accuracy. Then repeat the bn-drift cell’s measurement on both trained networks. Does training itself fix the activation scales?
(Code.) Data augmentation as a third road: on the 30-image shoe task, train from scratch with random horizontal flips and ±3-pixel shifts (Chapter 6’s exercise, now as a tool). Does augmentation close the gap to the lecture’s full-data numbers further than transfer did? Why might augmentation and transfer help in different regimes?