9  Modern CNNs and Transfer Learning

Chapter 8 ended with a working machine and a report card. LeNet reads garments at 82.5% with 61,706 parameters; graceful under shift where the MLP cliff-dived. But the card has two demerits we wrote down explicitly: 96% of those parameters sit in the dense head rather than in the convolutions that do the seeing, and the shift cliff was softened, not abolished. And one IOU: batch normalization, promised for this chapter.

What happened after LeNet is a decade of architecture research, and it can drown you in named networks. Four design questions will keep the mechanism visible; we let them drive:

1. Why \(5 \times 5\)? Could smaller filters do the same job with fewer parameters?
2. How do we go deeper without gradients dying on the way down?
3. Where do the parameters actually live, and how do we evict them?
4. Can we design reusable blocks: optimized layer-combinations we stack, keeping a bird’s-eye view instead of re-designing what already works?

Each question has a famous answer (VGG, ResNet, NiN’s \(1\times1\) + global pooling, and the block habit itself), and each answer is a constraint-shaped idea we can test on our own data. At the end, the practical superpower this sequence builds to, reusing a pretrained backbone, gets the most honest experiment in this book.

Figure 9.1: The chapter’s construction grammar. Individual operators become reusable blocks; architectures repeat those blocks and attach a head. VGG repeats small-kernel atoms, NiN adds per-pixel channel mixing and global averaging, and ResNet adds an identity route around each learned correction.

Read the figure from left to right whenever a named architecture becomes noisy. First identify the operators, then the block contract, then the repetition schedule and head. The rest of the chapter changes one of those three levels at a time.

Both trainers below are Listing 4.1 with one factoring: the inner loop (train_existing) accepts a model and optimizer that already exist, because this chapter’s whole subject is training models it did not just construct — fresh heads on frozen trunks, and selectively thawed layers at their own learning rates.

1. Load imports, data, and the shared training recipe.
2. Define the reusable helpers: train_existing and train_model.
3. Define the reusable helpers: accuracy and count.

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
from collections.abc import Callable

# [1]
torch.manual_seed(6050)
train = torch.load("../../data/fashion-train.pt")
test = torch.load("../../data/fashion-test.pt")
X_tr = train["X"].float().unsqueeze(1) / 255.0    # (1200, 1, 28, 28)
X_te = test["X"].float().unsqueeze(1) / 255.0     # (600, 1, 28, 28)
y_tr, y_te, classes = train["y"], test["y"], train["classes"]

# [2]
def train_existing(
    model: nn.Module, opt: torch.optim.Optimizer, epochs: int,
    batch: int = 128,
    data: tuple[torch.Tensor, torch.Tensor] | None = None,
) -> nn.Module:
    X, y = data if data is not None else (X_tr, y_tr)
    n = len(X)
    for _ in range(epochs):
        perm = torch.randperm(n)
        for i in range(0, n, batch):
            idx = perm[i:i + batch]
            model.train()
            loss = F.cross_entropy(model(X[idx]), y[idx])
            opt.zero_grad(); loss.backward(); opt.step()
    model.eval()
    return model

def train_model(model_fn: Callable[[], nn.Module], epochs: int,
                batch: int = 128, seed: int = 6050, lr: float = 1e-3,
                data: tuple[torch.Tensor, torch.Tensor] | None = None) -> nn.Module:
    torch.manual_seed(seed)
    model = model_fn()
    opt = torch.optim.Adam(model.parameters(), lr=lr)
    return train_existing(model, opt, epochs, batch, data)

# [3]
@torch.no_grad()
def accuracy(model: nn.Module, X: torch.Tensor, y: torch.Tensor) -> float:
    model.eval()
    return (model(X).argmax(1) == y).float().mean().item()

def count(model: nn.Module) -> int:
    return sum(p.numel() for p in model.parameters())

One evaluation label needs an honesty note. Chapter 8 already opened the 600-image holdout, and this chapter queries it repeatedly while comparing architectures. We keep the familiar X_te name, but treat it as the book’s fixed benchmark. These comparisons are descriptive, not an unbiased estimate after model selection; a final claim would require a fresh, untouched test set.

Two small novelties in the recipe, both because this chapter’s networks contain batch normalization: model.train() before each step and model.eval() before each evaluation. Why that matters is the subject of the second section.

9.1 Question 1: why \(5 \times 5\)? Stack small kernels instead

VGG’s answer (Simonyan & Zisserman, 2014) is the cleanest idea in this chapter. Recall the receptive-field ledger of Chapter 8: stacking grows the field. Two stacked \(3 \times 3\) convolutions let the second layer see \(5 \times 5\) of the input, the same receptive field as one \(5 \times 5\) kernel. But compare the bills, for a layer with \(C\) channels in and \(C\) out:

\[ \underbrace{25\,C^2}_{\text{one } 5\times5} \qquad\text{versus}\qquad \underbrace{2 \times 9\,C^2 = 18\,C^2}_{\text{two } 3\times3\text{s}}, \]

a 28% saving. The stacked pair also fires a ReLU twice where the big kernel fires once. Same sight, fewer parameters, more nonlinearity. Three \(3\times3\)s reach a \(7\times7\) field for \(27C^2\) against \(49C^2\): the deeper you take the idea, the better the deal gets.

1. Prepare the inputs and fixed settings for the example.
2. Implement the parameter bill, C = 32.

# [1]
C = 32
one_5x5 = nn.Conv2d(C, C, 5, padding=2)
two_3x3 = nn.Sequential(nn.Conv2d(C, C, 3, padding=1), nn.ReLU(),
                        nn.Conv2d(C, C, 3, padding=1))
# [2]
print(f"one 5x5:  {count(one_5x5):,} parameters, 1 ReLU after")
print(f"two 3x3s: {count(two_3x3):,} parameters, 2 ReLUs")

one 5x5:  25,632 parameters, 1 ReLU after
two 3x3s: 18,496 parameters, 2 ReLUs

One honest caveat before you re-derive the field equations of vision from this: the \(18C^2 < 25C^2\) arithmetic assumes the channel width stays \(C\) through the stack. When a layer grows channels (as LeNet’s \(6 \rightarrow 16\) did), splitting it into two growing \(3\times3\)s can cost more, not less. VGG’s design sidesteps this by keeping width constant inside each block and changing it only between blocks, which is exactly the shape our code will take. (Exercise 1 makes you find the break-even point.)

9.2 The stabilizer we owe you: batch normalization

Before we stack deeper than ever, we need the tool Chapter 8 used and we deferred. Here is the problem it solves. Watch what happens to the scale of activations as a signal crosses twelve freshly initialized \(3\times3\) conv layers:

1. Define the reusable stack12 helper.
2. Prepare the inputs and fixed settings for the example.
3. Activation scale through 12 conv layers, with and without BN.
4. Report or visualize the measured result.

# [1]
def stack12(with_bn: bool, ch: int = 16) -> nn.Sequential:
    layers = [nn.Conv2d(1, ch, 3, padding=1)]
    for _ in range(11):
        if with_bn:
            layers.append(nn.BatchNorm2d(ch))
        layers += [nn.ReLU(), nn.Conv2d(ch, ch, 3, padding=1)]
    return nn.Sequential(*layers)

# [2]
torch.manual_seed(0)
x = X_tr[:256]
plain_stack = stack12(False)
bn_stack = stack12(True)
plain_convs = [m for m in plain_stack if isinstance(m, nn.Conv2d)]
bn_convs = [m for m in bn_stack if isinstance(m, nn.Conv2d)]
# [3]
for source, target in zip(plain_convs, bn_convs):
    target.load_state_dict(source.state_dict())       # identical convolution weights

# [4]
for tag, network in [("without BN", plain_stack), ("with BN   ", bn_stack)]:
    h, stds = x, []
    with torch.no_grad():
        for layer in network:
            h = layer(h)
            if isinstance(layer, nn.Conv2d):
                stds.append(h.std().item())
    print(f"{tag} layer stds: " + "  ".join(f"{s:.3f}" for s in stds[::2]))

without BN layer stds: 0.344  0.059  0.043  0.046  0.049  0.061
with BN    layer stds: 0.344  0.404  0.395  0.385  0.432  0.384

Without normalization the signal’s spread collapses by an order of magnitude within a few layers and keeps sagging. Forward activation scales and backward gradients are not the same quantity, but both are shaped by the stack’s weights and nonlinearities; poorly scaled activations are therefore a warning to inspect gradient flow directly, as we do below.

Batch normalization (Ioffe & Szegedy, 2015) re-standardizes at every layer. For each channel of an NCHW tensor, BatchNorm2d computes the mean and variance across the current minibatch and both spatial axes (\(N\), \(H\), and \(W\)), normalizes the activations, then lets the layer undo that transformation through two learnable knobs:

\[ \hat{x} = \frac{x - \mu_{\text{batch}}}{\sqrt{\sigma^2_{\text{batch}} + \epsilon}}, \qquad y = \gamma\,\hat{x} + \beta . \tag{9.1}\]

The \(\gamma, \beta\) pair matters: normalization is a reset to a healthy scale, not a straitjacket. The network can learn to restore any mean and spread that helps, but it starts every layer from sane numbers. In the printout above, the BN column holds near a constant spread through all twelve layers: every layer trains from day one. The standard placement we adopt is conv \(\rightarrow\) BN \(\rightarrow\) ReLU, with bias=False on the convolution, since \(\beta\) already provides the shift.

WarningBN is two different machines: tell PyTorch which one you’re running

At training time BN normalizes by the current batch’s statistics. At evaluation time there may be no batch (one image!), so it uses running averages collected during training. model.train() and model.eval() switch between the two. Forgetting the switch is the classic BN bug: evaluate in train mode and your predictions depend on whatever else happens to be in the batch; train in eval mode and BN never learns its statistics. Our train_model recipe flips the switch in both directions; look for it. Corollary: BN needs real batches to estimate statistics, so it gets unreliable at tiny batch sizes.

NoteNormalize across what? A seed for Part IV

Batch statistics are one choice among several. When we build transformers (Chapter 14), batches of variable-length sequences will make per-batch statistics awkward, and the same standardize-then-rescale idea will reappear computed per token instead: layer normalization. Same equation, different axis. Remember Equation 9.1 when you meet it.

9.3 Building with blocks: a small VGG

Now assemble Question 1’s answer with Question 4’s habit. Define the atom — conv, BN, ReLU — once; a block stacks atoms and pools; a network stacks blocks. This is the point about VGG versus its hand-tuned predecessors: you stop re-designing and start repeating.

1. Define the reusable helpers: atom and VGGSmall.
2. VGGSmall — two blocks of stacked 3×3 atoms.
3. Report or visualize the measured result.

# [1]
def atom(c_in: int, c_out: int) -> list[nn.Module]:
    return [nn.Conv2d(c_in, c_out, 3, padding=1, bias=False),
            nn.BatchNorm2d(c_out), nn.ReLU()]

class VGGSmall(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            *atom(1, 16), *atom(16, 16), nn.MaxPool2d(2),    # -> (16, 14, 14)
            *atom(16, 32), *atom(32, 32), nn.MaxPool2d(2),   # -> (32, 7, 7)
        )
        self.head = nn.Sequential(nn.Flatten(),
                                  nn.Linear(32 * 7 * 7, 128), nn.ReLU(),
                                  nn.Linear(128, 10))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.features(x))

# [2]
vgg = train_model(VGGSmall, epochs=60)
n_head = count(vgg.head)
# [3]
print(f"VGGSmall: {count(vgg):,} parameters ({n_head:,} in the head, "
      f"{n_head / count(vgg):.0%})")
print(f"test accuracy {accuracy(vgg, X_te, y_te):.1%}")

VGGSmall: 218,586 parameters (202,122 in the head, 92%)
test accuracy 86.7%

Two readings again. The good: this VGG-style recipe reaches 86.7%, four points above LeNet in this run. Because depth, width, normalization, kernel sizes, parameter count, and training duration all changed together, this comparison does not isolate which ingredient earned the gain. The bad: 218,586 parameters, three and a half times LeNet’s total, and the head’s share got worse — 92% of the network is a dense layer reading a flattened grid. The convolutional diet succeeded and the total got fatter anyway. Which forces the third question: where do the parameters live, and where can we do the most damage to the count?

9.4 Question 3: \(1 \times 1\) convolutions, and firing the flatten head

The bloat has one address: nn.Linear(1568, 128). To evict it we need one more tool, odd-looking at first sight: a convolution whose kernel is \(1 \times 1\).

A \(1 \times 1\) kernel does no spatial mixing at all — it looks at a single pixel. But across channels it does exactly what Equation 8.1 says: at each position, take the \(C_{\text{in}}\) channel values and form \(C_{\text{out}}\) weighted combinations plus bias. Pause on what that is: a linear model across the channels, run separately at every pixel — Chapter 1’s machine, miniaturized and stamped across the image. Add a ReLU and each pixel is running a tiny MLP over its own feature vector. We summarize channels, we do not discard them: compressing 64 feature maps into 32 loses far less than pooling them away, and each compression injects another nonlinearity almost for free.

That tool enables the Network-in-Network design (Lin et al., 2013), and with it the clean head. A NiN block is one \(3\times3\) (some spatial mixing) followed by two \(1\times1\)s (per-pixel channel mixing). And the classifier becomes:

1. Let the last block produce a stack of feature maps.
2. Global average pooling (GAP): average each map over all positions — 64 maps in, 64 numbers out. No parameters at all.
3. One small linear layer to the logits.

1. Define the reusable helpers: nin_block and NINSmall.
2. Prepare the inputs and fixed settings for the example.
3. NINSmall — architecture and first 75 epochs.

# [1]
def nin_block(c_in: int, c_out: int) -> list[nn.Module]:
    return [nn.Conv2d(c_in, c_out, 3, padding=1, bias=False),
            nn.BatchNorm2d(c_out), nn.ReLU(),
            nn.Conv2d(c_out, c_out, 1), nn.ReLU(),
            nn.Conv2d(c_out, c_out, 1), nn.ReLU()]

class NINSmall(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            *nin_block(1, 16), nn.MaxPool2d(2),
            *nin_block(16, 32), nn.MaxPool2d(2),
            *nin_block(32, 64),
        )
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(64, 10))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.features(x))

# [2]
torch.manual_seed(6050)
nin = NINSmall()
nin_opt = torch.optim.Adam(nin.parameters(), lr=1e-3)
# [3]
nin = train_existing(nin, nin_opt, epochs=75)

1. NINSmall — continue the same optimizer through epoch 150.
2. Report or visualize the measured result.

# [1]
nin = train_existing(nin, nin_opt, epochs=75)
# [2]
print(f"NINSmall: {count(nin):,} parameters "
      f"(head: {count(nin.head):,})")
print(f"test accuracy {accuracy(nin, X_te, y_te):.1%}")

NINSmall: 35,034 parameters (head: 650)
test accuracy 76.2%

The parameter story is a rout: 35,034 total, a 650-parameter head, six times smaller than VGGSmall. The accuracy is 76.2%, and that dip is worth more attention than the win, so let us not rush past it. First, though, collect the promised payoff. Chapter 8 said the remaining shift-cliff was the flatten head’s fault: it reads the final grid positionally. GAP averages over positions, so the head no longer assigns a different weight to every location. The rematch of the rematch:

1. Define the reusable helpers: LeNet and shift_right.
2. Shift curves, flatten head vs GAP head.
3. Report or visualize the measured result.

# [1]
class LeNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, 5, padding=2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(400, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = F.max_pool2d(F.relu(self.conv1(x)), 2)
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.flatten(1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.fc3(x)

# [2]
lenet = train_model(LeNet, epochs=150)          # Chapter 8's model, same recipe

def shift_right(X: torch.Tensor, px: int) -> torch.Tensor:
    if px == 0:
        return X.clone()
    out = torch.zeros_like(X)
    out[..., px:] = X[..., :-px]
    return out

shifts = list(range(5))
acc_lenet = [accuracy(lenet, shift_right(X_te, s), y_te) for s in shifts]
acc_nin = [accuracy(nin, shift_right(X_te, s), y_te) for s in shifts]
# [3]
for s in shifts:
    print(f"shift {s}px:   LeNet {acc_lenet[s]:.1%}   NIN+GAP {acc_nin[s]:.1%}")

plt.figure(figsize=(5.5, 3.2))
plt.plot(
    shifts, acc_lenet, "s-", color="#232D4B", lw=2, ms=7,
    label="LeNet (flatten head)",
)
plt.plot(
    shifts, acc_nin, "^-", color="#E57200", lw=2, ms=7,
    label="NINSmall (GAP head)",
)
plt.axhline(0.1, ls=":", color="#B8B8A8")
plt.xlabel("shift (pixels right)"); plt.ylabel("test accuracy")
plt.ylim(0, 0.9); plt.xticks(shifts); plt.legend()
plt.tight_layout(); plt.show()

shift 0px:   LeNet 82.5%   NIN+GAP 76.2%
shift 1px:   LeNet 74.8%   NIN+GAP 70.3%
shift 2px:   LeNet 62.3%   NIN+GAP 69.3%
shift 3px:   LeNet 45.3%   NIN+GAP 67.5%
shift 4px:   LeNet 26.8%   NIN+GAP 66.5%

Figure 9.2: Chapter 8’s experiment, third round. LeNet’s flatten head still slides below 30% by four pixels of shift. NINSmall with its global-average-pool head barely tilts: per-channel averages change much less when features move. GAP removes position-specific weights from the head, but padding, stride, pooling, and content clipped at the image edge keep the complete network from being exactly shift-invariant.

The curve that fell from 82% to 27% across this book’s Part II is now a gentle slope from 76% to 67%. Position-specific weights are gone from NINSmall’s head, and the result is consistent with the promised robustness benefit. It is not a matched-head ablation: the trunks and training recipes differ too. The complete CNN is only approximately shift-tolerant; boundaries, strides, and pooling can still change its features.

Now the dip. On clean, centered test garments NINSmall trails LeNet by six points. That result is consistent with Chapter 8’s warning: on centered data, position can carry information, and a GAP head cannot assign separate weights to separate locations. But GAP is not the only changed ingredient, so this run does not prove it caused the gap. On the full 60,000-image dataset, the book’s pinned Rivanna runs put NiN at \(92.78\% \pm 0.08\%\) across seeds 6050–6052. At our scale the comparison exposes a hypothesis worth testing with a matched-head ablation: shift tolerance may be bought with positional information.

NoteQuestion 4’s other famous block: Inception, in one breath

GoogLeNet’s Inception block (2014) answers “which kernel size?” with “all of them”: parallel \(1\times1\), \(3\times3\), \(5\times5\), and pooling branches, concatenated. Its enabling trick is the \(1\times1\) bottleneck: compress channels before the expensive spatial kernels. For one \(5\times5\) branch at 256 channels: direct, \(5^2 \times 256 \times 128 \approx 819\)k parameters; with a \(1\times1\) squeeze to 32 first, \(256 \times 32 + 5^2 \times 32 \times 128 \approx 111\)k — an 86% cut for the same nominal operation. We won’t build Inception here (the principle, channel compression before spatial expense, is the transferable part), but Exercise 2 walks the arithmetic and the course assignment has you build the block itself.

NoteArchitecture bridge (non-examinable): DenseNet concatenates the history

The residual block below preserves an old representation by addition: \(x_{\ell+1}=x_\ell+F_\ell(x_\ell)\). A DenseNet block makes a different connectivity choice. If \(x_j\) denotes layer \(j\)’s output feature map, layer \(\ell\) receives the channel-wise concatenation \([x_0,x_1,\ldots,x_{\ell-1}]\) and contributes a small new group of feature maps for later layers. Earlier features therefore remain directly available instead of being recreated, while the channel axis grows; transition blocks compress channels and downsample between dense blocks. In Appendix B’s language, the defining operation is concatenation along the feature-channel axis, not another kind of residual addition (Appendix B).

9.5 Question 2: going deeper, and the wall you hit

VGG says depth is the direction. So push it: stack twenty of our two-conv blocks (with BN, per the recipe, on a \(14\times14\) grid so the experiment fits a laptop) and compare against the identical stack with one change we’ll reveal after the numbers.

Predict before running: two networks, identical parameter counts, twenty blocks each — one plain, one residual. Will the plain net merely trail, or fail to fit the training set at all?

1. Define the reusable helpers: Block and DeepNet.
2. Train the plain 20-block network.
3. Report or visualize the measured result.

# [1]
class Block(nn.Module):
    def __init__(self, ch: int, residual: bool):
        super().__init__()
        self.c1 = nn.Conv2d(ch, ch, 3, padding=1, bias=False)
        self.b1 = nn.BatchNorm2d(ch)
        self.c2 = nn.Conv2d(ch, ch, 3, padding=1, bias=False)
        self.b2 = nn.BatchNorm2d(ch)
        self.residual = residual
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        y = self.b2(self.c2(F.relu(self.b1(self.c1(x)))))
        return F.relu(y + x) if self.residual else F.relu(y)

class DeepNet(nn.Module):
    def __init__(self, nblocks: int, residual: bool, ch: int = 12):
        super().__init__()
        self.stem = nn.Conv2d(1, ch, 3, padding=1)
        self.pool = nn.MaxPool2d(2)                       # 28 -> 14 early
        self.blocks = nn.Sequential(*[Block(ch, residual)
                                      for _ in range(nblocks)])
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(ch, 10))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.blocks(self.pool(F.relu(self.stem(x)))))

# [2]
plain = train_model(lambda: DeepNet(20, residual=False), epochs=30)
# [3]
print(f"plain-20     train {accuracy(plain, X_tr, y_tr):.1%}   "
      f"test {accuracy(plain, X_te, y_te):.1%}")

plain-20     train 60.8%   test 51.0%

1. Train the matched residual 20-block network.
2. Report or visualize the measured result.

# [1]
resid = train_model(lambda: DeepNet(20, residual=True), epochs=30)
# [2]
print(f"residual-20  train {accuracy(resid, X_tr, y_tr):.1%}   "
      f"test {accuracy(resid, X_te, y_te):.1%}")

residual-20  train 100.0%   test 77.8%

Look at the training column first, because it carries the whole lesson. The plain 40-conv-layer network cannot even fit the 1,200 images it sees every epoch: 61% train accuracy, while its residual twin memorizes them and generalizes twenty-seven points better. This is an optimization failure in the same family as the degradation problem He et al. reported in 2015, where their 56-layer plain net trained worse than their 20-layer one. Our experiment isolates the residual rescue at one depth; a strict demonstration that adding depth degrades a plain network would also require a shallower plain control.

The one change: each block computes \(F(x)\) and outputs \(F(x) + x\). A residual connection lets the input skip over the block and adds it back.

\[ H(x) = F(x) + x \qquad\Longrightarrow\qquad \frac{\partial L}{\partial x} = \frac{\partial L}{\partial H}\left(\frac{\partial F}{\partial x} + I\right). \tag{9.2}\]

Figure 9.3: A residual block has two routes. The learned branch computes a correction \(F(x)\); the identity branch carries \(x\) unchanged to the addition. Even when the learned branch has a difficult Jacobian, the shortcut leaves a direct route for activations and gradients.

Read the right-hand side with Chapter 5 eyes. Blame arriving at a block’s output reaches its input through two additive terms: the learned branch \(\partial F /\partial x\) and the identity \(I\). The identity contribution gives gradient flow a direct route that does not itself multiply by the learned weights. It is powerful, not magical: the learned Jacobian can still reinforce, distort, or even partly cancel that term. Chapter 5 called ReLU’s open gate a gradient superhighway through a single unit; a skip connection builds a direct lane into every block. The block only needs to learn the residual, the correction on top of “pass it through,” and doing nothing is easy to represent: \(F = 0\).

Here is the highway visible at initialization, extending Chapter 5’s depth experiment from dense layers to conv stacks:

1. Define the reusable helpers: PlainNoBN and stem_grad.
2. Prepare the inputs and fixed settings for the example.
3. Gradient at the stem vs. depth, three designs.

# [1]
class PlainNoBN(nn.Module):
    def __init__(self, nconvs: int, ch: int = 12):
        super().__init__()
        self.stem = nn.Conv2d(1, ch, 3, padding=1)
        self.pool = nn.MaxPool2d(2)
        self.layers = nn.Sequential(*[m for _ in range(nconvs)
                                      for m in (nn.Conv2d(ch, ch, 3, padding=1),
                                                nn.ReLU())])
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(ch, 10))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.layers(self.pool(F.relu(self.stem(x)))))

def stem_grad(
    make_net: Callable[[], nn.Module],
) -> tuple[float, float, int, int]:
    torch.manual_seed(0)
    net = make_net()
    F.cross_entropy(net(X_tr[:128]), y_tr[:128]).backward()
    grad = net.stem.weight.grad
    return (grad.double().norm().item(), grad.abs().max().item(),
            torch.count_nonzero(grad).item(), grad.numel())

# [2]
depths = [4, 12, 24]                                  # blocks (2 convs each)
diagnostics = {
    "plain, no BN": [stem_grad(lambda d=d: PlainNoBN(2 * d)) for d in depths],
    "plain + BN": [stem_grad(lambda d=d: DeepNet(d, False)) for d in depths],
    "residual + BN": [stem_grad(lambda d=d: DeepNet(d, True)) for d in depths],
}
curves = {name: [row[0] for row in rows]
          for name, rows in diagnostics.items()}
# [3]
for name, ys in curves.items():
    print(f"{name:14s}: " + "   ".join(f"{v:.1e}" for v in ys))
norm64, maxabs, nonzero, total = diagnostics["plain, no BN"][-1]
print(
    f"48-layer no-BN check: max |component| {maxabs:.1e}; "
    f"nonzero {nonzero}/{total}; float64 norm {norm64:.1e}"
)

plt.figure(figsize=(6.2, 3.4))
for (name, ys), color in zip(curves.items(),
                             ["#B8B8A8", "#E57200", "#232D4B"]):
    xs = [2 * d for d in depths]
    plt.semilogy(xs, ys, "o-", ms=5, color=color, label=name)
plt.xlabel("conv layers")
plt.ylabel(r"$\|\partial L / \partial W^{(\mathrm{stem})}\|$")
plt.legend(); plt.tight_layout(); plt.show()

plain, no BN  : 9.0e-06   3.3e-13   2.2e-23
plain + BN    : 7.6e-02   5.0e-01   8.3e+01
residual + BN : 1.6e-01   4.9e-01   5.0e-01
48-layer no-BN check: max |component| 5.2e-24; nonzero 108/108; float64 norm 2.2e-23

Figure 9.4: Gradient magnitude reaching the stem (first layer) at initialization, versus depth, for three designs. In the 48-layer no-BN stack, the true norm is 2.2e-23; a float32 norm calculation reports zero because squaring these nonzero components underflows. BN fixes the vanishing but overshoots: in plain stacks the gradient grows with depth. Residual blocks keep the gradient within one order of magnitude across these depths.

The final no-BN row also exposes a numerical trap: its components remain nonzero, but squaring them in float32 can underflow before the norm is summed. The range, resolution, and accumulator roles behind that distinction are gathered in Appendix C.

TipThe most important seed this chapter plants

Residual connections are not only a CNN trick. They became a central design device in many deep architectures; when we assemble a transformer in Chapter 14, every attention layer and every feedforward layer will be wrapped as \(x + F(x)\), and the freshly-met layer normalization will sit beside each skip. The picture to carry forward: a residual stream with direct additive routes through the network, each block reading from it and writing a correction back. Those routes improve conditioning without making the stream immune to learned-branch interactions. Attention will be one kind of correction. You now own both parts of that skeleton.

9.6 The scorecard: what a decade of design bought

The chapter’s small-data studies isolated mechanisms. The architecture comparison deserves the full task: all 60,000 Fashion-MNIST training images, split once into 50,000 for fitting and 10,000 for validation, with the official 10,000-image test set opened only after validation selected the checkpoint. We ran LeNet, NiN, VGG, and a nine-block residual network for three end-to-end seeds on Rivanna:

1. Load the 12 pinned full-data Rivanna records.
2. Summarize test accuracy and parameter count by architecture.

# [1]
import json
from pathlib import Path

scorecard_root = Path("../../experiments/rivanna/results/scorecard")
scorecard_records = [
    json.loads(path.read_text()) for path in sorted(scorecard_root.glob("*.json"))
]
scorecard_order = ["lenet", "nin", "vgg", "resnet"]
scorecard_colors = ["#B8B8A8", "#232D4B", "#E57200", "#5379AA"]

# [2]
scorecard = {}
for model_name in scorecard_order:
    rows = [row for row in scorecard_records if row["model"] == model_name]
    accuracies = torch.tensor([row["test_accuracy"] for row in rows])
    scorecard[model_name] = {
        "parameters": rows[0]["parameter_count"],
        "mean": accuracies.mean().item(),
        "sd": accuracies.std(unbiased=True).item(),
    }

Figure 9.5: The full-data Part II scorecard on Rivanna. Points show mean official-test accuracy and bars one sample standard deviation across seeds 6050–6052. LeNet reaches 92.52%, NiN 92.78%, VGG 93.73%, and the residual network 94.20%. NiN is the most parameter-efficient point; VGG buys more accuracy with depth; residual blocks reach the strongest endpoint in this declared training regime. Parameter count is not training compute, and these four points are not a universal architecture ranking.

The scale changes the verdict from the 1,200-image mechanism studies. NiN’s global head no longer pays a large clean-accuracy penalty, and all four architectures clear 92%. The residual network leads this declared recipe, but the plot still has one horizontal axis too few: parameters measure storage, not optimizer steps, activation memory, or total training work.

9.7 Transfer learning: the mechanics, and an honest experiment

Everything so far trains from scratch. The final move is the one that defines practice today: most image features are shared — edges, textures, parts transfer across tasks — so train one strong backbone once, on enormous data, and reuse it everywhere. The mechanics come in two strengths:

• Linear probe: freeze the pretrained trunk entirely (requires_grad = False), extract its features, train only a new linear head on your labels.
• Fine-tuning: additionally unfreeze the last block or two, training them at a much smaller learning rate than the fresh head, so the pretrained features adapt gently instead of being trampled.

We will test the idea properly. The task: classify the three shoe classes (sandal, sneaker, ankle boot) from ten labeled examples each. The 30-image regime is exactly where transfer should shine — too few labels to learn features, goes the story, so imported features should dominate. Three contenders:

1. From scratch: our VGG-style trunk trained on the 30 images alone.
2. Our own pretrained trunk: the same trunk pretrained on the seven non-shoe classes (863 images), frozen, linear probe.
3. A real pretrained backbone: SqueezeNet 1.1 trained on ImageNet — 1.2 million photographs, 1,000 classes — loaded from weights committed with this book, frozen, linear probe on its 512-dimensional features. (Fashion images get upsampled to 96×96 and repeated to three channels to fit its expectations. That awkwardness is part of the experiment, and part of the lesson.)

1. Prepare the inputs and fixed settings for the example.
2. Define the reusable helpers: Trunk and squeeze_features.
3. Implement the shoe task, our own trunk, and the frozen probes.
4. Report or visualize the measured result.
5. Define the reusable helpers: own_features and linear_probe.

from torchvision.models import squeezenet1_1

# [1]
shoe = torch.tensor([5, 7, 9])
new_label = {5: 0, 7: 1, 9: 2}
is_shoe_tr = (y_tr[:, None] == shoe).any(1)
is_shoe_te = (y_te[:, None] == shoe).any(1)
X_shoe_te = X_te[is_shoe_te]
y_shoe_te = torch.tensor([new_label[int(c)] for c in y_te[is_shoe_te]])

# our own trunk: pretrain on the 7 non-shoe classes
src = [0, 1, 2, 3, 4, 6, 8]
remap = {c: i for i, c in enumerate(src)}
X_src = X_tr[~is_shoe_tr]
y_src = torch.tensor([remap[int(c)] for c in y_tr[~is_shoe_tr]])

# [2]
class Trunk(nn.Module):
    def __init__(self, n_classes: int):
        super().__init__()
        self.features = nn.Sequential(
            *atom(1, 16), *atom(16, 16), nn.MaxPool2d(2),
            *atom(16, 32), *atom(32, 32), nn.MaxPool2d(2), *atom(32, 64))
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                                  nn.Linear(64, n_classes))
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.head(self.features(x))

# [3]
own_trunk = train_model(lambda: Trunk(7), epochs=100, data=(X_src, y_src))
X_src_te = X_te[~is_shoe_te]
y_src_te = torch.tensor([remap[int(c)] for c in y_te[~is_shoe_te]])
# [4]
print(f"own trunk, source task (7 classes): "
      f"{accuracy(own_trunk, X_src_te, y_src_te):.1%}")

# the real thing: ImageNet SqueezeNet, from committed weights (no download)
sq = squeezenet1_1(weights=None)
sq.load_state_dict(torch.load("../../data/squeezenet1_1-imagenet.pt"))
sq.eval()
for p in sq.parameters():
    p.requires_grad = False

IMNET_MEAN = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
IMNET_STD = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)

@torch.no_grad()
def squeeze_features(X: torch.Tensor) -> torch.Tensor:  # (N,1,28,28) -> (N,512)
    x = X.repeat(1, 3, 1, 1)
    x = F.interpolate(x, size=96, mode="bilinear", align_corners=False)
    f = sq.features((x - IMNET_MEAN) / IMNET_STD)
    return F.adaptive_avg_pool2d(f, 1).flatten(1)

# [5]
@torch.no_grad()
def own_features(X: torch.Tensor) -> torch.Tensor:      # (N,1,28,28) -> (N,64)
    return F.adaptive_avg_pool2d(own_trunk.features(X), 1).flatten(1)

def linear_probe(Z_tr: torch.Tensor, y_f: torch.Tensor,
                 Z_te: torch.Tensor, seed: int) -> float:
    torch.manual_seed(seed)
    head = nn.Linear(Z_tr.shape[1], 3)
    opt = torch.optim.Adam(head.parameters(), lr=1e-2,
                           weight_decay=1e-3)   # built-in L2 penalty (ch. 1's ridge)
    for _ in range(400):
        loss = F.cross_entropy(head(Z_tr), y_f)
        opt.zero_grad(); loss.backward(); opt.step()
    return (head(Z_te).argmax(1) == y_shoe_te).float().mean().item()

own trunk, source task (7 classes): 79.7%

1. Prepare the inputs and fixed settings for the example.
2. Scratch vs. own trunk vs. ImageNet trunk, three seeds.
3. Report or visualize the measured result.

# [1]
K = 10
rows = []
# [2]
for seed in [0, 1, 6050]:
    torch.manual_seed(seed)
    idx = torch.cat([(y_tr == c).nonzero().squeeze(1)
                     [torch.randperm(int((y_tr == c).sum()))[:K]] for c in shoe])
    X_few = X_tr[idx]
    y_few = torch.tensor([new_label[int(c)] for c in y_tr[idx]])

    scratch = train_model(lambda: Trunk(3), epochs=100, seed=seed,
                          data=(X_few, y_few))
    rows.append((accuracy(scratch, X_shoe_te, y_shoe_te),
                 linear_probe(own_features(X_few), y_few,
                              own_features(X_shoe_te), seed),
                 linear_probe(squeeze_features(X_few), y_few,
                              squeeze_features(X_shoe_te), seed)))

# [3]
print("seed     scratch   own-trunk probe   ImageNet probe")
for seed, r in zip([0, 1, 6050], rows):
    print(f"{seed:4d}     {r[0]:.1%}       {r[1]:.1%}           {r[2]:.1%}")
mean = [sum(c) / 3 for c in zip(*rows)]
print(f"mean     {mean[0]:.1%}       {mean[1]:.1%}           {mean[2]:.1%}")

seed     scratch   own-trunk probe   ImageNet probe
   0     86.4%       68.9%           88.1%
   1     85.3%       65.5%           85.9%
6050     88.1%       77.4%           87.6%
mean     86.6%       70.6%           87.2%

Read that table slowly, because it does not say what the textbook story predicts, and the discrepancy is the best lesson in this chapter. Training from scratch on thirty images fights the mighty ImageNet backbone to a dead heat — the means differ by less than a point, well inside seed noise. And our own pretrained trunk, perfectly competent on its source task per the printout above, transfers worse than nothing. Three reasons, each a general principle:

1. A trunk can only donate what its data taught it. Our own trunk saw 863 shirts, bags, and trousers. Nothing in that curriculum required foot-shaped detectors, so its 64 features flatten sandals, sneakers, and boots into nearly the same point. Pretraining is not magic; it is curriculum.
2. Domain and resolution gaps tax the donation. SqueezeNet’s filters expect 224-pixel color photographs; we feed it 28-pixel grayscale icons inflated to 96. Its early layers hunt for detail that simply is not there. (The matching recipe, resize and normalize to match the pretraining pipeline — is doing real work; we complied as far as the data allows.)
3. Transfer wins when the target is big relative to your labels — not small. This is the quiet one. Three shoe silhouettes at \(28 \times 28\) is a small problem: thirty images genuinely suffice, so the scratch baseline is strong and there is little room for imported knowledge to pay rent. The full ten-class task below is different: 50,000 fitting labels, 224-pixel inputs, and a ResNet-18-sized feature extractor.

The full-data rematch

The full task lets us separate three questions cleanly: what ImageNet features can do without changing, what the last residual stage can learn when allowed to adapt, and what the architecture can learn from scratch with all target labels available.

1. Load the nine pinned ResNet-18 transfer records.
2. Summarize official-test accuracy by training regime.

# [1]
transfer_root = Path("../../experiments/rivanna/results/transfer")
transfer_records = [
    json.loads(path.read_text()) for path in sorted(transfer_root.glob("*.json"))
]
transfer_order = ["probe", "finetune", "scratch"]

# [2]
transfer_summary = {}
for regime in transfer_order:
    values = torch.tensor([
        row["test_accuracy"] for row in transfer_records if row["regime"] == regime
    ])
    transfer_summary[regime] = {
        "values": values,
        "mean": values.mean().item(),
        "sd": values.std(unbiased=True).item(),
    }

Figure 9.6: ImageNet transfer on full Fashion-MNIST at 224 pixels. Across seeds 6050–6052, the frozen linear probe reaches 88.79% ± 0.08% official-test accuracy, last-block fine-tuning 93.92% ± 0.12%, and scratch 94.14% ± 0.08%. Fine-tuning repairs most of the domain mismatch, but 50,000 target labels are enough for scratch to match it in this regime. Bars are means, dots are seeds, and error bars are one sample standard deviation.

The frozen probe’s 88.79% is the distribution gap made visible: features learned from natural photographs do not linearly separate all ten grayscale garment classes at this resolution. Updating only the last residual stage recovers more than five points. Scratch ends 0.22 points above fine-tuning, smaller than the run-to-run variation one would need to resolve as a family-level claim. With 50,000 target labels, scratch is no longer feature-starved. Transfer changed the starting point and permitted updates; it did not guarantee the best final endpoint.

TipWhen to reach for a pretrained backbone

The honest decision rule supported by the experiments is: transfer pays when (your labels are scarce) and (the target task is feature-hungry) and (the pretraining data plausibly covers the target’s features at a matched scale). At 224 pixels and 18 landmark classes — the course assignment — all three hold, and transfer is the winning move by a wide margin. At 28 pixels and 3 silhouettes, the conditions fail and scratch fights the superpower to a draw. Knowing which regime you are in is the skill; the mechanics (requires_grad, learning-rate splits) are the easy part.

The mechanics, for completeness, since you will use them constantly from Part V onward: fine-tuning SqueezeNet’s last block with a two-learning-rate recipe (fresh head fast, pretrained layers slow):

1. Define the reusable prep helper.
2. Prepare the inputs and fixed settings for the example.
3. Fine-tuning — unfreeze the last block, two learning rates.
4. Report or visualize the measured result.

# [1]
def prep(X: torch.Tensor) -> torch.Tensor:
    x = X.repeat(1, 3, 1, 1)
    x = F.interpolate(x, size=96, mode="bilinear", align_corners=False)
    return (x - IMNET_MEAN) / IMNET_STD

# [2]
torch.manual_seed(6050)
idx = torch.cat([(y_tr == c).nonzero().squeeze(1)
                 [torch.randperm(int((y_tr == c).sum()))[:K]] for c in shoe])
X_few, y_few = prep(X_tr[idx]), torch.tensor([new_label[int(c)]
                                              for c in y_tr[idx]])

ft = squeezenet1_1(weights=None)
ft.load_state_dict(torch.load("../../data/squeezenet1_1-imagenet.pt"))
head = nn.Linear(512, 3)
# [3]
for p in ft.parameters():
    p.requires_grad = False
for p in ft.features[-1].parameters():         # last Fire block only
    p.requires_grad = True

opt = torch.optim.Adam([
    {"params": head.parameters(), "lr": 1e-3},       # fresh head: normal
    {"params": ft.features[-1].parameters(), "lr": 1e-4},  # trunk: gentle
], weight_decay=1e-3)

ft.train()
for _ in range(60):
    z = F.adaptive_avg_pool2d(ft.features(X_few), 1).flatten(1)
    loss = F.cross_entropy(head(z), y_few)
    opt.zero_grad(); loss.backward(); opt.step()
ft.eval()
# [4]
with torch.no_grad():
    z = F.adaptive_avg_pool2d(ft.features(prep(X_shoe_te)), 1).flatten(1)
    print(f"fine-tuned (last block + head): "
          f"{(head(z).argmax(1) == y_shoe_te).float().mean():.1%}")

fine-tuned (last block + head): 88.7%

Fine-tuning edges the frozen probe on this seed and lands in the same tie with scratch. This is consistent with the regime analysis above: adaptation helps at the margin, but no amount of it makes a small target task need imported features.

Keep the pattern, though: this loop, scaled to real backbones and feature-hungry tasks, is how most of applied deep learning ships today. It is also this book’s destination: Part V is what happened when the field noticed that “pretrain big, adapt cheaply” was not a vision trick but the recipe for language too. The BERT moment (Chapter 15) is this section’s idea, taken seriously at scale.

NoteCheck yourself

Close the book for one minute and reconstruct the architecture and transfer choices.

• Why can several small kernels replace one large kernel?
• What changes between a frozen probe, partial fine-tuning, and training from scratch?
• Which evidence here is a mechanism study, and which comes from the pinned Rivanna runs?

9.8 Okay, so — modern CNNs are an optimization story

1. Small kernels, stacked: two \(3\times3\)s see what a \(5\times5\) sees, for \(18C^2\) against \(25C^2\), with twice the nonlinearity — provided width holds constant through the stack. Design in blocks, repeat the block.
2. BatchNorm re-standardizes every channel over the batch and spatial axes, then hands back two knobs (\(\gamma, \beta\)); conv→BN→ReLU with bias=False; train and eval are different machines, so flip the switch. Its per-token cousin, LayerNorm, awaits in Chapter 14. For the graduate treatment of normalization as a choice of invariance, finite-sample state, and derivative path, continue in Normalization Chooses an Invariance.
3. \(1\times1\) convolutions run Chapter 1’s linear model across channels at every pixel: summarize channels, don’t discard them. With global average pooling they fire the flatten head: parameters collapse (218k \(\rightarrow\) 35k) and position-specific weights leave the head, at a clean-accuracy price our small dataset makes visible. Boundaries and downsampling still prevent exact invariance.
4. Depth can hit an optimization wall: our plain 40-layer net couldn’t fit its own training set. The residual connection \(H(x) = F(x) + x\) adds an identity lane to the Jacobian, Chapter 5’s gradient superhighway as infrastructure, and the same-depth twin trains to 100%. Transformer stacks will reuse this residual pattern around each attention and feedforward sublayer.
5. Transfer learning = freeze + probe, or gently fine-tune. Our honest experiment found scratch and the ImageNet probe in a near tie across three seeds at 28-pixel scale. Pretraining is curriculum, gaps are taxed, and small targets may leave little room for imports. The pinned full-scale Rivanna runs report \(88.79\% \pm 0.08\%\) for the frozen probe, \(93.92\% \pm 0.12\%\) for fine-tuning, and \(94.14\% \pm 0.08\%\) for scratch with ResNet-18. They illustrate a richer regime where adaptation repairs most of the feature mismatch. Diagnosing the regime is the skill.

Sources and further reading

• Xiao, Rasul, and Vollgraf, Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms: the target benchmark for the architecture and transfer studies.
• Deng et al., ImageNet: A Large-Scale Hierarchical Image Database: the source benchmark behind the pretrained visual features used here.
• Iandola et al., SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size: the compact pretrained architecture used in the chapter’s first transfer experiment.
• Simonyan & Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition: the small-kernel, repeated-block argument behind VGG.
• Lin, Chen, & Yan, Network In Network: introduces the \(1\times1\) channel mixer and global-average-pooling classifier.
• Ioffe & Szegedy, Batch Normalization: the original normalization method and its train/evaluation distinction.
• He et al., Deep Residual Learning for Image Recognition: the degradation experiment and residual-block solution.
• Huang et al., Densely Connected Convolutional Networks: dense connectivity through feature-map concatenation and transition layers.
• Yosinski et al., How Transferable Are Features in Deep Neural Networks?: a careful empirical account of when transferred features remain useful.
• Hardt and Recht, Patterns, Predictions, and Actions: Datasets: a framework for treating a benchmark as a measured artifact rather than an unquestionable oracle.
• Northcutt, Athalye, and Mueller, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks: a large-scale audit of suspected benchmark-label errors and their effect on model comparison.

Exercises

1. (Pencil.) Generalize the kernel-stacking arithmetic: \(n\) stacked \(3\times3\) layers versus one \((2n{+}1)\times(2n{+}1)\) kernel, at constant width \(C\). Then redo it for a layer that grows channels \(C \rightarrow 2C\): split into two \(3\times3\)s (\(C \rightarrow C \rightarrow 2C\) and \(C \rightarrow 2C \rightarrow 2C\)) and find when the split stops saving parameters.
2. (Pencil.) Verify the Inception bottleneck arithmetic from the callout (\(819{,}200\) versus \(110{,}592\)), then design the cheapest \(1\times1\)-plus- \(5\times5\) pipeline from 128 channels to 64 output channels that keeps the squeeze width at least a quarter of the input width.
3. (Code.) Make VGGSmall’s blocks depthwise separable: replace each \(3\times3\) conv with a per-channel \(3\times3\) (groups=c_in) followed by a \(1\times1\) mixer. Count parameters, retrain, and report the accuracy-per-parameter ratio against the original. (This is MobileNet’s depthwise-separable construction.)
4. (Code.) Ablate BatchNorm: retrain VGGSmall with the BN layers removed, same budget. Compare final accuracy and the first ten epochs’ training accuracy. Then repeat the bn-drift cell’s measurement on both trained networks. Does training itself fix the activation scales?
5. (Code.) Data augmentation as a third road: on the 30-image shoe task, train from scratch with random horizontal flips and ±3-pixel shifts (Chapter 6’s exercise, now as a tool). Does augmentation close the gap to the pinned full-data results further than transfer did? Why might augmentation and transfer help in different regimes?
6. (Audit.) Retrain VGGSmall and NINSmall under seed 6050, then collect the 20 Fashion-MNIST test items on which the models disagree with each other or with the supplied label while at least one model assigns its prediction probability above 0.9. Preserve the images, supplied labels, predictions, and confidences before adjudication. Have a second reader independently label the 20 items, report agreement and a binomial interval for the suspected-label-error share, and list ambiguous cases separately. Explain why confident disagreement is a biased sample of the full test set, in which direction that selection can distort prevalence, and what the audit can still reveal.