Chapter 7 closed on a question it had spent a whole chapter earning: we built a machine that is local, weight-shared, and translation-equivariant away from boundaries, with nine hand-picked numbers sitting inside it. Nine numbers \(\rightarrow\) nine knobs \(\rightarrow\) we have spent half a book tuning knobs against a loss.
So: declare the kernel a parameter. That single change is this chapter, and it is the whole revolution. Everything else here — channels, padding, stride, pooling — is the supporting cast that turns one learnable filter into a working network. By the end we will have assembled LeNet, the first great convolutional architecture, trained it on our garments, and re-run the cruelest experiment of Chapter 6 to see what the inductive bias actually bought.
Note the shapes. In Part I we flattened every image into a 784-vector before the model ever saw it. That stops now: the images stay rectangular, and they pick up a channel dimension, for reasons that will occupy us shortly.
WarningThe book’s batch convention: NCHW
Throughout this book, every image batch uses PyTorch’s NCHW order: (batch, channels, height, width). A single grayscale batch is therefore shaped (1, 1, 28, 28). Conv2d can also accept an unbatched (C, H, W) input, but keeping the batch dimension explicit makes downstream shapes predictable. The kernels of a conv layer live in a 4-D tensor too: (out-channels, in-channels, height, width). This bookkeeping is the single most common source of errors in convolutional code: when a shape error greets you, count dimensions before touching anything else. Chapter 7’s conv2d helper hid this with x[None, None]; from here on we handle it in the open.
8.1 Nine numbers, learned
Here is a game. I have a feature detector — one of Chapter 7’s zoo, but I won’t tell you which. I will only show you what it does: you hand it images, it hands back feature maps. Your job is to build a detector that matches it.
With the tools of Part I, this is not even hard. A kernel applied by Equation 7.1 is just nine numbers entering a dot product \(\rightarrow\) the output is differentiable with respect to those numbers \(\rightarrow\) gradient descent applies. Start from a random kernel and run the loop we have run since Chapter 1: predict \(\rightarrow\) measure \(\rightarrow\) step.
Code: recover a mystery kernel by gradient descent
step 0 loss 1.37e+00
step 200 loss 5.53e-04
step 400 loss 4.80e-05
step 600 loss 4.42e-06
learned kernel, rounded:
tensor([[-1.0100, -0.0000, 1.0100],
[-1.9800, -0.0100, 1.9900],
[-1.0100, 0.0100, 1.0000]])
max |learned - mystery| = 0.018
The loss falls more than five orders of magnitude, and the learned kernel is the vertical Sobel detector to within 0.02 in every entry. Gradient descent, given nothing but input–output pairs, has reinvented a filter that took computer-vision researchers years of squinting at image gradients to design.
Figure 8.1: The mystery-detector game. Left: the random starting kernel. Middle: the kernel gradient descent found from input–output pairs alone. Right: the hand-crafted vertical Sobel detector from Chapter 7’s zoo. Nobody told the middle panel what an edge is.
TipThe pivot of Part II
Read that cell again, because the entire deep-learning revolution in vision is that cell writ large. Chapter 7 ended with the bottleneck: hand-crafted kernels are limited to patterns a human can articulate in nine numbers — nobody can write down “cat ear.” The fix is not better articulation. It is to stop articulating: put the numbers on the gradient tape and let the task pull the detector it needs out of the data. What if the template were learnable? It is, and the machinery has been in our hands since Chapter 4.
Two honest disclaimers about the game, both of which make the real thing more remarkable, not less. First, the game was rigged: the target came from a known kernel, so a perfect answer existed and the loss surface was a clean convex bowl (the output is linear in \(V\), and squared error in a linear map is Chapter 1’s setting all over again). Second, and more important: in a real CNN, nobody supplies the target feature maps. The only supervision is the classification loss at the far end of the network. Blame flows backward through every layer — exactly the mechanics of Chapter 5 — and the kernels that emerge are whatever detectors the task itself demands. We chose Sobel here; a trained network might discover it needs something no one has named.
One backpropagation detail deserves a sentence, because it is the same fan-out rule we met in Chapter 5. The same nine numbers are used at every position of the slide \(\rightarrow\) every position contributes blame to the shared kernel. Those contributions accumulate, then the loss reduction supplies its scale; F.mse_loss with the default reduction="mean" averages over batch, channel, and spatial elements. Weight sharing is not just a parameter economy. At training time every patch of every image teaches the one shared template.
8.2 From one detector to a layer of detectives
One learnable kernel gives one feature map: one specialist, sweeping the image with one magnifying glass. But Chapter 7’s zoo already told us one specialist is not enough — vertical edges, horizontal edges, blobs, textures all carry information. So we hire a team. The lecture’s image is worth keeping: employ six detectives, each with a different expertise, each producing their own annotated map of the crime scene.
That is all the channel machinery is:
Out-channels: more detectives. A conv layer with 6 output channels owns 6 independent kernels and produces a stack of 6 feature maps.
In-channels: detectives read the whole stack. The next layer’s input is that 6-deep stack, so each of its kernels grows a third dimension: a \(6 \times 5 \times
5\) volume that looks at all six incoming maps simultaneously and sums the evidence into one output map, one bias per output channel at the end.
For an input stack \(X\) with \(C_{\text{in}}\) channels, output map \(o\) is
which is Equation 7.1 run once per input channel and summed — followed, as always since Chapter 3, by a nonlinearity. A convolutional layer is Chapter 3’s recipe with a structured linear map: dot products \(\rightarrow\) add bias \(\rightarrow\) ReLU. Nothing in the recipe changed; only the wiring of the linear part did.
Look at conv2.weight: sixteen kernels, each \(6 \times 5 \times 5\). This is where features start composing. Suppose detective 3 in the first layer is a vertical-edge expert and detective 5 hunts horizontal edges. A second-layer kernel that weights both maps positively in the same neighborhood fires precisely when both experts agree \(\rightarrow\) a corner detector, built from evidence no single first-layer kernel could see. The cross-talk between channels is how edges become corners, corners become textures, and textures become parts. We promised this compositional ladder in Chapter 3; here is the hardware it runs on.
Chapter 7’s zoo can stage the story before any learning enters. Give the first layer two hand-crafted experts — vertical and horizontal edge energy — then hand-build one second-layer kernel that reads both reports at once:
Code: a corner detector built from two edge experts
Figure 8.2: Cross-talk staged by hand. Two first-layer experts report edge energy (absolute Sobel response) on a square. One second-layer kernel spanning both channels (all-positive weights; Equation 8.1 with two input channels) fires hardest exactly where both experts agree — the four corners, a feature neither channel could see alone.
The four brightest cells of the right panel are exactly the square’s four corners: a corner detector, assembled from experts that individually know nothing about corners. In a trained CNN, gradient descent builds combinations like this — and far stranger ones — on its own.
Tipnn versus F, appliances versus recipes
The lecture’s analogy: nn modules are appliances — they have knobs and memory, and PyTorch carries their state around for you (nn.Conv2d owns its kernels and biases as nn.Parameters, registered for autograd and visible to the optimizer). F functions are recipes — stateless, everything passed in by hand. In Chapter 7 the kernel was ours, so F.conv2d was the honest choice; now the kernel is the layer’s business, so nn.Conv2d is. Use appliances for anything with parameters, recipes for everything else (F.relu, F.max_pool2d).
8.3 Padding and stride: the bookkeeping with a payoff
Chapter 7 left an administrative debt: a \(k \times k\) kernel on an \(n \times n\) image yields only \((n-k+1) \times (n-k+1)\) outputs, because the window must stay inside the frame. For one layer, a nuisance. For a deep stack, fatal: \(28 \rightarrow 24
\rightarrow 20 \rightarrow \cdots\) and the feature map shrinks toward nothing, with edge pixels contributing to ever fewer windows along the way.
The fix is blunt: padding. Add a border of zeros around the input so the window can center itself on every real pixel. For a \(5 \times 5\) kernel, a 2-pixel border returns exactly the input size \(\rightarrow\) that is why conv1 above says padding=2. Padding decouples “how deep can the network be” from “how fast do the maps shrink,” and that decoupling is what makes deep stacks possible at all.
Sometimes, though, we want to shrink — coarser maps mean less computation and a wider view per pixel. Then we shrink on purpose with stride: slide the window \(s\) pixels at a time instead of one. Both knobs live in one formula, worth memorizing because you will run it in your head for every architecture you ever read:
The numerator is the padded length minus the window, i.e. how far the window can travel; dividing by the step and flooring counts the stops; \(+1\) counts the starting position. The lecture’s three worked cases, verified:
Code: the output-size formula, three regimes
x8 = torch.randn(1, 1, 8, 8)for p, s in [(0, 1), (1, 1), (1, 2)]: out = F.conv2d(x8, torch.randn(1, 1, 3, 3), padding=p, stride=s) n_out = (8+2* p -3) // s +1print(f"n=8, k=3, p={p}, s={s}: formula {n_out} torch {tuple(out.shape[2:])}")
n=8, k=3, p=0, s=1: formula 6 torch (6, 6)
n=8, k=3, p=1, s=1: formula 8 torch (8, 8)
n=8, k=3, p=1, s=2: formula 4 torch (4, 4)
8.4 Pooling: trading exact location for local tolerance
Now for the deepest design decision in the network, and it starts from the asset Chapter 7 banked. Under matched boundary rules, convolution is equivariant on the interior: shift the garment, and every feature map shifts in lockstep. The detective’s map of clues faithfully tracks the scene. But the network’s final job is not mapping clues; it is a verdict. Is this a trouser? The answer must not depend on where the trouser stands. The feature-finding stage wants equivariance (where things are matters); the decision stage wants invariance (only what was found matters). Equivariance is what we have; invariance is what the classifier head needs \(\rightarrow\) something must reduce sensitivity to location.
That converter is pooling. Slide a window over each feature map (no parameters this time) and summarize each neighborhood by one number:
Max pooling asks: what is the strongest clue here? Keep the loudest activation, discard the rest.
Average pooling asks: what is the general vibe of this neighborhood? Smoother, but a single sharp clue gets diluted by quiet neighbors.
Code: max and average pooling on a known grid
t = torch.arange(16.).reshape(1, 1, 4, 4)print(f"input:\n{t.squeeze()}")print(f"max pool 2x2:\n{F.max_pool2d(t, 2).squeeze()}")print(f"avg pool 2x2:\n{F.avg_pool2d(t, 2).squeeze()}")
The common configuration is the one above: \(2 \times 2\) windows with stride 2, non-overlapping, halving each spatial dimension. Max is the default choice in practice, precisely because feature detection is about the strongest evidence, not the committee average.
How can this buy local tolerance? Start with a deliberately clean construction in which a shift becomes invisible:
Code: one constructed shift that pooling cannot see
scene = torch.zeros(1, 1, 4, 4)scene[0, 0, 1, 0] =9.0# a strong clue, upper-left regionscene[0, 0, 2, 2] =3.0# a weaker clue, lower-right regionshifted = torch.zeros_like(scene)shifted[0, 0, 1, 1] =9.0# both clues moved one pixel rightshifted[0, 0, 2, 3] =3.0print(f"pooled original: {F.max_pool2d(scene, 2).squeeze().tolist()}")print(f"pooled shifted: {F.max_pool2d(shifted, 2).squeeze().tolist()}")
pooled original: [[9.0, 0.0], [0.0, 3.0]]
pooled shifted: [[9.0, 0.0], [0.0, 3.0]]
Both clues moved, yet these particular pooled maps are identical: each isolated maximum stayed inside its \(2 \times 2\) window. That equality belongs to this construction, not to max pooling in general. A shift can change which values share a window, which maximum wins, or what crosses the boundary. Pooling can therefore buy local tolerance to jitter, but the amount depends on the activations and alignment; the end of this chapter measures it rather than assuming it.
WarningTolerance has a price tag
Pooling can buy tolerance by throwing away position. After two rounds, the network knows a strong vertical edge exists in a region; it no longer knows exactly where. For classification this is the right trade — but it is a trade, and it will come due twice in this book. Deeper in Part II, tasks that need precise locations will have to be more careful with resolution. And in Part IV the tables turn completely: self-attention (Chapter 14) is so thoroughly position-agnostic that we will have to pay to put position back in. Remember, when we get there, that position was something we once discarded on purpose.
8.5 How far a deep neuron sees
There is one more consequence of stacking, and it quietly resolves the tension Chapter 6 left us with. Every kernel here is tiny — \(5 \times 5\), resolutely local. Yet recognizing a garment is a global judgment. Where does globality come from?
From depth. A neuron in the first feature map sees a \(5 \times 5\) patch of the image: its receptive field. A neuron one layer up sees a \(5 \times 5\) patch of feature maps, each pixel of which already summarizes a patch below \(\rightarrow\) its receptive field on the original image is wider. Pooling accelerates this: after a \(2\times2\)/stride-2 pool, neighboring pixels in the pooled map stand two original pixels apart, so every later window covers twice the ground. For our upcoming network the ledger reads: conv1 sees \(5 \times 5\)\(\rightarrow\) pooling nudges it to \(6 \times
6\)\(\rightarrow\) conv2’s \(5\times5\) window, at stride-2 spacing, expands it to \(14
\times 14\)\(\rightarrow\) the final pool reaches \(16 \times 16\) — over half the frame in every direction, from nothing but \(5 \times 5\) looks.
Figure 8.3: What one neuron sees, by depth. Boxes mark the receptive field of a single unit in each stage of our network, centered on the same spot: conv1 sees a 5×5 patch, conv2 sees 14×14, the final pooled cell 16×16. Local wiring, increasingly global sight; the dense head then reads all 25 such overlapping views at once.
This is the compositional hierarchy of Chapter 3 given a strong architectural invitation. The MLP could represent features-of-features but nothing encouraged it to; locality and growing receptive fields encourage later layers to compose earlier responses. Learned channels often progress from edge-like patterns to textures and parts, but the architecture does not force those human labels. The CNN buys global sight gradually, paying for it with depth; hold that thought until Part IV, where attention will buy global sight in a single step and pay for it differently (Chapter 13).
8.6 LeNet: the whole machine
Time to assemble. The canonical blueprint is LeNet, Yann LeCun’s digit-reading network, and its rhythm is the CNN design pattern to this day: convolve, shrink, deepen; repeat; then decide. Spatial size falls (28 \(\rightarrow\) 14 \(\rightarrow\) 5) while feature depth grows (1 \(\rightarrow\) 6 \(\rightarrow\) 16): the network trades “where” for “what” stage by stage, until a small dense head reads the final \(16 \times
5 \times 5\) summary and votes.
Every line is a tool we already own: Equation 8.1 twice, Equation 8.2 silently fixing padding=2, pooling twice, then Chapter 3’s MLP as the decision head — and, per Chapter 2’s advice, the model returns raw logits and lets F.cross_entropy handle the probabilities.
NoteLeNet, then and now
The LeNet family (LeCun and colleagues, 1989–1998) began by reading handwritten ZIP codes for the US Postal Service, and its mature form, LeNet-5, went on to read a substantial share of America’s bank checks — convolutional networks were literally cashing checks decades before they were fashionable. Our variant keeps the classic skeleton but upgrades the internals with Part I’s verdicts: ReLU where the original used sigmoid-family activations (Chapter 3, Chapter 5), max pooling where it used average-flavored subsampling, Adam as the optimizer. The lecture’s live session adds one more modern stabilizer, batch normalization, which we meet properly in Chapter 9.
Before training, the parameter audit — because parameter economy was half of Chapter 7’s sales pitch, and Chapter 6’s MLP is the yardstick:
Code: parameter audit, LeNet vs. Chapter 6’s MLP
def make_mlp() -> nn.Sequential:return nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))lenet, mlp = LeNet(), make_mlp()for name, p in lenet.named_parameters():print(f"{name:14s}{str(tuple(p.shape)):16s}{p.numel():>7,}")n_conv =sum(p.numel() for n, p in lenet.named_parameters() if"conv"in n)n_lenet =sum(p.numel() for p in lenet.parameters())n_mlp =sum(p.numel() for p in mlp.parameters())print(f"\nLeNet: {n_lenet:,} parameters ({n_conv:,} of them convolutional)")print(f"MLP: {n_mlp:,} parameters")
Two readings of this table. Against the MLP: LeNet carries 61,706 parameters to the MLP’s 203,530 (less than a third) while performing far richer spatial computation. Weight sharing is doing exactly what Chapter 7’s matrix view promised. Within LeNet: look where the parameters live. The two conv layers, the part that actually sees, own 2,572 parameters (about 4%) while the dense head hoards the rest. That imbalance is a design smell we will fix in Chapter 9, where modern architectures go deeper in convolution and lighter in the head.
Now train both models under one shared optimization recipe: same fitting split, optimizer, minibatch size, number of epochs, and therefore number of parameter updates. The comparison does not match parameter count or floating-point work; a convolution and a dense layer spend different computation per update. Architecture is the intended difference, compute is not controlled:
Read the clean-validation column honestly: the MLP scores 75.5% and LeNet 74.5% in this single seeded run. A one-point difference is descriptive, not an uncertainty estimate, so it does not establish which architecture has higher expected clean accuracy. Chapter 6 already told us why a landslide is unlikely here: centered garments do not expose the MLP’s main weakness. Both models nearly interpolate the fitting split. Our small-data regime makes a narrower point: LeNet reaches comparable clean accuracy with less than a third of the parameters. The shift experiment now asks whether it learned a different kind of representation.
8.7 The rematch
Chapter 6 convicted our MLP with one experiment: shift every validation garment two pixels right, and accuracy fell off a cliff — the full-frame templates kept scoring sleeves against background. We then promised that an architecture built on locality and weight sharing would face the same trial. The rematch:
Code: accuracy vs. shift, both models
def shift_right(X: torch.Tensor, px: int) -> torch.Tensor: out = torch.zeros_like(X)if px ==0:return X.clone() out[..., px:] = X[..., :-px]return outshifts =list(range(5))acc_mlp = [accuracy(mlp, shift_right(X_val, s), y_val) for s in shifts]acc_cnn = [accuracy(lenet, shift_right(X_val, s), y_val) for s in shifts]for s in shifts:print(f"shift {s}px: MLP {acc_mlp[s]:.1%} LeNet {acc_cnn[s]:.1%}")plt.figure(figsize=(5.5, 3.2))plt.plot(shifts, acc_mlp, "o-", color="#E57200", lw=2, ms=7, label="MLP (Ch. 6)")plt.plot(shifts, acc_cnn, "s-", color="#232D4B", lw=2, ms=7, label="LeNet")plt.axhline(0.1, ls=":", color="#B8B8A8")plt.xlabel("shift (pixels right)"); plt.ylabel("validation accuracy")plt.ylim(0, 0.9); plt.xticks(shifts); plt.legend()plt.tight_layout(); plt.show()
Figure 8.4: The validation rematch. Both architectures use the same optimizer schedule, though not matched compute. At two pixels the MLP falls from 75.5% to 42.0%, while LeNet falls from 74.5% to 57.5%. The dotted line is chance.
There is the payoff, and it is worth stating precisely. At two pixels the MLP falls from 75.5% to 42.0%, while LeNet falls from 74.5% to 57.5%. That is one deterministic validation split, not a sampling distribution, but the within-run contrast is large. The mechanism is everything this chapter built: the kernels travel with the garment (equivariance), so the features are still found; pooling forgives some sub-window jitter (local insensitivity); only the coarse layout entering the head changes at all.
And yet the navy curve falls too. Also worth stating precisely, because it teaches the architecture’s fine print. Two rounds of 2× pooling buy tolerance measured in a few pixels, not unlimited; and after flatten(1), the dense head reads the final \(5
\times 5\) grid positionally — a four-pixel input shift moves features roughly one full cell in that grid, and the head is as brittle to cell-level shifts as Chapter 6’s MLP was to pixel-level ones. The inductive bias did not abolish the cliff; it moved it outward and flattened it into a slope. Reducing the remaining sensitivity is the next chapter’s business, where we revisit depth and the position-sensitive head.
One last look inside, because Chapter 6 also showed us what the MLP’s first layer looked like — global, smeared, garment-shaped templates (Chapter 6) — and promised that structure would change:
Figure 8.5: What LeNet learned to look for. Top: six first-layer 5×5 kernels (red positive, blue negative). Bottom: their feature maps on one validation garment. The maps respond differently, but at this toy scale several kernels are diffuse and their semantics should not be over-interpreted. Compare their local support with Chapter 6’s full-frame templates.
Six small, local kernels produce distinct response maps on the same garment. Some show directional structure; others remain noisy enough that naming a detector would tell more story than this one run supports. Nobody designed them. The classification loss, pulling blame backward through Equation 8.1 for roughly 1,200 optimizer steps, chose these local measurements.
8.8 One sealed test check
We have now fixed both architectures, the optimizer schedule, the shift size, and the two reported metrics. We refit each model on all 1,200 development images, then open the 600-image test set once:
The sealed check confirms the validation diagnosis. Clean accuracy is 82.2% for the MLP and 82.5% for LeNet, a descriptive 0.3-point gap from one seed. Under the predeclared two-pixel shift, the gap becomes 41.8% versus 62.3%.
8.9 Okay, so —
One change made the revolution: the kernel’s numbers became parameters. Gradient descent can recover a hand-crafted detector from data (our rigged game) and, more importantly, discover detectors nobody designed, supervised only by the task loss at the far end.
Channels turn one detector into a team: out-channels stack specialists’ feature maps; in-channels let the next layer read all maps at once (Equation 8.1), which is how edges compose into corners and parts.
Padding and stride are the shape budget: \(n_{\text{out}} = \lfloor (n + 2p -
k)/s \rfloor + 1\) (Equation 8.2). Padding lets networks go deep without shrinking to nothing; stride shrinks on purpose.
Pooling can turn equivariance into local shift tolerance — the strongest clue per window, some position discarded. It is a purchase, and position is the currency.
Receptive fields grow with depth: \(5 \rightarrow 6 \rightarrow 14 \rightarrow
16\) in our LeNet. Local wiring earns global sight gradually; the hierarchy Chapter 3 could only hope for is now encouraged by architecture.
The verdict: on the sealed test set, one seed puts clean accuracy at 82.2% for the MLP and 82.5% for LeNet; this does not establish an expected clean-accuracy difference. Under the predeclared two-pixel shift, the same models score 41.8% and 62.3%. The bias did not add capacity; it added the right constraints, exactly Chapter 6’s prescription.
(Pencil.) Run Equation 8.2 through LeNet layer by layer, starting from \(28
\times 28\): verify every shape comment in the LeNet class, then verify the parameter counts of conv1 and conv2 from the table (weights and biases).
(Pencil.) Verify the receptive-field ledger \(5 \rightarrow 6 \rightarrow 14
\rightarrow 16\). (Track two numbers per stage: the field size, and the jump — the spacing in input pixels between neighboring units — which each stride-2 pool doubles.) Then show that no convolutional or pooling unit in LeNet ever sees the full \(28 \times 28\) frame. Where does globality finally enter the network?
(Code.) Play the mystery-detector game with the \(3 \times 3\) blur kernel as the target. Does gradient descent recover it as cleanly as Sobel? Now rerun with only 8 training images instead of 256. What happens to the recovered kernel, and what does that say about data volume versus parameter identifiability?
(Code.) Swap LeNet’s max pooling for average pooling and retrain. Compare clean accuracy and the shift curve. Which changes more, and does the result match the “strongest clue vs. general vibe” story?
(Pencil.) The final dense head receives 25 spatial positions per feature map. Explain why this can reintroduce shift sensitivity after equivariant convolutions. Describe what information a position-insensitive head would keep and discard; we will build one in Chapter 9.
