What you'll learn

Builds on Modules 1–9

≈6.5 h · 2 h video · 20 min frontier outline · 2 h reading · 2 h coding

Implement patchify + linear embedding and explain ViT's data-versus-bias trade-off
Contrast BERT, T5, and GPT through their pretraining objectives and attention masks
Explain knowledge distillation and why a CNN teacher can lend useful inductive bias to ViT
Use scaling laws to reason jointly about parameters, training tokens, and compute
Compare KV retention, factorized sufficient statistics, and bounded online state in regression vocabulary; critique test-time planning under a compute-matched rematch

10.1 Vision Transformer (ViT)

10.2 Pretrained Transformer Models

10.3 Scaling of Decoder Transformer Models

10.4 – Test-time Regression and Test-time Control

A 20-minute research-frontier segment: start with the precise local-constant regression statement, compare three memory contracts, then ask what changes when a layer evaluates possible futures before predicting. The memory-versus-planning split is an architectural bet, not settled science; the System 1/System 2 language in the paper is only a loose metaphor.

20-minute lecture outline

0–3 min
Precision before the slogan
At a query $q$ , softmax attention solves a local-constant kernel fit: minimize $\frac12\sum_\tau \kappa(q,k_\tau)\lVert c-v_\tau\rVert^2$ over one constant $c$ . An unrestricted function with $w=1$ and no regularizer would not, by itself, produce this weighted average.
3–7 min
One objective, four dials
Read $M_t=\arg\min_M\;\frac12\sum_{\tau\le t}w_{t,\tau}\lVert M(k_\tau)-v_\tau\rVert^2+\Omega(M)$ . Track four choices: learned key/value/query views, history weights, regularization, and the solver. The solver is the new rung.
7–12 min
The corrected memory spectrum
Exact softmax keeps the prefix’s $K/V$ pairs as the nonparametric regressor’s dataset. A factorized kernel can instead keep the exact sufficient-statistic pair $(S_t,z_t)$ . An online linear update keeps a bounded matrix state, as in the Delta-rule view. These are different statistical contracts—no state compression versus fixed-size, lossy compression—not settled equivalences.
12–15 min
What selection can learn
Use Mamba-style selectivity only as an interpretive bridge: token-dependent state retention and input injection play roles analogous to a learned forgetting and update schedule. That analogy does not derive Mamba from the regression objective or cover selective-scan implementation details.
15–17 min
Two minutes of scalar LQR
Contrast fitting the prefix with evaluating a finite horizon. Step backward through $P\leftarrow q_c+a^2P-(abP)^2/(r_c+b^2P)$ , then inspect the first gain $K_1=-(r_c+b^2P)^{-1}bPa$ . Changing the horizon can change the first action before any new token is emitted.
17–20 min
A rematch, not a referendum
End with the discussion prompt below: separate parameter matching, generated-token matching, and total inference-compute matching before interpreting a frontier result.

Discussion-leading prompt

Read Table 2 and §E.2 of Wang et al. (2026). Which comparisons are approximately parameter-matched, which share the prompting and output-token protocol, and which are not matched for total inference compute? Design a horizon- $T$ rematch that fixes the backbone, data, prompt, sampling, hardware, and output cap; then equalizes measured FLOPs or wall-clock cost per item, including the planner’s internal work. What accuracy-versus-compute curve, uncertainty, and memory-only baseline would you report? Treat this as a rematch, not a referendum.

📚 Resources, Readings & Colab

Required reading · course book

Module 10 – Transformer Models in Vision and Text