What you'll learn

Builds on Modules 1–8

≈6 h · 2 h video · 2 h reading · 2 h coding

Implement a transformer block and train a tiny Transformer
Compare positional encodings; pre- vs post-norm; analyze patterns
Use warmup and dropout; document stability/perf. impacts

Lecture 9 – Transformer

Lecture 9.2 – Transformer (Part 2)

📚 Resources & Lecture Code

Course book · primary reading

Ch. 14 · Self-Attention and the Transformer

Research lens · read after the course book

These papers use the regression lens as architecture-design language; begin with the named sections after completing the book derivation.

Alternative reading from Dive into Deep Learning (D2L):

Ch. 11 Attention Mechanisms and Transformers

Suggested reading:

Vaswani et al., "Attention Is All You Need" (NeurIPS 2017)

Open Colab Notebook

🧠 Test your understanding

Try each question before revealing the answer — these mirror the ideas the module quiz checks.

Q1.What is the computational complexity of self-attention with respect to sequence length

n

and embedding dimension

d

$O(nd^2)$
$O(n^2d)$
$O(nd)$
$O(n^2d^2)$

Show answer ▾

Answer: $O(n^2d)$

Computing $QK^T$ requires an $(n \times d)(d \times n)$ multiplication, giving $O(n^2d)$ work and an $n \times n$ attention matrix. That quadratic pressure makes long contexts costly, although sparse, fused, and memory-efficient implementations can extend practical context lengths substantially.

Q2.In the self-attention formula

\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

, why do we scale by

\sqrt{d_k}

To make the computation faster
To prevent numerical overflow in the softmax
To counteract the large magnitude of dot products when $d_k$ is large
To normalize the output values

Show answer ▾

Answer: To counteract the large magnitude of dot products when $d_k$ is large

Without scaling, large $d_k$ produces large dot products $QK^T$ , pushing softmax into regions with vanishing gradients. Scaling by $\sqrt{d_k}$ keeps the variance of dot products stable, ensuring non-extreme attention weights and healthy gradient flow during backpropagation.

Q3.According to the lecture, what is the primary role of Feed-Forward Networks (FFNs) in the Transformer (the ``librarian analogy'')?

To route information between tokens (the librarian finding aisles)
To act as dense, associative memory storing compressed knowledge (the content on shelves)
To normalize activations across tokens
To implement dropout for regularization

Show answer ▾

Answer: To act as dense, associative memory storing compressed knowledge (the content on shelves)

Transformers separate concerns: attention performs content-based routing across tokens; FFNs perform token-local, non-linear computation and store fact/phrase associations. FFN layers (width $\approx 4d_{\text{model}}$ ) contain most parameters, providing high-capacity storage for memorized knowledge about next-token predictions.

Q4.Which key property enables relative position to be expressed linearly from sinusoidal encodings?

Normalization of all encodings to unit magnitude
A shift by $\delta$ positions is a rotation: $\begin{bmatrix}P_{i+\delta,2j} \\ P_{i+\delta,2j+1}\end{bmatrix} = R_{\delta\omega_j}\begin{bmatrix}P_{i,2j} \\ P_{i,2j+1}\end{bmatrix}$
Frequencies are multiples of a common base tone
$\sin^2 + \cos^2 = 1$ always holds

Show answer ▾

Answer: A shift by $\delta$ positions is a rotation: $\begin{bmatrix}P_{i+\delta,2j} \\ P_{i+\delta,2j+1}\end{bmatrix} = R_{\delta\omega_j}\begin{bmatrix}P_{i,2j} \\ P_{i,2j+1}\end{bmatrix}$

Trigonometric identities yield rotation matrices that depend only on offset $\delta$ , giving a linear transformation for relative shifts. This enables the Transformer to learn relative positional relationships without explicit position indices, unlike learned embeddings.

Q5.Why use a geometric (not linear) progression of frequencies in sinusoidal positional encoding?

It is computationally faster to compute
It efficiently spans many octaves, capturing both local and global structure
It reduces the embedding dimension needed
It equalizes importance across all frequency dimensions

Show answer ▾

Answer: It efficiently spans many octaves, capturing both local and global structure

Geometric spacing (frequencies via $10000^{-2j/d}$ ) covers a wide frequency range without redundancy, yielding multi-scale positional features. Linear spacing would either waste dimensions on low frequencies or leave large gaps at high frequencies, failing to capture fine-grained local structure.

Q6.In multi-head attention, why do we use multiple heads instead of a single attention head?

Multiple heads process the sequence in parallel, making computation faster
Single attention heads average information from all positions, which dilutes the model's ability to focus on different semantic relationships simultaneously; multiple heads operate on different representation subspaces to preserve resolution
Multiple heads allow different layers of the network to attend to different tokens
Multiple heads enable the use of different values of $d_k$ for better numerical stability

Show answer ▾

Answer: Single attention heads average information from all positions, which dilutes the model's ability to focus on different semantic relationships simultaneously; multiple heads operate on different representation subspaces to preserve resolution

The lecturer explains that a single attention head computes a weighted average of value vectors, which can lose information when compressing multiple relationship types (semantic roles, positional context) into one output. Using multiple heads, each attending to a different subspace of the embedding dimension, allows the model to simultaneously capture different linguistic phenomena without dilution.

Module 8

Module 10