What you'll learn

Builds on Modules 1–7

≈6 h · 2 h video · 2 h reading · 2 h coding

Implement QKV attention; visualize attention weights
Build a toy seq2seq (copy/reverse) with attention; interpret alignments
Run a small ablation on attention variants and report findings

Lecture 8 – Attention Mechanism (Part 1)

Lecture 8.2 – Attention Mechanism (Part 2)

Lecture 8.3 – Implementing Attention in seq2seq Decoder

📚 Resources & Live Coding

Course book · primary reading

Research lens · read after the course book

These papers use the regression lens as architecture-design language; begin with the named sections after completing the book derivation.

Alternative reading from Dive into Deep Learning (D2L):

Homework 4: Add cross-attention to your prior GRU-based seq2seq model so that the decoder can attend over all encoder hidden states at each decoding step (same translation task). Report and compare accuracy/bleu vs your previous best.

Seq2Seq Cross-Attention Diagram preview:

Open Live Coding Colab

🧠 Test your understanding

Try each question before revealing the answer — these mirror the ideas the module quiz checks.

Q1.What is the fundamental problem with vanilla seq2seq models that attention mechanisms solve?

They use too much memory
They compress all information into a single fixed-size vector
They train too slowly
They cannot handle variable-length sequences

Show answer ▾

Answer: They compress all information into a single fixed-size vector

Vanilla seq2seq forces all encoder information through a single context vector, creating a bottleneck that causes significant information loss for longer sequences. Attention solves this by maintaining the full memory bank of all encoder hidden states and dynamically selecting relevant information at each decoder step.

Q2.Why do we scale by

1/\sqrt{d_k}

in dot-product attention?

To make computation faster
To keep the variance of scores stable regardless of dimension
To normalize the output
It is arbitrary — any scaling works

Show answer ▾

Answer: To keep the variance of scores stable regardless of dimension

For random vectors with unit variance, $\text{Var}(\mathbf{q}^\top \mathbf{k}) = d_k$ . Without scaling, large $d_k$ leads to huge logits that saturate softmax, causing vanishing gradients. Scaling by $1/\sqrt{d_k}$ ensures $\text{Var}(\mathbf{q}^\top \mathbf{k} / \sqrt{d_k}) = 1$ , maintaining stable gradient flow.

Q3.In cross-attention for seq2seq, do we use a causal mask (preventing attention to future positions)?

Yes, always
No — the decoder can attend to all encoder positions
Only during training
Only for long sequences

Show answer ▾

Answer: No — the decoder can attend to all encoder positions

Cross-attention connects the decoder to the already-available encoder memory, so it needs no future-token causal mask over source positions. It may still use padding or task-specific visibility masks. Encoder self-attention is likewise non-causal in the standard encoder, while decoder self-attention uses a causal mask.

Q4.Why are

\mathbf{W}_Q

and

\mathbf{W}_K

separate matrices in dot-product attention?

To save memory
To allow asymmetric roles: queries ask 'what do I need?', keys answer 'what do I offer?'
They should be the same matrix
For numerical stability

Show answer ▾

Answer: To allow asymmetric roles: queries ask 'what do I need?', keys answer 'what do I offer?'

Separate $\mathbf{W}_Q$ and $\mathbf{W}_K$ create distinct feature spaces for queries and keys. Queries learn what to look for while keys learn how to be found. If they were identical, the model would compute similarity in a single shared space, reducing expressiveness and the ability to learn task-specific attention patterns.

Q5.What fundamental limitation do both vanilla seq2seq AND attention-augmented RNNs share?

They cannot handle long sequences
Sequential computation prevents parallelization during training
They do not use neural networks
They cannot do translation

Show answer ▾

Answer: Sequential computation prevents parallelization during training

Both architectures rely on RNNs where $\mathbf{h}_t$ depends on $\mathbf{h}_{t-1}$ , preventing parallelization across time steps. Attention solved the information bottleneck but not the computational bottleneck. Transformers address this by replacing RNNs entirely with self-attention, enabling massive parallelization of sequence processing.

Q6.Why is scaled dot-product attention usually more accelerator-friendly than additive (Bahdanau) attention?

Additive attention cannot be vectorized over query–key pairs
Dot-product scoring reduces the pairwise core to large matrix multiplications, while additive scoring needs broadcasted sums, a nonlinear activation, and a reduction for every pair
Dot-product attention never uses softmax
Additive attention cannot process batches

Show answer ▾

Answer: Dot-product scoring reduces the pairwise core to large matrix multiplications, while additive scoring needs broadcasted sums, a nonlinear activation, and a reduction for every pair

Both mechanisms can be batched and vectorized. The difference is the computational shape: dot-product scores are one GEMM followed by scaling and softmax, whereas additive attention materializes or fuses pairwise projected sums, applies tanh, and reduces with another vector. That extra elementwise work and memory traffic is generally less friendly to dense-matrix hardware.

Module 7

Module 9