Skip to main content
What you'll learn
Builds on Modules 1–6
≈7 h · 2.5 h video · 2 h reading · 2.5 h coding
  • Implement char-RNN and visualize gradient flow over time steps
  • Build LSTM from scratch; run gate ablation studies
  • Compare LSTM vs GRU performance and computational efficiency
  • Apply gradient clipping and truncated BPTT; analyze training stability
Lecture 7 – Recurrent Neural Networks
Lecture 7.2 – Recurrent Neural Networks (Part 2)
Lecture 7.3 – Recurrent Neural Networks (Part 3)
📚 Resources & Lecture Code

Course book · primary reading

Alternative reading from Dive into Deep Learning (D2L):

Open Colab Notebook
Useful decorators in PyTorch (Colab)
🧠 Test your understanding

Try each question before revealing the answer — these mirror the ideas the module quiz checks.

Q1.According to the lecture, why do autoencoders fail for sequence-to-sequence tasks like translation?
  • They are too slow
  • They suffer from information bottleneck, lack sequential structure, and cannot stream
  • They have too many parameters
  • They only work on images
Show answer ▾

Answer: They suffer from information bottleneck, lack sequential structure, and cannot stream

Autoencoders compress entire variable-length sequences into a fixed-size latent vector, creating an information bottleneck where per-token capacity approaches zero as sequence length increases. They also lack inherent sequential structure and cannot process tokens incrementally.

Q2.What is the key innovation that defines an RNN in terms of parameter usage across time steps?
  • Using different functions at each time step: Ht=ft(Ht−1,Xt)H_t = f_t(H_{t-1}, X_t)
  • Using the same function at every time step: Ht=f(Ht−1,Xt)H_t = f(H_{t-1}, X_t)
  • Using no memory at all
  • Using only the current input
Show answer ▾

Answer: Using the same function at every time step: Ht=f(Ht−1,Xt)H_t = f(H_{t-1}, X_t)

Parameter sharing (also called recurrence) is the defining property of RNNs. By reusing the same weight matrices across all time steps, RNNs make model size independent of sequence length and enable generalization across temporal positions.

Q3.Why do vanilla RNNs suffer from vanishing gradients during backpropagation through time?
  • Too much data
  • Repeated multiplication by Whh\mathbf{W}_{hh} across time steps; if ∥Whh∥<1\|\mathbf{W}_{hh}\| < 1, gradients decay exponentially
  • Too few layers
  • Using tanh activation
Show answer ▾

Answer: Repeated multiplication by Whh\mathbf{W}_{hh} across time steps; if ∥Whh∥<1\|\mathbf{W}_{hh}\| < 1, gradients decay exponentially

Gradients in RNNs involve products of Jacobians along the chain of recurrence. If the weight matrix norm is less than 1, each multiplication shrinks the gradient, leading to exponential decay over long sequences—for example, 0.550≈8.88×10−160.5^{50} \approx 8.88 \times 10^{-16}.

Q4.How does the LSTM cell state update equation solve the vanishing gradient problem compared to vanilla RNNs?
  • It has more parameters
  • The additive cell state update Ct=Ft⊙Ct−1+It⊙C~t\mathbf{C}_t = \mathbf{F}_t \odot \mathbf{C}_{t-1} + \mathbf{I}_t \odot \tilde{\mathbf{C}}_t provides a gradient highway where ∂Ct/∂Ct−1≈Ft\partial \mathbf{C}_t/\partial \mathbf{C}_{t-1} \approx \mathbf{F}_t can stay near 1
  • It trains faster
  • It uses different activations
Show answer ▾

Answer: The additive cell state update Ct=Ft⊙Ct−1+It⊙C~t\mathbf{C}_t = \mathbf{F}_t \odot \mathbf{C}_{t-1} + \mathbf{I}_t \odot \tilde{\mathbf{C}}_t provides a gradient highway where ∂Ct/∂Ct−1≈Ft\partial \mathbf{C}_t/\partial \mathbf{C}_{t-1} \approx \mathbf{F}_t can stay near 1

Unlike vanilla RNNs where gradients multiply through a fixed weight matrix, LSTM's additive cell state update creates a direct gradient path where the forget gate Ft\mathbf{F}_t is learned and dynamic. When Ft≈1\mathbf{F}_t \approx 1, gradients flow nearly unchanged, enabling long-term dependency learning.

Q5.What is the fundamental architectural difference between LSTM and GRU?
  • LSTM is always better
  • LSTM maintains separate cell state Ct\mathbf{C}_t (long-term memory) and hidden state ht\mathbf{h}_t (short-term memory), while GRU merges them into a single state ht\mathbf{h}_t
  • GRU has more parameters
  • LSTM cannot handle long sequences
Show answer ▾

Answer: LSTM maintains separate cell state Ct\mathbf{C}_t (long-term memory) and hidden state ht\mathbf{h}_t (short-term memory), while GRU merges them into a single state ht\mathbf{h}_t

LSTM uses 3 gates (forget, input, output) to control separate cell and hidden states, providing more flexible control. GRU uses only 2 gates (update, reset) and a single state, making it more parameter-efficient while still capturing long-range dependencies effectively.

Q6.What trade-off does truncated backpropagation through time (TBPTT) make?
  • It deletes all context before each chunk
  • It carries recurrent-state values across contiguous chunks but detaches the graph, bounding memory and compute while preventing credit assignment across the boundary
  • It is exact BPTT implemented in smaller batches
  • It guarantees that dependencies beyond 50 tokens never matter
Show answer ▾

Answer: It carries recurrent-state values across contiguous chunks but detaches the graph, bounding memory and compute while preventing credit assignment across the boundary

TBPTT preserves running context by carrying the state value forward, then detaches that value so the backward graph has a finite horizon. The benefit is bounded memory and compute; the cost is that the loss cannot assign gradient credit beyond the chosen horizon. There is no universal 20–50-token cutoff—the right horizon depends on the task and architecture.

Module 6
Module 8