Skip to main content
What you'll learn
Builds on Modules 1–5
≈5 h · 1.5 h video · 1.5 h reading · 2 h coding
  • Implement an autoencoder from scratch; visualize learned representations
  • Build a CNN encoder-decoder for image tasks
Lecture 6 – Encoder-Decoder Architectures
Lecture 6.2 – Encoder-Decoder Architectures (Part 2)
Lecture 6.3 – Encoder-Decoder Architectures (Part 3)
📚 Resources & Lecture Code

Course book · primary reading

Alternative reading from Dive into Deep Learning (D2L):

The Colab notebook contains the lecture code for Module 6 (encoder–decoder architectures). Use it to follow along with the in-class demos.

Open Colab Notebook
🧠 Test your understanding

Try each question before revealing the answer — these mirror the ideas the module quiz checks.

Q1.Why can a low-dimensional bottleneck encourage an autoencoder to learn useful structure?
  • Because the encoder has more parameters than the input
  • Because k≪dk \ll d limits the information that can pass directly through the latent code
  • Because the decoder uses different weights than the encoder
  • Because autoencoders only work on labeled data
Show answer ▾

Answer: Because k≪dk \ll d limits the information that can pass directly through the latent code

A narrow bottleneck creates pressure to compress recurring structure instead of copying every coordinate directly. It is an inductive bias, not a guarantee: an expressive network can still memorize a finite training set, so generalization must be checked on held-out data.

Q2.Under what conditions does a linear autoencoder recover the PCA principal subspace?
  • For any nonlinear decoder
  • For centered data, an undercomplete linear encoder-decoder, squared reconstruction loss, and a global optimum
  • Only when the data already has zero reconstruction error
  • Only with classification labels
Show answer ▾

Answer: For centered data, an undercomplete linear encoder-decoder, squared reconstruction loss, and a global optimum

Under those conditions, the optimal reconstruction projects onto the same leading principal subspace as PCA. Individual encoder coordinates need not equal the uniquely ordered PCA vectors: invertible changes of basis inside the latent space can represent the same projection.

Q3.In PCA viewed as an encoder-decoder, what is the encoding step?
  • z=x+Vk\mathbf{z} = \mathbf{x} + \mathbf{V}_k
  • z=Vkx\mathbf{z} = \mathbf{V}_k \mathbf{x}
  • z=VkTx\mathbf{z} = \mathbf{V}_k^T \mathbf{x}
  • z=xVkT\mathbf{z} = \mathbf{x} \mathbf{V}_k^T
Show answer ▾

Answer: z=VkTx\mathbf{z} = \mathbf{V}_k^T \mathbf{x}

The transpose projects the input onto the principal component directions. Reconstruction then uses x^=Vkz\hat{\mathbf{x}} = \mathbf{V}_k \mathbf{z}, recovering data in the original space.

Q4.What is the most defensible claim about an autoencoder bottleneck?
  • It guarantees semantic features
  • It encourages selective compression, whose usefulness must be evaluated
  • It prevents memorization in every architecture
  • It always removes noise
Show answer ▾

Answer: It encourages selective compression, whose usefulness must be evaluated

The bottleneck constrains the representation, but the reconstruction objective does not specify which information is semantically useful. Inspect reconstructions and test the representation on held-out or downstream tasks before making that claim.

Q5.Why is the standard autoencoder reconstruction objective called self-supervised?
  • The input itself supplies the reconstruction target
  • A pretrained classifier supplies semantic labels
  • The decoder is never trained
  • The method can only use perfectly clean data
Show answer ▾

Answer: The input itself supplies the reconstruction target

The training pair is constructed automatically as (x,x)(x,x): no external semantic annotation is required. Autoencoders are often grouped under unsupervised representation learning, while self-supervised names the more precise mechanism that creates their target.

Module 5
Module 7