Skip to main content
What you'll learn
Builds on Modules 1–11
≈8 h · 2.5 h video · 3 h reading · 2.5 h coding
  • Explain symmetric image–text contrastive learning and audit cross-modal retrieval
  • Implement a tiny VAE; explore the latent space qualitatively
  • Train a small GAN; apply simple stability tweaks; assess samples
  • Explain forward/reverse diffusion and demonstrate sampling
  • Briefly discuss evaluation metrics and ethical considerations
12.1 – Multimodal Learning

Vision-language alignment, contrastive objectives (e.g., CLIP), and fusion strategies for building multimodal systems.

Slide preview

View SlidesOpen Colab Notebook
12.2 – Diffusion Models

Noise schedules, forward/reverse processes, and sampling recipes that power modern generative diffusion pipelines.

Slide preview

View SlidesOpen Colab Notebook
12.3 – Variational Autoencoders (VAE)

Latent variable modeling with encoder/decoder pairs, ELBO optimization, and practical VAE architectures.

Slide preview

View SlidesOpen Colab Notebook
12.4 – Generative Adversarial Networks (GAN)

Adversarial training, loss variants, and practical tips for stabilizing GANs for image generation.

Slide preview

View SlidesOpen Colab Notebook
📚 Resources & Lecture Code

Course book · primary reading

🧠 Test your understanding

Try each question before revealing the answer — these mirror the ideas the module quiz checks.

Q1.What is the primary goal of contrastive learning in multimodal AI systems like CLIP?
  • To generate new images from text descriptions
  • To map different modalities into a shared embedding space
  • To classify images into predefined categories
  • To compress images into smaller representations
Show answer ▾

Answer: To map different modalities into a shared embedding space

Contrastive learning aligns different modalities (e.g., images and text) by maximizing similarity between semantically related pairs while minimizing similarity between unrelated pairs, creating a joint representation space essential for zero-shot transfer.

Q2.Why does predicting a masked image patch directly using MSE loss typically result in blurry images, and how does diffusion solve this?
  • MSE is computationally expensive; diffusion uses faster gradient descent
  • A one-shot MSE predictor averages competing completions; diffusion learns denoising information across noise levels and samples through repeated refinement
  • MSE cannot handle color images; diffusion learns in latent space
  • MSE requires too much training data; diffusion uses smaller datasets
Show answer ▾

Answer: A one-shot MSE predictor averages competing completions; diffusion learns denoising information across noise levels and samples through repeated refinement

When several completions are plausible, a deterministic one-shot squared-error prediction tends toward their conditional mean, which can look blurry. A diffusion model learns a denoising or score field at many noise levels and follows a stochastic reverse process, so separate runs can realize different plausible modes.

Q3.What is the score function in diffusion models, and what does Tweedie's formula reveal about its relationship to noise prediction?
  • The loss function used for training; it is independent of noise prediction
  • The gradient of log-probability ∇xlog⁡p(x)\nabla_{\mathbf{x}} \log p(\mathbf{x}); predicting noise is mathematically equivalent to learning the score function
  • The probability of an image being real; it requires a separate classifier
  • The reconstruction error; noise prediction and score learning are competing objectives
Show answer ▾

Answer: The gradient of log-probability ∇xlog⁡p(x)\nabla_{\mathbf{x}} \log p(\mathbf{x}); predicting noise is mathematically equivalent to learning the score function

The score function points toward high-probability regions in data space. Tweedie's formula shows that ∇xtlog⁡pt(xt)=−11−αˉtE[ϵ∣xt]\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t) = -\frac{1}{\sqrt{1-\bar{\alpha}_t}} \mathbb{E}[\boldsymbol{\epsilon} \mid \mathbf{x}_t], meaning noise prediction directly provides the score needed for sampling via Langevin dynamics.

Q4.What is the key architectural difference between a standard autoencoder and a VAE, and what problem does it solve?
  • VAEs use deeper networks; this improves reconstruction quality
  • VAEs encode data as probability distributions rather than fixed points; this enables sampling and a well-structured latent space
  • VAEs use convolutional layers; this reduces computational cost
  • VAEs require labeled data; this enables supervised learning
Show answer ▾

Answer: VAEs encode data as probability distributions rather than fixed points; this enables sampling and a well-structured latent space

Standard autoencoders map inputs to fixed latent vectors; VAEs map inputs to distributions (mean and variance), then sample z∼N(μ,σ2)\mathbf{z} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma}^2). The KL regularization forces the latent space near N(0,I)\mathcal{N}(0,I), enabling generation by sampling from the prior without the encoder.

Q5.At the theoretical optimum of the original GAN objective, what happens when pg=pdatap_g=p_{data}?
  • The optimal discriminator outputs 0.5 wherever either distribution has support
  • The discriminator memorizes every training image
  • The generator's loss must be zero
  • The batch size becomes irrelevant
Show answer ▾

Answer: The optimal discriminator outputs 0.5 wherever either distribution has support

For the original minimax objective and an optimal discriminator, matching the generated and data distributions makes the density ratio equal, so D∗(x)=1/2D^*(x)=1/2. Observing a value near 0.5 in an actual run is not by itself proof of equilibrium: the discriminator could also be undertrained or otherwise ineffective.

Module 11
End of course