What you'll learn

None — start here

≈6 h · 2 h video · 2 h reading · 2 h coding

Compute matrix–vector products and visualize linear transforms
Explain XOR nonlinearity and demonstrate failure → fix with ReLU
Implement a minimal MLP with PyTorch tensors/modules and describe SGD updates

Lecture 1 – From Linear Regression to Neural Networks

Lecture 1.2 – Linear Regression via Neural Networks in Python

📚 Resources & Lecture Code

Course book · primary reading

Alternative reading from Dive into Deep Learning (D2L):

These Colab notebooks contain the lecture codes demonstrated in class and help you with the homework 1. Read the instructions at the top and run the cells in order.

Open Colab Notebook 1

Open Colab Notebook 2

🧠 Test your understanding

Try each question before revealing the answer — these mirror the ideas the module quiz checks.

Q1.The diagram illustrating the relationship between model complexity, training loss, and generalization loss shows an 'Overfitting' region. What does this region represent?

A situation where the model is too simple to capture the underlying patterns in the data.
The optimal point where the model performs equally well on training and unseen data.
A situation where the model performs very well on the training data but poorly on new, unseen data.
The point where the training loss and generalization loss are both high.

Show answer ▾

Answer: A situation where the model performs very well on the training data but poorly on new, unseen data.

Overfitting occurs when a model learns the training data too well, capturing noise and random fluctuations rather than true underlying patterns. This leads to low training error but high generalization error.

Q2.Based on the softmax function's definition, what is its primary function in a classification model?

To calculate the difference between the model's output and the true labels.
To convert the linear outputs (logits $o_1, o_2, o_3$ ) into a valid probability distribution.
To reduce the number of features from the input layer to the output layer.
To adjust the weights ( $\mathbf{w}$ ) and biases ( $b$ ) to improve the model.

Show answer ▾

Answer: To convert the linear outputs (logits $o_1, o_2, o_3$ ) into a valid probability distribution.

The softmax function transforms raw outputs (logits) to ensure they are non-negative and sum to 1, just like probabilities. The softmax operation is: $\hat{y}_j = \frac{\exp(o_j)}{\sum_{k} \exp(o_k)}$ . This ensures the output can be interpreted as a probability distribution over classes.

Q3.Why is a non-linear activation function, such as ReLU (max(0, z)), a critical component in deep neural networks?

Because stacking multiple linear layers is equivalent to a single linear layer, preventing the model from learning complex patterns.
Because it ensures that all weights and biases in the network remain positive.
Because it is the only function that allows for the use of stochastic gradient descent.
Because it simplifies the model by converting it back into a linear regression problem.

Show answer ▾

Answer: Because stacking multiple linear layers is equivalent to a single linear layer, preventing the model from learning complex patterns.

Stacking multiple linear layers simply results in another linear transformation. To learn complex, non-linear relationships found in real-world data, activation functions like ReLU introduce non-linearity, allowing networks to approximate vastly more complex functions.

Q4.What does variance measure in the context of the bias-variance trade-off?

How far the average model prediction deviates from the true value due to limiting assumptions like linearity.
The randomness inherent to the data generating process that cannot be learned away.
How much the model's prediction would change if it were trained on a different set of training data.
The model's accuracy on the training set.

Show answer ▾

Answer: How much the model's prediction would change if it were trained on a different set of training data.

Variance measures how much the prediction would change if we collected a fresh training set. It captures the model's sensitivity to small fluctuations in training data. High variance risks overfitting.

Q5.When three ReLU hinges combine to create a compact triangular 'bump' in the input space, this illustrates the intuition behind what theoretical guarantee?

The Normal Equation for solving linear regression analytically.
The Universal Approximation Theorem.
The bias-variance trade-off.
The central limit theorem for error estimation.

Show answer ▾

Answer: The Universal Approximation Theorem.

Three hinges can start a rise, reverse its slope, and cancel it back to zero. Creating and summing many such localized pieces gives the intuition behind uniform approximation of continuous functions on compact domains.

Q6.For softmax regression with cross-entropy loss, what is the gradient of the loss with respect to the logits

\mathbf{o}

(before softmax)?Show answer ▾

Answer: $\nabla_{\mathbf{o}} L = \hat{\mathbf{y}} - \mathbf{y}$ (predicted probabilities minus one-hot target)

This is the elegant identity that emerges from the chain rule when differentiating softmax followed by cross-entropy. The result is the difference between predicted and true probabilities, which directly informs weight updates in classification networks and is why softmax+CE is the standard pairing.

Start of course

Module 2