Module 3: Optimization and Training

SGD, Adam, learning rates, and regularization techniques

4-5 hours

Master gradient descent and its variants (SGD, Adam, RMSprop)

Understand learning rate scheduling strategies

Apply regularization techniques to prevent overfitting

Implement optimization algorithms from scratch

Lecture Materials

Core optimization concepts and algorithms

Mathematical foundations and intuitions behind gradient descent.

35 min

Modern optimization algorithms that power state-of-the-art models.

30 min

Required Readings

Theoretical foundations and research papers

The original Adam optimizer paper - essential reading.

45 min

Code Examples & Notebooks

Implement optimizers and training loops

Build the Adam optimizer step by step.

Additional Resources

Tools and references for optimization

Interactive visualizations of different optimization algorithms.

Ready to continue your learning journey?Return to Course