Zahri-Edu › Math-University › Mathematics for Machine Learning & AI

AI 131

Optimization for Machine Learning

Convex sets and functions, gradient and stochastic gradient methods, momentum and Adam, constrained optimization, duality and convergence rates.

In preparation

The lecture notes for this course are being written. Below is the planned table of contents; chapters will be published one by one.

Table of contents

6 chapters · 15 sections

1

Convex Analysis

  1. 1.1Convex sets and functions
  2. 1.2Strong convexity and smoothness
2

Gradient Methods

  1. 2.1Gradient descent
  2. 2.2Convergence rates
  3. 2.3Line search
3

Stochastic Optimisation

  1. 3.1Stochastic gradient descent
  2. 3.2Mini-batches
  3. 3.3Momentum, RMSProp, Adam
4

Constrained Optimisation

  1. 4.1Lagrange multipliers
  2. 4.2KKT conditions
  3. 4.3Duality
5

Second-Order Methods

  1. 5.1Newton's method
  2. 5.2Quasi-Newton (BFGS)
6

Non-convex Landscapes

  1. 6.1Saddle points
  2. 6.2Training deep networks