Gradient Descent
Training a model means minimising a loss L(θ). Gradient descent repeatedly steps downhill: θ ← θ − η ∇L(θ), where η is the learning rate.
Too small η: slow. Too large η: the iterates overshoot and can diverge. Momentum (β) remembers previous steps and rolls through narrow valleys.
Loss shown: the elongated bowl L(x, y) = x² + 10y² or the Rosenbrock "banana" valley, a classic test for optimisers.