Saddle Points

Saddle points are critical points that are minima in some directions and maxima in others; they dominate high-dimensional nonconvex landscapes.

Neither minimum nor maximum

A saddle point is a point where the gradient is zero but which is not a local extremum. Along some directions the function curves upward (like a minimum) and along others it curves downward (like a maximum). The canonical example is f(x, y) = x^2 - y^2 at the origin: a valley along x, a ridge along y. The Hessian at a saddle point is indefinite, having both positive and negative eigenvalues.

Why they matter in high dimensions

In low dimensions saddle points seem exotic, but in high-dimensional nonconvex problems they are the dominant kind of critical point. For a random critical point, being a local minimum requires all Hessian eigenvalues to be positive; as the dimension grows, the chance that every one of hundreds of eigenvalues happens to be positive becomes vanishingly small. So most points with zero gradient are saddles, and true local minima are comparatively rare and tend to have low objective values.

The trouble they cause

Near a saddle point the gradient is small, so gradient descent slows dramatically and can appear to have converged when it has merely stalled on a plateau. The flat directions and the presence of negative curvature that gradient methods cannot see make saddles the main obstacle to first-order optimization of neural networks, more so than poor local minima.

Escaping saddles

Several mechanisms help optimizers escape. The noise in stochastic gradient descent perturbs iterates off the exact saddle, letting them slide down a negative-curvature direction; adding explicit small noise provably escapes strict saddles in polynomial time. Second-order methods detect negative curvature through the Hessian and step directly downhill along it. Momentum also helps carry iterates through flat regions. Because strict saddles can be escaped and most minima are good, gradient methods succeed on many nonconvex problems despite the abundance of saddles.