Saddle Points and Nonconvexity

In high-dimensional nonconvex problems, saddle points, not local minima, are the main obstacle to gradient-based training.

Beyond convexity

Convex problems have a single global minimum, but most modern models, including deep neural networks, minimize nonconvex objectives with many stationary points. Understanding the geometry of these landscapes, especially saddle points, explains why training sometimes stalls and why certain optimizers succeed where others crawl.

Types of stationary points

• Local minimum: Hessian positive definite; all directions curve upward.
• Local maximum: Hessian negative definite; all directions curve downward.
• Saddle point: Hessian indefinite; some directions curve up, others down.
• Degenerate (flat) point: Hessian has zero eigenvalues, a plateau.

Why saddles dominate in high dimensions

At a random stationary point in d dimensions, each Hessian eigenvalue is roughly equally likely to be positive or negative. For that point to be a local minimum, all d eigenvalues must be positive, which becomes exponentially unlikely as d grows. So in high dimensions the overwhelming majority of stationary points are saddles, not minima.

Escaping saddles

Gradient descent slows near a saddle because the gradient is small there, and it can stall along flat or near-flat directions. Noise helps: stochastic gradient descent's randomness pushes iterates off saddles along negative-curvature directions. Adding deliberate perturbations (perturbed gradient descent) provably escapes strict saddles efficiently.

Practical consequences

For many deep networks, most local minima have similar, low objective values, so getting trapped in a bad local minimum is rarely the problem; slow progress through saddle regions and plateaus is. This motivates momentum, adaptive optimizers, and noise, all of which help traverse flat and saddle-dominated regions faster than plain gradient descent.

import numpy as np
def classify(H):
    ev = np.linalg.eigvalsh(H)
    if np.all(ev>0): return 'min'
    if np.all(ev<0): return 'max'
    return 'saddle'

Understanding saddle-dominated landscapes guides optimizer choice when training the large nonconvex models used for scientific surrogates and analysis.