Loss Landscape Geometry
The geometry of a model's loss surface, its valleys, barriers, and flatness, shapes how training proceeds and how well the result generalizes.
The surface training moves on
Training a model is descent on a loss landscape: a surface over the space of parameters whose height is the loss. For large models this surface lives in millions of dimensions and cannot be seen directly, but its geometry, the shape of its valleys, the height of barriers between them, and the flatness of its minima, determines both whether optimization succeeds and how the trained model behaves on new data.
Flat versus sharp minima
Not all minima are equal even at the same loss value. A flat minimum sits in a wide basin where the loss changes slowly with the parameters; a sharp minimum sits in a narrow one where small parameter changes raise the loss quickly. Flat minima are widely associated with better generalization, because a small mismatch between the training loss surface and the true loss surface perturbs a flat solution less. Optimizers and regularizers that favor flat regions, such as sharpness-aware minimization, exploit this.
- Flat minima: wide basins, robust to perturbation, generalize better
- Sharp minima: narrow basins, sensitive, often generalize worse
- Barriers: high-loss ridges separating distinct solutions
- Plateaus: near-flat regions that slow first-order progress
Connectivity of solutions
A surprising empirical finding is mode connectivity: distinct minima found by separate training runs are often linked by paths through parameter space along which the loss stays low, rather than being isolated in deep separate wells. This reveals the landscape's minima as parts of a connected low-loss manifold, not scattered points, and it underlies techniques such as weight averaging and ensembling that exploit the region between good solutions.
Why the geometry is studied
Understanding landscape geometry guides practical choices: batch size and learning rate influence whether training settles in flat or sharp regions; noise shapes which basins are reachable; visualization along random or curvature directions helps diagnose training failures. For any simulation-driven optimization, mapping whether the objective is smoothly bowl-shaped or riddled with narrow traps decides whether a local method suffices or a global search is required.