The Nelder-Mead Method

The Nelder-Mead simplex method minimizes a function by moving and reshaping a geometric simplex of points, using only function values.

The simplex

Nelder-Mead, from 1965, is a derivative-free method that maintains a simplex: a set of n+1 points in n-dimensional space, a triangle in two dimensions, a tetrahedron in three. It evaluates the objective at each vertex and transforms the simplex each iteration by moving its worst vertex, gradually crawling and shrinking the simplex toward a minimum. No gradient or matrix is ever formed.

The four moves

Each iteration identifies the best, second-worst, and worst vertices and computes the centroid of all but the worst. It then tries a sequence of geometric operations. Reflection flips the worst vertex through the centroid. If that is very good, expansion pushes further. If reflection is poor, contraction pulls the worst vertex toward the centroid. If nothing helps, shrink pulls all vertices toward the best one. These adaptive moves let the simplex stretch along valleys and squeeze into minima.

• Reflection: mirror the worst point through the centroid
• Expansion: extend a successful reflection
• Contraction: retreat when reflection fails
• Shrink: collapse the whole simplex toward the best vertex

from scipy.optimize import minimize

def rosen(x):
    return sum(100*(x[1:]-x[:-1]**2)**2 + (1-x[:-1])**2)

res = minimize(rosen, x0=[-1.2, 1.0], method='Nelder-Mead')
print(res.x, res.fun)

Strengths and limits

Nelder-Mead is simple, needs no derivatives, and works well for smooth low-dimensional problems and quick tuning where each evaluation is cheap. Its weaknesses are real: it has no convergence guarantee for general functions, can stagnate or collapse onto a false minimum, and scales poorly beyond roughly ten dimensions because the simplex becomes unwieldy. It is also sensitive to the initial simplex.

Place among methods

For noisy or higher-dimensional black-box problems, CMA-ES or Bayesian optimization is usually more reliable. But Nelder-Mead remains a popular first attempt for calibrating a handful of parameters against a simulation, precisely because it is easy to apply and needs nothing but the ability to evaluate the objective.