Gradient-Free Optimization
Optimize when gradients are unavailable, unreliable, or expensive, using only objective values to guide the search.
When derivatives are out of reach
Many objectives offer no usable gradient: black-box simulations, noisy physical experiments, nonsmooth or discontinuous functions, and legacy code. Gradient-free (derivative-free) optimization works with objective values alone. It trades the fast convergence of gradient methods for the ability to handle problems where gradients simply do not exist or cannot be computed.
Direct search methods
- Nelder-Mead simplex: reflects, expands, and contracts a simplex of points to crawl downhill.
- Pattern search (Hooke-Jeeves, generalized pattern search): probe along a mesh of directions and refine the mesh, with convergence guarantees.
- Coordinate search: try steps along each axis in turn.
Model-based methods
Derivative-free trust-region methods build a local interpolation model (often quadratic) from sampled points and minimize it within a trust region, updating the model as new points arrive. These methods, such as those in the NEWUOA family, are efficient for smooth but derivative-free objectives, using far fewer evaluations than direct search.
Stochastic and global methods
When the landscape is multimodal or very noisy, population-based metaheuristics apply: simulated annealing, evolutionary strategies like CMA-ES, particle swarm, and differential evolution. Bayesian optimization is the method of choice when each evaluation is very expensive, since it minimizes the number of function calls.
Choosing a method
For smooth, cheap, low-dimensional problems, model-based derivative-free trust-region methods are efficient. For noisy or multimodal problems, CMA-ES and differential evolution are robust. For very expensive evaluations, Bayesian optimization dominates. All scale worse with dimension than gradient methods, so exploiting any available gradient, even approximate finite differences, is worthwhile when possible.
from scipy.optimize import minimize
res = minimize(f, x0, method='Nelder-Mead') # no gradient neededGradient-free methods optimize designs against black-box physics simulations that provide no derivatives, a common situation in device engineering.