Powell's Method

Powell's method minimizes a function by successive line searches along a set of directions that it updates to become mutually conjugate, without derivatives.

Direction-set minimization

Powell's method is a derivative-free optimizer that reduces a multidimensional problem to a sequence of one-dimensional line searches. It keeps a set of search directions, initially the coordinate axes, and in each cycle minimizes the objective along each direction in turn, moving to the minimum found before starting the next. The cleverness is in how it revises the direction set between cycles.

Building conjugate directions

Simply minimizing along fixed coordinate directions is slow on elongated valleys. Powell's method instead replaces one direction each cycle with the net displacement made over the whole cycle. For a quadratic objective this construction produces conjugate directions, directions that do not interfere with one another under the quadratic's curvature, so the method minimizes a quadratic in n dimensions in a finite number of cycles, matching conjugate-gradient efficiency but using no derivatives.

• Line-search along each current direction in sequence
• Track the total move over the cycle as a new direction
• Replace the direction of greatest decrease with the net move
• Directions become conjugate for quadratic objectives

from scipy.optimize import minimize

def f(x):
    return (x[0]-3)**2 + 10*(x[1]-x[0]**2)**2

res = minimize(f, x0=[0.0, 0.0], method='Powell')
print(res.x, res.fun)  # derivative-free, uses conjugate directions

Practical notes

The naive direction-replacement rule can make the direction set nearly linearly dependent, collapsing the search into a lower-dimensional subspace. Practical implementations include safeguards that skip the replacement when it would degrade the direction set, preserving good coverage of the space. The quality of the inner one-dimensional line search also matters for overall speed.

When to reach for it

Powell's method suits smooth objectives where gradients are unavailable or unreliable and the dimension is modest. It often outperforms Nelder-Mead on smooth functions because its conjugate-direction structure exploits curvature implicitly. Like other direct-search methods it is a reasonable default for calibrating a simulation or fitting a model when only function values can be computed.