Optimization

Lagrange Multipliers

Convert an equality-constrained optimization into stationarity of a single Lagrangian, with multipliers measuring constraint sensitivity.

The method

To minimize f(x) subject to g(x) = 0, form the Lagrangian L(x, lambda) = f(x) + lambda dot g(x). At a constrained optimum, the gradient of L with respect to both x and lambda vanishes. Setting grad_x L = 0 and grad_lambda L = 0 recovers stationarity and the constraint, turning a constrained problem into a system of equations.

Geometric intuition

Kronos motion — sensitivity

At a constrained minimum the objective cannot decrease along any feasible direction. This means grad f is perpendicular to the constraint surface, hence parallel to grad g: grad f(x*) = -lambda * grad g(x*). The multiplier lambda is the proportionality constant that aligns the two gradients.

Multiple constraints

With several equality constraints g_i(x) = 0, the condition becomes grad f = -sum_i lambda_i grad g_i: the objective gradient lies in the span of the constraint gradients. One multiplier accompanies each constraint. Linear independence of the constraint gradients (a constraint qualification) ensures the multipliers are well defined.

Meaning of the multiplier

lambda_i equals the rate of change of the optimal objective as the right-hand side of constraint i is perturbed.
It is the sensitivity, or shadow price, of that constraint.
Large multipliers flag constraints that strongly limit the optimum.

Extension to inequalities

Lagrange multipliers handle equality constraints directly. Inequality constraints require the sign conditions and complementary slackness of the KKT conditions, which generalize the method. Together they form the backbone of constrained optimization theory.

python

import sympy as sp
x,y,lam = sp.symbols('x y lam')
L = x**2 + y**2 + lam*(x + y - 1)  # min x^2+y^2 s.t. x+y=1
sol = sp.solve([sp.diff(L,x), sp.diff(L,y), sp.diff(L,lam)])

Multipliers quantify how hard each physical or budget constraint pushes on an optimal engineering design, guiding where to relax limits for the most gain.