Optimization

L-BFGS

Limited-memory BFGS stores only a handful of recent vectors, scaling quasi-Newton optimization to millions of variables.

Trading memory for scale

Full BFGS stores a dense n-by-n inverse-Hessian approximation, which is impossible when n is in the millions. Limited-memory BFGS (L-BFGS) never forms this matrix. Instead it keeps the last m pairs of step and gradient-difference vectors (m typically 5 to 20) and reconstructs the search direction on the fly.

The two-loop recursion

Kronos motion — confinement scaling

L-BFGS computes the product of the implicit inverse-Hessian with the current gradient using a two-loop recursion over the stored (s, y) pairs. The cost is O(m*n) per iteration in both time and memory, linear in the dimension, versus O(n^2) for full BFGS. Old pairs are discarded as new ones arrive.

Where it excels

Large-scale smooth unconstrained problems: neural network fine-tuning, logistic regression, inverse problems.
Cases where the Hessian is dense but a few curvature pairs capture most of the useful information.
Deterministic full-batch settings; it is less suited to very noisy stochastic gradients.

Bound-constrained variant

L-BFGS-B extends the method to simple bound constraints (lower and upper limits on each variable) using a gradient-projection step to identify active bounds, then an L-BFGS step on the free variables. It is a common general-purpose solver for box-constrained problems.

Practical notes

L-BFGS needs a line search satisfying the Wolfe conditions to maintain a good curvature estimate. It converges superlinearly in practice on well-behaved problems. With noisy gradients it can degrade, so stochastic or mini-batch settings usually favor SGD-family methods instead.

python

from scipy.optimize import minimize
res = minimize(f, x0, method='L-BFGS-B', jac=grad,
               bounds=[(0, None)]*len(x0))

L-BFGS-B is a standard engine for large calibration and inverse problems where each variable has physical bounds.