Bayesian Optimization

Bayesian optimization efficiently optimizes expensive black-box functions by modeling them and choosing where to sample next.

Optimizing what is costly to evaluate

Bayesian optimization tackles problems where each function evaluation is expensive, training a model, running a simulation, conducting an experiment, so you can afford only a limited number. It replaces blind or exhaustive search with a strategy that reasons about where a good value is most likely, making each evaluation count.

Surrogate and acquisition

The method keeps a surrogate model, typically a Gaussian process, that predicts the objective everywhere along with its uncertainty. An acquisition function then turns that prediction into a decision about where to sample next, balancing exploitation (sample where the surrogate predicts good values) against exploration (sample where uncertainty is high). Common acquisitions are Expected Improvement, Upper Confidence Bound, and Probability of Improvement.

The loop

for t in range(budget):
    surrogate.fit(X_tried, y_tried)
    x_next = argmax(acquisition(surrogate))
    y_next = expensive_objective(x_next)
    X_tried.append(x_next); y_tried.append(y_next)

Strengths and limits

Bayesian optimization is highly sample-efficient, often finding strong settings in far fewer trials than random search, and it naturally reports uncertainty. Its main limits are dimensionality (Gaussian processes struggle much beyond a few dozen parameters, though tree-based and neural surrogates extend the range), and cost per proposal, since fitting the surrogate and maximizing the acquisition themselves take time. Parallel and batch variants propose several points at once to use many workers.

It is the engine behind much of hyperparameter optimization and AutoML, and it applies broadly to experimental design and simulation-based engineering where each trial is genuinely expensive.