Hyperparameter Optimization
Hyperparameter optimization searches for the training settings, learning rate, depth, regularization, that yield the best model.
Settings you choose, not learn
Hyperparameters are configuration values fixed before training, learning rate, tree depth, regularization strength, number of layers, that the learning algorithm does not adjust on its own. They strongly determine final performance, and finding good values is an optimization problem in its own right, layered on top of model training.
Basic search methods
- Grid search: try every combination on a predefined grid; simple but scales exponentially with the number of hyperparameters
- Random search: sample combinations at random; often better than grid search because only a few hyperparameters usually matter, and random sampling covers those dimensions more finely
- Manual search: expert-guided, effective but not reproducible or scalable
Smarter search
Bayesian optimization builds a probabilistic model of performance versus hyperparameters and proposes the next trial where improvement is most likely, spending evaluations wisely. Hyperband and successive halving allocate a small budget to many configurations, then progressively concentrate resources on the promising ones, exploiting the fact that bad configurations reveal themselves early. Population-based training evolves a population of models and their hyperparameters jointly during a single run.
Doing it honestly
Hyperparameter search must be validated correctly. Tuning on the test set leaks information and inflates results, so a separate validation set or nested cross-validation is required: an inner loop selects hyperparameters and an outer loop estimates generalization. Because search itself can overfit the validation set, especially with many trials, the final chosen configuration must be judged on data untouched during the search. Log every trial for reproducibility, a core concern of MLOps.