Hyperparameter
A configuration value set before training that governs how a model learns, not learned from the data itself.
Definition
A hyperparameter is a setting chosen before training that controls the learning process or model structure, as opposed to a parameter, which is learned from data. Examples include the learning rate, number of layers, batch size, and regularization strength.
There is a hierarchy of costs: some hyperparameters can be swept cheaply, while others, like model size, are effectively fixed by budget. Practitioners often tune the cheap ones thoroughly and set the expensive ones from prior experience or scaling relationships.
Search cost dominates when each trial is a full training run, so efficient strategies matter: Bayesian optimization models the objective to propose promising settings, and successive-halving methods like Hyperband allocate more budget to configurations that look strong early. The goal is to spend the search budget where it most improves the final model rather than exhaustively covering a grid that is mostly uninformative.
Tuning strategies
- Grid search: try every combination on a predefined grid.
- Random search: sample combinations, often more efficient.
- Bayesian optimization: model the objective to propose promising settings.
- Validation via cross-validation to avoid overfitting the choice.
Why it matters
Hyperparameters strongly affect performance; a good model class with poor hyperparameters can underperform a simple baseline. Systematic tuning, kept separate from the final test set, is essential for honest evaluation.
Fusion connection
Kronos tunes surrogate hyperparameters against held-out simulation folds so the model generalizes across the Hyperion design space rather than to one region.