Hyperparameter Tuning

Hyperparameters set before training, learning rate, depth, regularization, are searched to find the best model configuration.

Parameters versus hyperparameters

Parameters (weights) are learned from data during training. Hyperparameters are set before training and govern the learning process itself: learning rate, tree depth, number of estimators, regularization strength, k in k-NN. They are not learned by the optimizer, so they must be searched separately.

Search strategies

• Grid search: try every combination on a predefined grid; exhaustive but scales badly with dimensions.
• Random search: sample combinations at random; often finds good settings faster than a grid.
• Bayesian optimization: model the score surface and propose promising points; sample-efficient.
• Successive halving / Hyperband: give more budget to promising configurations, kill weak ones early.

from sklearn.model_selection import RandomizedSearchCV
search = RandomizedSearchCV(model, param_distributions=grid,
                            n_iter=50, cv=5, scoring='f1')
search.fit(X_train, y_train)
best = search.best_estimator_

Evaluate with cross-validation

Each candidate is scored by cross-validation on the training data, never the test set. Because you are choosing the best of many candidates, the winning validation score is itself optimistic. Nested cross-validation, an inner loop for tuning and an outer loop for estimation, corrects this bias when you need an honest performance figure.

Practical advice

Random search beats grid search when only a few hyperparameters truly matter, which is usual. Sample continuous ranges on a log scale for learning rates and regularization strengths. Tune the few high-impact hyperparameters first. Always keep the test set untouched until tuning is finished, or the reported performance will be inflated.