What Is Hyperparameter Tuning? A Plain-Language Explanation
Hyperparameter tuning is the process of choosing the settings that control how a machine learning model trains, such as learning rate or the number of neighbors, then running repeated experiments to find the combination that performs well on validation data, often using grid search, random search, or Bayesian optimization.
Hyperparameters versus model parameters
Model parameters are learned from data during training, such as the weights in a neural network. Hyperparameters are set before training begins and guide the learning process itself.
| Example hyperparameter | Where it appears |
|---|---|
| Learning rate | Gradient-based models |
| Number of neighbors (k) | k-nearest neighbors |
| Regularization value (C) | Support vector machines |
| Maximum tree depth | Decision trees |
| Number of clusters (k) | k-means |
How tuning works
A common workflow defines a set of candidate values for each hyperparameter, trains a separate model for each combination, and compares the results with cross-validation. The combination with strong validation performance is selected, then checked once on a held-out test set.
Common search methods
| Method | How it chooses candidates |
|---|---|
| Grid search | Tries every combination in a predefined grid |
| Random search | Samples combinations at random, useful when only a few settings matter |
| Bayesian optimization | Uses past results to decide which combination to try next |
Grid search is simple and reproducible, but the number of experiments grows quickly as more settings are added. Random search and Bayesian methods can use computing resources more efficiently in larger search spaces.
Why it matters
Hyperparameters affect how quickly a model learns and how well it generalizes. A model that performs strongly on training data but poorly on new data may simply be using settings that encourage overfitting rather than needing a different algorithm.
Honest caveats
Tuning is computationally intensive because every candidate requires a training run. Repeatedly selecting the highest score on the same validation set can indirectly overfit that set, so keeping a separate, untouched test set for the final check is a sensible practice.
FAQ
What is the difference between a parameter and a hyperparameter?
Parameters are learned from data; hyperparameters are chosen before training and control the learning process.
What is grid search?
A method that trains a model for every combination of hyperparameter values in a predefined grid.
Why is cross-validation used in tuning?
It estimates how each setting performs across different data splits, reducing the chance of a result that depends on one lucky split.
Can tuning replace a better model or more data?
No; tuning helps a suitable model reach its potential, but it cannot fix the wrong algorithm or fundamentally insufficient data.
Key takeaways
- Hyperparameters are training settings chosen in advance.
- Grid, random, and Bayesian searches are common tuning methods.
- Cross-validation compares candidate settings.
- Tuning costs computing time and a separate test set helps avoid overfitting.
Continue with what is cross-validation and what is gradient descent.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free