Parameters vs Hyperparameters
The difference between the internal numbers the machine learns and the external settings you choose.
Verdict: Parameters are learned by the algorithm from the data. Hyperparameters are chosen by you before training starts to control how the algorithm learns.
The Short Answer
In machine learning, you are dealing with two completely different sets of numbers that govern how a model works.
Parameters are the internal variables that the model learns on its own. You do not set these. The optimization algorithm (like Gradient Descent) calculates them by looking at the training data. (e.g., The weights and biases in a neural network).
Hyperparameters are the external configuration settings. You must set these before training even begins. They control the overarching architecture of the model and the behavior of the learning process itself. (e.g., The learning rate, the batch size, or the number of trees in a random forest).
Where They Differ
| Feature | Parameters | Hyperparameters |
|---|---|---|
| Who sets them? | The Optimization Algorithm. | You (the Data Scientist). |
| When are they set? | Continuously during training. | Before training begins. |
| What do they do? | Store the actual patterns learned from the data. | Control how the model learns those patterns. |
| Neural Network Examples | Weights (), Biases (). | Learning Rate, Number of Epochs, Number of Hidden Layers. |
| Decision Tree Examples | The specific feature and threshold used at each split (e.g., Age > 25). | Max Tree Depth, Minimum samples per leaf. |
| K-Means Examples | The coordinates of the cluster centroids. | The number of clusters (). |
The Optimization Loop
To see how they interact, consider the standard training loop of a neural network:
- You choose a Hyperparameter (Learning Rate = 0.01).
- You click "Train".
- The model looks at a batch of data.
- The model calculates its error.
- The model adjusts its internal Parameters (Weights) by a step size determined by your Learning Rate.
- Go back to step 3.
If you chose a bad Hyperparameter in step 1, the model will fail to learn good Parameters in step 5.
How to find the best Hyperparameters
Because the model cannot learn hyperparameters on its own, finding the best ones requires Hyperparameter Tuning (or Search).
You basically guess a configuration, train the model, check its validation score, and then try another configuration. Common strategies include:
- Grid Search: Trying every single combination of a predefined list of settings (very slow).
- Random Search: Randomly sampling combinations (surprisingly effective).
- Bayesian Optimization: Using a secondary machine learning model to predict which hyperparameters will work best based on past experiments.
What People Get Wrong
Tuning Hyperparameters on the Test Set
If you train a model, check its score on the Test Set, tweak a hyperparameter, and try again, you have committed data leakage. You are manually leaking information about the Test Set into the model via your hyperparameter choices. You must use a separate Validation Set for tuning, keeping the Test Set locked away until the very end.