What Is a Loss Function? A Plain-Language Explanation
A loss function is a mathematical rule that measures how far a model's predictions are from the correct values, producing a single number that is larger when the model is wrong; training adjusts the model's parameters to make that number as small as practical.
What a loss function does
During training, the model makes a prediction for each input, and the loss function compares that prediction with the known answer, often called the ground truth. If the prediction is accurate, the loss is small; if it is far off, the loss grows. A loss of zero means the model makes no error on that sample.
The loss is often divided by the number of samples so it can be read as an average error per sample, independent of how large the dataset is.
Loss, cost, and objective: the terms
The terms are closely related and sometimes used interchangeably.
| Term | Meaning |
|---|---|
| Loss | Error on a single prediction or sample |
| Cost | The aggregated loss over the whole training set |
| Objective function | Any function training aims to minimize or maximize |
A loss function is the specific case where minimizing the value is the training goal. Some objectives also include a regularization term that penalizes overly complex models.
Common loss functions
Different problems use different loss functions.
| Loss function | Typical task | Idea |
|---|---|---|
| Mean squared error (L2) | Regression | Squares the difference between prediction and target |
| Binary cross-entropy | Two-class classification | Penalizes confident wrong probabilities |
| Categorical cross-entropy | Multi-class classification | Compares predicted class probabilities with labels |
| Hinge loss | Support vector machines | Penalizes points on the wrong side of a margin |
| Log-loss | Logistic regression | Rewards well-calibrated probabilities |
For a regression problem, the mean squared error is written as:
[ \mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n}\left(\hat{y}_i - y_i\right)^2 ]
where (\hat{y}_i) is the prediction and (y_i) is the correct value.
How loss drives training
The loss alone only reports the error. An optimization algorithm, commonly gradient descent, uses the loss to update each parameter in the direction that reduces the error. Repeating this cycle over many iterations gradually lowers the loss, which is how the model learns.
Training loss versus validation loss
Training loss is measured on the data the model learns from. Validation loss is measured on data the model has not seen, and it helps reveal overfitting. If training loss keeps falling while validation loss rises, the model may be memorizing the training data rather than generalizing.
Honest caveats
A low training loss does not by itself indicate a useful model, because the number can be driven down by overfitting. Loss values are also not directly comparable across different functions, so a model with a smaller MSE is not automatically judged the same way as one with a smaller cross-entropy.
FAQ
What is the difference between a loss function and a cost function?
Loss usually refers to the error on one sample, while cost refers to the aggregated loss over the entire dataset. In practice, the two terms are often used interchangeably.
Why do classification models use cross-entropy?
Cross-entropy penalizes confident wrong answers heavily, which encourages the model to produce calibrated class probabilities.
Can a loss of zero be achieved on real data?
A loss of zero means error-free predictions, which is rare on real-world data and can indicate overfitting if it occurs on the training set.
Is the loss function the same as the optimization algorithm?
No. The loss function measures the error; the optimization algorithm, such as gradient descent, uses that measurement to update the parameters.
Key takeaways
- A loss function quantifies the gap between predictions and correct values.
- Regression commonly uses squared error; classification commonly uses cross-entropy.
- Optimization algorithms update parameters to reduce the loss.
- Validation loss helps distinguish learning from overfitting.
Continue with what is gradient descent and what is regularization.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free