What Is the Adam Optimizer? A Simple Explanation for Beginners

Updated · By Viviai

The Adam optimizer (short for adaptive moment estimation) is a training algorithm that gives every weight its own adjustable step size by keeping two running averages: one of past gradients and one of past squared gradients. It was introduced by Diederik Kingma and Jimmy Ba in 2014 and combines ideas from two older methods — momentum and RMSProp — into one algorithm that usually works with little manual tuning. If you have read about gradient descent, think of Adam as a version that both remembers the direction it has been moving and scales each step according to how noisy that weight's gradients are.

Because Adam needs little memory and handles sparse or noisy gradients well, it is a common starting choice when training a neural network, including many deep learning models.

The Problem Adam Solves

Plain gradient descent uses one shared learning rate for every weight and takes a step based only on the current gradient. This creates two practical issues:

  1. A single learning rate may be too large for some weights and too small for others.
  2. Gradients can be noisy or point in conflicting directions from one batch to the next.

Adam addresses both by looking at recent history rather than a single gradient. It adapts the step for each weight separately, which removes much of the need to hand-tune a global learning rate, though hyperparameter tuning still matters for difficult tasks.

The Two Moments

Adam maintains two values for every parameter, both starting at zero:

At each step, with gradient (g), the two moments are updated as

[ m = \beta_1 m + (1-\beta_1)g, \qquad v = \beta_2 v + (1-\beta_2)g^2 ]

where the square is applied element-wise. The final step divides the smoothed gradient by the square root of the smoothed squared gradient, so weights with large or noisy gradients take smaller steps.

Why Bias Correction Is Needed

Because (m) and (v) start at zero and the decay rates are close to one, their early values are biased toward zero. Adam corrects this by dividing each moment by (1-\beta^t),

[ \hat{m} = \frac{m}{1-\beta_1^t}, \qquad \hat{v} = \frac{v}{1-\beta_2^t} ]

The correction matters most in the first few iterations, when the averages otherwise underestimate the true gradient. As training continues, the factor approaches one and has little effect.

Common Default Settings

The original paper proposed values that most frameworks, such as Keras and PyTorch, use as defaults.

Hyperparameter Common value Role
Learning rate (\eta) 0.001 Overall size of each update
Beta 1 (\beta_1) 0.9 Decay for the first moment (gradient)
Beta 2 (\beta_2) 0.999 Decay for the second moment (squared gradient)
Epsilon (\epsilon) (10^{-8}) Small constant that prevents division by zero

These defaults are a reasonable starting point, not fixed rules; some models benefit from a smaller learning rate or adjusted betas.

Adam vs SGD

Aspect Adam SGD (with momentum)
Step size Adapted per weight One shared learning rate
Tuning effort Often works with defaults Usually needs more manual tuning
Noisy or sparse gradients Handles them well Can be more sensitive
Early progress Often fast Can be slower to start
Generalization Usually strong, sometimes weaker Can generalize well when tuned

In practice Adam often reaches a low training error quickly, while some studies find well-tuned SGD can produce comparable or slightly better generalization on certain tasks.

Variants and Limitations

Frequently Asked Questions

What is Adam optimizer in simple words?

It is an optimizer that keeps a smoothed average of past gradients and of past squared gradients, then uses both to choose a separate step size for each weight. This makes it robust and reduces manual tuning.

Why is it called Adam?

The name comes from adaptive moment estimation, because the algorithm estimates the first and second moments of the gradient.

What is the default learning rate for Adam?

Many frameworks use 0.001 as the default, along with beta 1 of 0.9 and beta 2 of 0.999, but these are starting values rather than universal rules.

Is Adam better than SGD?

Adam often converges with less tuning and handles noisy gradients well, but a carefully tuned SGD can sometimes generalize comparably. The choice depends on the task and how much tuning is practical.

What is the difference between Adam and AdamW?

AdamW changes how weight decay is applied, keeping it separate from the momentum-style update. This often works better for transformer and large language model training.

Summary

Adam is an adaptive optimizer that combines a momentum-like average of gradients with an RMSProp-like average of squared gradients, then applies bias correction so early updates are not underestimated. Its default settings make it a practical starting choice for many neural networks, while variants such as AdamW are widely used for transformers. For related ideas, see gradient descent, backpropagation, and batch normalization.

Sources

  1. https://arxiv.org/pdf/1412.6980v4
  2. https://optimization.cbe.cornell.edu/index.php?title=Adam
  3. https://keras.io/api/optimizers/adam/
  4. https://www.ultralytics.com/glossary/adam-optimizer

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free