What Is a Learning Rate? Step Size in Machine Learning Explained

Updated · By Viviai

A learning rate is a hyperparameter that controls how large a step a machine learning model takes each time it updates its weights during training. It scales the gradient: a small value moves the weights gently, while a large value moves them aggressively. The learning rate is often written as the Greek letter eta (η) or alpha (α), and it sits at the center of whether a model converges slowly, trains smoothly, or diverges.

The one equation that matters

In plain gradient descent, the weights at the next step follow this rule:

[ \theta_{t+1} = \theta_t - \eta \, \nabla L(\theta_t) ]

Here (\theta_t) is the current set of weights, (\nabla L(\theta_t)) is the gradient of the loss (the direction that would make the loss go up), and (\eta) is the learning rate. The minus sign flips the direction so the model walks downhill, toward lower loss. The gradient points the way; the learning rate decides how far to walk.

You can find the same idea in more detail in the gradient descent explainer.

Why the number matters: walking down a valley

Think of training as walking down the side of a valley toward the lowest point in thick fog. You can feel which way the ground slopes, but you cannot see the bottom.

The learning rate is the length of those strides. The same gradient can produce a stable model or a broken one depending only on this multiplier.

Too high vs. too low

Learning rate What happens during training Typical sign
Too small Weight updates are tiny; progress is slow and the model may stop early in a poor local minimum Loss falls, but very slowly, and barely moves after many epochs
In a sensible range Loss decreases steadily and settles near a low value without wild swings Smooth loss curve with small, shrinking oscillations
Too large Updates overshoot the minimum; the loss oscillates, plateaus at a high value, or grows without bound Loss jumps around or turns into NaN

The numerical value alone tells you nothing. A rate that works for one model can be far too large for another, because the effect depends on the scale of the loss, how the features are normalized, and the optimizer. A value of 0.01 is common, not universal.

Common starting points

There is no single correct number, but these ranges show up often enough to be useful starting guesses:

Setup Typical starting learning rate
Plain gradient descent 0.01 to 0.1
SGD with momentum 0.01 to 0.05
Adam and other adaptive optimizers 0.001 (1e-3)
Fine-tuning a large pretrained model 1e-5 to 5e-5

Treat these as first guesses to test, not constants. Many practitioners start with a default such as 0.001 for Adam and then adjust based on the loss curve. If the loss explodes, lower it by roughly a factor of 10; if it barely moves, raise it.

Learning rate schedules

A constant rate is the simplest choice, but most deep networks do better when the rate changes over time. Early on, larger steps help the model approach the region of a good solution; later, smaller steps let it settle without overshooting.

A 2019 study catalogued 13 different learning rate functions and found that dynamic schedules often reach higher accuracy than a fixed constant, at the cost of being harder to tune.

Adaptive optimizers

Some algorithms do not use one global learning rate. They keep a separate, self-adjusting rate for every parameter, based on the history of its gradients.

The Adam optimizer guide walks through those moving averages. Even adaptive methods still expose a base learning rate, so the concept never disappears; it simply becomes a starting point that the algorithm modifies.

How to choose one in practice

  1. Start with a known default for your optimizer (for example, 1e-3 for Adam).
  2. Plot training and validation loss each epoch rather than watching a single number.
  3. If the loss is flat, try a rate roughly 3 to 10 times larger.
  4. If the loss oscillates or becomes NaN, divide the rate by 10 and check that input features are normalized.
  5. Once a stable range works, add a schedule such as cosine decay and compare the result.

A learning rate finder can automate the first step: it runs a short training while steadily increasing the rate and records where the loss falls fastest, suggesting a value just below that point.

Frequently asked questions

Is a smaller learning rate always safer? Not always. A very small rate is unlikely to diverge, but it can make training painfully slow and leave the model stuck in a poor local minimum. The goal is a rate that makes steady progress, not the smallest number you can find.

What is the difference between learning rate and step size? They refer to the same quantity. "Step size" is the older term from numerical optimization, while "learning rate" is the common name in machine learning. Both describe how far each weight update moves.

Does the learning rate affect model accuracy? Yes, indirectly. It controls how completely and how stably the optimization converges. A poorly chosen rate can stop the model far below the accuracy it is capable of, even when the data and architecture are sound.

How does learning rate relate to batch size? They interact. Larger batches produce smoother gradients and often tolerate a slightly larger learning rate. When you scale the batch size up, you sometimes scale the learning rate up as well, then verify on the loss curve.

Key takeaways

For the surrounding concepts, see the loss function and RMSNorm explainers, and the Adam optimizer guide for adaptive updates.

Sources

  1. https://www.ibm.com/think/topics/learning-rate
  2. https://en.wikipedia.org/wiki/Learning_rate
  3. https://www.aiwiki.ai/wiki/learning_rate
  4. https://factory.fpt.ai/ai-insights/what-is-learning-rate
  5. https://arxiv.org/pdf/1908.06477v1

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free