What Is an Activation Function? A Plain-Language Guide
An activation function is a mathematical rule applied to a neuron's weighted sum that decides what signal it passes to the next layer; adding a non-linear activation function lets a neural network learn complex patterns instead of only straight-line relationships.
Where an activation function sits
A neuron first multiplies each input by a weight, adds the results together with a bias, and produces a weighted sum often written as (z). The activation function then transforms that sum into the neuron's output (a = f(z)). You can think of it as a gatekeeper that controls how strongly a neuron responds, or whether it responds at all.
Why networks need non-linearity
If every neuron used a linear activation, stacking many layers would not add real power: a chain of linear operations collapses into a single linear operation, so even a deep network would behave like a linear model. A non-linear activation function gives the network the ability to approximate curved relationships and handle data that is not separable by a straight line.
Common activation functions
| Function | Output range | Typical place | Note |
|---|---|---|---|
| Linear (identity) | Any value | Output layer for regression | Adds no non-linearity |
| Sigmoid | 0 to 1 | Output layer for binary classification | Produces probability-like values |
| Tanh | -1 to 1 | Hidden layers | Zero-centered, S-shaped |
| ReLU | 0 to any positive value | Hidden layers | Simple and widely used |
| Leaky ReLU | Small negative slope allowed | Hidden layers | Aims to reduce inactive neurons |
| Softmax | 0 to 1, values sum to 1 | Output layer for multi-class tasks | Converts scores into class probabilities |
The ReLU returns zero for negative inputs and keeps positive inputs unchanged:
[ \mathrm{ReLU}(z) = \max(0, z) ]
The sigmoid maps any input into the interval between 0 and 1:
[ \sigma(z) = \frac{1}{1 + e^{-z}} ]
Hidden layers versus output layers
The choice depends on where the function is used. In hidden layers, ReLU is a common starting point because it is simple and works well in many settings, with leaky variants available when neurons become inactive. At the output layer, the choice follows the task: sigmoid for binary classification, softmax for a single multi-class label, and a linear activation for regression that predicts a continuous number.
How activations connect to training
During backpropagation, the network uses the derivative of the activation function to work out how each weight should change. Functions that are smooth and differentiable, such as sigmoid and tanh, support this gradient-based update. A practical issue with ReLU is that a neuron can output zero for every input and stop contributing, often called a dying ReLU, which leaky variants are designed to address.
Honest caveats
There is no single activation function that fits every model, and the choice depends on the data, the layer, and the task. A sigmoid can produce very small gradients for large inputs, which can slow learning in deep networks, and softmax outputs are designed for cases where classes are mutually exclusive rather than multi-label.
FAQ
What does an activation function do in simple terms?
It takes a neuron's weighted sum and decides what signal to send onward, similar to a gate that can open fully, partially, or stay closed.
Why is ReLU so commonly used?
ReLU is simple to compute and helps deep networks learn efficiently, although it can leave some neurons inactive, which leaky variants try to reduce.
What is the difference between sigmoid and softmax?
Sigmoid produces a value between 0 and 1 for one output, while softmax produces a set of probabilities across several classes that add up to 1.
Can a neural network work without an activation function?
It can run, but with only linear activations it stays limited to linear relationships regardless of depth, so non-linear activations are what give deep networks much of their flexibility.
Key takeaways
- An activation function transforms a neuron's weighted sum into its output.
- Non-linear activations allow networks to learn complex, curved patterns.
- ReLU is common in hidden layers; sigmoid and softmax are common at the output.
- The derivative of the activation function supports backpropagation.
Continue with what is a neural network and what is a loss function.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free