Activation Functions

The bend between layers that decides what passes and what dies. Its slope sets how much training signal reaches the early layers.

Sigmoid flattens at both ends so its slope falls toward zero there, while ReLU holds a slope of exactly one across its whole positive side

Why Does This Exist?

Stack two linear layers with nothing between them and you still own one linear layer: $W_2(W_1 x) = (W_2 W_1)x$ , where each $W$ is a weight matrix, $x$ is the layer input, and the product in parentheses is just another matrix. Depth without a bend adds parameters but no new shapes, which is why the multi-layer perceptron only solves its showcase problems once something bends the signal between layers.

That bend has a second job, and the second job picked every winner on this page. Backpropagation walks the loss backward through each layer, and at each activation it multiplies by that activation's derivative, the slope of the curve at the value that just passed through. (If slopes are new, derivatives and partial derivatives builds them from zero.) Twenty layers means twenty slopes multiplied, so a slope under 1 at every layer compounds toward nothing:

0.25^{20} = \left( \frac{1}{4} \right)^{20} \approx 9.1 \times 10^{-13}

Keep one network in mind through the rest of this page: a twenty-layer grader watching a belt of coffee beans and sorting each bean as sound, insect-damaged, mouldy, or broken. Its early layers only train if the gradient survives the trip back through all twenty bends. Sigmoid, the obvious first pick, cannot make that trip. Everything after it is a fix with a price tag.

This page is about the bends themselves: what each one computes, what its slope does to training, and which to use where. The artificial neuron is where the bend lives, and vanishing and exploding gradients measures what depth costs once the bends are in. The one idea you need first: each neuron forms a weighted sum $z$ of its inputs plus a bias, then outputs $a = f(z)$ . The activation $f$ is the bend, and $f'(z)$ is the slope the backward pass multiplies by.

Think of It Like This

A hallway of twenty shower valves

You run a long hallway with twenty showers in a row, fed by one pipe, and you stand at the far end turning the last valve by hand while a friend watches the first showerhead. Your question: how much does the first shower's flow change when you turn the last valve a little?

An old sigmoid valve spends most of its travel nearly shut or nearly wide open, where turning changes almost nothing. Mid-travel is its best spot, and even there one full turn of your hand moves the stem a quarter turn. Twenty valves in a row, and your full turn arrives at the far end as less than a trillionth of a turn. Your friend sees nothing move.

A ReLU valve is a different object. Above its threshold it is a straight pipe: one turn in, one turn out, all the way. Below its threshold it is capped: nothing passes, and turning changes nothing either. Twenty open pipes pass your turn along whole. But a capped pipe passes no hint of which way to turn, so a valve knocked shut by one rough twist can sit shut for the rest of the wash.

Where it stops: valves do not learn, and a capped pipe in your hallway can be reopened by hand. A dead unit in a network reopens only through gradient, and the cap is exactly what blocks gradient. The plumbing shows the multiplication. It does not show the recovery problem.

How It Actually Works

Where the bend sits

Each neuron computes $z = w \cdot x + b$ : $w$ holds one weight per input, $x$ holds the inputs, $b$ is a single bias number, and $z$ , the pre-activation, is their weighted sum. Then it outputs $a = f(z)$ . The whole page is the choice of $f$ . Hidden layers share one pick across the network; the output layer follows the task, as the last section shows.

Sigmoid and tanh flatten at the ends

Sigmoid squashes any input into the range from 0 to 1:

\sigma(z) = \left( 1 + e^{-z} \right)^{-1}

Here $\sigma$ is the sigmoid function, $z$ is the pre-activation, and $e$ is Euler's number. Its slope is $\sigma(z)(1 - \sigma(z))$ , which peaks at 0.25 when $z = 0$ and sinks toward 0 in both directions. At $z = 3$ the output is 0.953 and the slope is 0.045. Past that the unit is saturated: large input changes move the output almost not at all, and the backward pass multiplies by almost nothing.

Tanh was the first patch, and it fixed centering rather than saturation. It runs from $-1$ to 1, so its outputs average near zero instead of all-positive. All-positive outputs force the next layer's weight gradients to share one sign, which makes optimization zigzag, so centering helps directly. Its slope peaks at 1 when $z = 0$ , kinder than 0.25, but past an input magnitude of about 2 the slope drops under 0.08 and the same saturation story repeats.

ReLU holds its slope and sometimes dies

ReLU is $\max(z, 0)$ : pass positives through whole, map everything else to zero. Its slope is exactly 1 for every positive input and exactly 0 for every negative one. At $z = 0$ there is a kink with no single slope, and frameworks simply use one side. Twenty layers of switched-on units multiply to 1, so the first layer's gradient arrives at full strength, and each unit costs one comparison instead of an exponential.

The price is the dead unit. A neuron whose pre-activation stays negative on every training example outputs zero on everything and receives zero slope on everything, so the update that could rescue its weights never arrives. One oversized step knocking the bias far negative is the usual cause, and it lasts until something outside reopens the gate.

Two patches, each with a cost. Leaky ReLU gives the negative side a small slope, usually 0.01, so a trickle of gradient survives and a stalled unit can climb back; on most vision workloads the gain over plain ReLU is small. GELU is smooth through zero instead of kinked, weighting each input by $\Phi(z)$ , the standard normal cumulative probability up to $z$ ; it trains transformer stacks better than ReLU, reliably enough that BERT and the GPT family adopted it. SwiGLU, common in large language models, splits the signal into two projections, bends one, and multiplies them, so each unit learns its own gate; it spends a third weight matrix in that block to buy it.

Worked example: one neuron, two inputs

Take weights $w = [2, -1]$ , inputs $x = [0.5, 1.0]$ , bias $b = 0.3$ . Then:

z = \left( 2 \times 0.5 \right) + \left( -1 \times 1.0 \right) + 0.3 = 0.3

At $z = 0.3$ :

\sigma(0.3) = \left( 1 + e^{-0.3} \right)^{-1} \approx 0.574 \quad \text{with slope} \quad 0.574 \times 0.426 \approx 0.244

In plain words: $e^{-0.3}$ is about 0.741, so the sigmoid is $1 / 1.741 \approx 0.574$ , and its slope is $0.574 \times 0.426 \approx 0.244$ . Alongside it, $\tanh(0.3) \approx 0.291$ with slope $1 - 0.291^2 \approx 0.915$ , $\text{ReLU}(0.3) = 0.3$ with slope 1, and $\text{GELU}(0.3) \approx 0.185$ .

Now the same neuron on a bad day: inputs $x = [0.0, 1.8]$ , same weights and bias, so $z = 0 - 1.8 + 0.3 = -1.5$ . Sigmoid gives 0.182 with slope about 0.149, tanh gives about $-0.905$ with slope about 0.181, ReLU gives 0 with slope 0, and Leaky ReLU gives $-0.015$ with slope 0.01. That last pair is the whole pitch for the leak: nearly the same forward values as ReLU, but the backward pass still carries a signal. These are the numbers drawn in the diagram above, where the marker line sits at $z = -1.5$ and the three curves stand visibly apart.

The output layer follows the task

Hidden layers: ReLU or GELU. Pick one and move on. The choice matters less than everything around it.

The last layer answers to the task, not to taste. Predicting an unbounded number, like a bean's mass in grams: no bend, hand over the raw score. One yes-or-no label: a sigmoid, which reads as a probability of yes. Exactly one class out of several: softmax across the scores, which splits one unit of probability between the classes. Several labels at once, since a bean can be mouldy and broken together: one independent sigmoid per label.

Softmax and independent sigmoids look interchangeable and are not. Softmax forces its outputs to sum to 1, so raising one class lowers the rest: right when exactly one label holds, wrong when several can hold at once.

Code

The snippet below replays the worked example and the depth comparison in one run: the four activations at both operating points, then twenty layers of best-case sigmoid slope against twenty layers of live ReLU slope.

1import numpy as np2from math import erf, sqrt3
4
5def sigmoid(z: float) -> float:6    return float(1.0 / (1.0 + np.exp(-z)))7
8
9def gelu(z: float) -> float:10    phi = 0.5 * (1.0 + erf(z / sqrt(2.0)))  # P(N(0, 1) <= z)11    return z * phi12
13
14def relu(z: float) -> float:15    return float(max(z, 0.0))16
17
18for z in (0.3, -1.5):19    print(round(z, 2), round(sigmoid(z), 3), round(float(np.tanh(z)), 3),20          round(relu(z), 3), round(gelu(z), 3))21# -> 0.3 0.574 0.291 0.3 0.18522# -> -1.5 0.182 -0.905 0.0 -0.123
24print(f"{0.25 ** 20:.3e}", f"{1.0 ** 20:.3e}")25# -> 9.095e-13 1.000e+00

The two rows match the hand computation above digit for digit. The last line is the depth story in miniature: twenty best-case sigmoid slopes shrink to about $9.1 \times 10^{-13}$ while twenty live ReLU slopes stay at 1.

Watch Out For

A layer where the same units output zero on every row

Take one batch, look at a hidden ReLU layer, and count the units that are zero on every row. Healthy layers sit near half zeros per row, with different units firing on different rows. A layer where the same 60 percent are zero on every row has lost 60 percent of its capacity, and those units are not coming back on their own.

The usual cause is a learning rate high enough that one early update drove those biases far negative. Lower the rate and restart from scratch, since a dead layer cannot be repaired in place. Then check the initialization scale: He initialization exists because ReLU halves the variance going forward, and starting too small crowds units toward the dead side.

Squashing the output and handing it to a loss that squashes again

Nearly every framework ships a cross-entropy loss that applies the sigmoid or softmax internally, for numerical reasons. Hand it probabilities that are already squashed and it squashes them twice. No error, no warning, no shape mismatch. The loss even falls for a while, then the model plateaus early and well short.

The gradient is why: one squashing multiplies it by a factor under 1, two multiply twice, and the second is evaluated where the function is flattest. Know which convention your loss follows and feed it raw scores when it asks for them. If a confidently wrong example draws a suspiciously small loss, double application is the first suspect.

The Quick Version

Stacked linear layers with no bend collapse into one linear layer, so depth without activations adds parameters but no new shapes.
Every layer multiplies the backward pass by its activation's slope, which means slopes under 1 compound toward nothing across depth.
Sigmoid peaks at slope 0.25 and saturates both ways, so twenty layers deliver about $10^{-12}$ of the gradient to the first.
Tanh centers its output at zero and peaks at slope 1, yet still saturates past an input magnitude near 2.
ReLU passes positives with slope exactly 1 and maps negatives to zero, which keeps deep gradients whole at the cost of units that can die.
Leaky ReLU keeps a small slope alive on the negative side, GELU smooths the kink for transformer stacks, and the output bend always follows the task.