Core ML

Activation Functions

Six functions, each fixing the last one's defect. Sigmoid's gradient dies with depth, ReLU's never shrinks, and the output layer's choice is the task's call.

Sigmoid and tanh flatten at both ends so their derivatives fall toward zero there, while ReLU holds a slope of exactly one across its whole positive side

Why Does This Exist?

Pull the nonlinearity out from between two layers and they collapse into one matrix — the multi-layer perceptron page walks through why. So something has to bend the signal.

The bend has a second job, and that job drove every change to this list since 1990. Whatever you drop in, the backward pass multiplies by its derivative — the slope at the value that came through. Twenty layers, twenty slopes multiplied, so any slope under 1 shrinks the product all the way down. (Derivatives covers the machinery.)

Keep one network in mind: twenty layers grading coffee beans off a belt as sound, insect-damaged, mouldy or broken.

Now the wall. Sigmoid squashes any number into the range from 0 to 1 with $\sigma(z) = 1/(1 + e^{-z})$ . Smooth, differentiable, and the obvious first choice because it looks like a firing rate. Its derivative is $\sigma(z)\big(1 - \sigma(z)\big)$ , which peaks at 0.25 when the input is zero and falls away in both directions.

So the best case at every layer is a quarter, and twenty layers of that is about $9 \times 10^{-13}$ . The first layer of the bean grader gets a gradient a trillion times weaker than the last layer's. Everything below is the sequence of fixes.

Think of It Like This

Twenty gearboxes bolted in series

Line up twenty gearboxes, input shaft to output shaft, and turn the last by hand. How far does the first shaft turn? Multiply the ratios.

A sigmoid layer is a gearbox whose best ratio is four-to-one, and that's at the sweet spot mid-travel: turn its output shaft a full revolution and the input shaft moves a quarter. Near either end of its range it's seized. Twenty in series, and a full revolution at the far end is under a trillionth of a turn at the near end.

A ReLU layer isn't a gearbox. It's a straight coupling with two states: engaged, passing the turn one-to-one, or declutched, passing nothing. Twenty engaged couplings transmit a full turn as a full turn. But a declutched coupling transmits exactly nothing, and if nothing re-engages it, that link is finished.

How It Actually Works

Sigmoid, tanh, and the break

Tanh was the first patch, and it fixed a separate problem. Sigmoid's output is always positive, so every unit above it gets inputs sharing a sign, which forces its weight gradients to share a sign and the weight vector zigzags. Tanh centres on zero, running from −1 to 1, with a derivative peaking at 1. Still saturates: past an input of about 2 its slope is under 0.08.

ReLU is the break, and it's almost embarrassingly plain: $\max(z, 0)$ . Its derivative is exactly 1 for any positive input, 0 for any negative one. Twenty layers of active units multiply to 1, so the first layer's gradient is the size of the loss's. It also costs one comparison rather than an exponential, which AlexNet's 2012 write-up named as part of why it trained.

The cost: half the input range outputs zero, with zero slope there too.

The dead unit, and the patches after it

A unit whose pre-activation is negative for every example outputs zero always and receives zero gradient always. It can't recover: the only thing that could move its weights is a gradient, and the gate carrying one is shut. Usually one oversized update shoved the bias far enough negative.

Leaky ReLU (2013) gives the negative side a small slope, typically 0.01, so a trickle survives and a stalled unit can climb back. Real improvement, small on most vision work.

GELU (2016) is smooth through zero instead of kinked, weighting each input by the probability a standard normal draw lands below it. Trains transformers better than ReLU, reliably enough that BERT and the GPT models adopted it.

SwiGLU (2020) leads in large language models. It splits a layer's input into two projections, passes one through a smooth activation, and multiplies them elementwise, so the gate is learned per unit. Cost: three weight matrices in that block instead of two.

The output layer is not a choice

Hidden layers: ReLU or GELU. Pick one and move on.

The output layer is decided by the task. An unbounded number, like a bean's mass — nothing, leave the score raw. One yes-or-no label — a sigmoid. Exactly one class out of several — softmax across the scores, as in softmax regression. Several labels at once, since a bean can be mouldy and broken together — independent sigmoids, one per label.

Those last two look interchangeable and aren't. Softmax forces the outputs to sum to 1, so raising one score lowers the others: right when one class holds, wrong when several can.

Show Me the Code

Twenty layers, two activations, printing the product of local slopes reaching layer one.

1import numpy as np2
3def depth_gradient(kind: str, layers: int = 20, width: int = 256) -> float:4    h = np.random.default_rng(0).normal(0.0, 1.0, width)5    reaching_layer_one = 1.0  # the product of local derivatives the chain rule multiplies6    for _ in range(layers):7        z = 0.9 * h  # a modest weight, so nothing below is an exploding-gradient artefact8        if kind == "sigmoid":9            h = 1.0 / (1.0 + np.exp(-z))10            local = float(np.mean(h * (1.0 - h)))  # tops out at 0.25 and only falls from there11        else:12            h = np.maximum(z, 0.0)13            local = float(np.mean(z > 0.0))  # exactly 1 on every unit that is switched on14        reaching_layer_one *= 0.9 * local15    return reaching_layer_one16
17for kind in ("sigmoid", "relu"):18    print(f"{kind:8s} {depth_gradient(kind):.3e}")19# -> sigmoid  2.047e-1420# -> relu     7.215e-08

A factor of three and a half million, from the choice of bend alone. ReLU's remaining shrinkage is units switched off rather than saturated — fixable.

Watch Out For

A layer where most units output zero for every input in the batch

One line measures it: take a batch, look at a hidden layer's output, count the fraction of units that are zero on every row. Healthy ReLU layers sit near half zeros per row, different units firing on different rows. A layer where the same 60 percent are zero on every row has 60 percent of its capacity switched off permanently.

The usual cause is a learning rate high enough that one early update drove those biases far negative. Lower the rate and restart, since a dead layer can't be repaired in place. Then check the initialisation scale: He initialisation exists because ReLU halves the variance.

Squashing the output and then handing it to a loss that wanted raw scores

Nearly every framework ships a cross-entropy loss that applies the sigmoid or softmax internally, for numerical reasons. Hand it probabilities that are already squashed and it squashes them again. No error, no warning, no shape mismatch. The loss even falls for a while, then the model plateaus early and well short.

The gradient is why: one squashing multiplies it by a factor under 1, two multiply twice, and the second is evaluated where the function is flattest. So know which convention your loss follows. Feed a confidently wrong example — doubly squashed, the loss is suspiciously small.

The Quick Version

Every layer multiplies the backward pass by its activation's slope, so slopes under 1 compound to nothing.
Sigmoid's slope peaks at 0.25. Twenty layers deliver around $10^{-12}$ to the first.
Tanh centres the output at zero, helping the next layer, and still saturates.
ReLU's slope is exactly 1 where the unit is on, so depth stops costing gradient.
Its price is the dead unit: always-negative input, always-zero output, always-zero gradient.
Leaky ReLU keeps a trickle alive, GELU is smooth through zero, SwiGLU adds a learned gate.