Neural Networks

Perceptron & Activation Functions

A neuron is a weighted sum followed by one squashing function — and the slope of that function, not the function itself, is what decides whether learning can happen at all.

Input value
Negative weight
Activation output
Slope of the activation

Input x₁

The first value entering the neuron.

Input x₂

The second value entering the neuron.

Weighted Sum (z)1.50

Two inputs and one output, with nothing between them yet but addition. Drag the input values and watch the number arrive.

Check your understanding

Push a value through a single neuron and watch each stage light up. Then drag along the activation curve until its derivative collapses to nothing.

A neural network is, at its core, built of very simple mathematical operations. Before we can talk about deep layers, backpropagation, or emergent behaviour, we have to look closely at the fundamental building block: the artificial neuron, or perceptron.

In this lab, we strip everything back to a single unit. You control two incoming values, the weight applied to each, and a bias term. The neuron simply multiplies each input by its weight, adds them together with the bias, and pushes the result through an activation function.

Why an activation function?

Without an activation function, a neural network would just be a series of linear transformations. No matter how many layers you stack, a composition of linear functions is always just another linear function. The network would only ever be able to draw straight lines, rendering depth entirely useless.

The activation function introduces non-linearity. It allows the network to bend and warp the space, giving it the capacity to learn complex, curving boundaries and intricate representations.

But this non-linearity comes with a hidden cost, and it's a cost that stalled the progress of deep learning for decades.

The vanishing gradient

During training, the network learns by adjusting its weights based on the error of its predictions. To know how to adjust a weight, the network uses calculus—specifically, the chain rule—to trace the error backward through the layers. This requires taking the derivative (slope) of the activation function.

If you push the weighted sum (z) too high or too low into a sigmoid or tanh function, the output flattens out. The slope approaches zero. When the slope is zero, the gradient becomes zero, and the error signal cannot travel back through that neuron. The network stops learning. This is the vanishing gradient problem.

The arithmetic is what makes it fatal rather than merely inconvenient. The sigmoid's slope never exceeds 0.25, anywhere, and a gradient travelling back through ten layers is multiplied by each layer's slope in turn. Ten layers at the very best possible slope already costs a factor of 0.25¹⁰ ≈ 9.5 × 10⁻⁷. Push one neuron past a sum of 2.89 and its slope drops under 0.05, where the same ten-layer product is about 10⁻¹³.

Move the Activation control between Sigmoid, Tanh and ReLU, and watch the dashed derivative curve change shape under the solid one. Then push Input x₁ to its maximum and watch the Gradient after 10 layers readout collapse. Tanh saturates even earlier than sigmoid, past a sum of 2.18 — what it buys is a peak slope of 1 instead of 0.25.

ReLU escapes the problem in one direction and invents a new one in the other: above zero its slope is exactly 1, so depth costs nothing, but at or below zero both its output and its slope are exactly 0 and no gradient will ever reach the weights that put it there. Select ReLU and drive the sum negative to see it.

Reference

Weighted sum: z = w₁x₁ + w₂x₂ + b
Sigmoid: σ(z) = 1 / (1 + e⁻ᶻ), σ′(z) = σ(z)·(1 − σ(z)) — peaks at 0.25
Tanh: tanh(z), derivative 1 − tanh²(z) — peaks at 1, and zero-centred
ReLU: max(0, z), derivative 1 for z > 0 and 0 for z < 0
Saturation: σ′ → 0 as |z| grows, so the gradient vanishes and learning halts
Saturation onset: σ′ falls below 0.05 past |z| = 2.89; tanh′ past |z| = 2.18
Dead ReLU: a unit stuck at z < 0 outputs 0 with derivative 0 — permanently
Depth cost: ten layers multiply the gradient by slope¹⁰ — 9.5e-7 at σ′s best, 1e-13 at 0.05
Why non-linear: composing linear maps gives a linear map, so depth would buy nothing

Break it on purpose

Push the weighted sum past 2.89 with a sigmoid and its derivative drops below 0.05: the neuron is saturated, and after ten such layers a gradient is down by a factor of 1e-13. Switch to ReLU and push the sum negative instead — the output and the derivative are both exactly zero, and the unit is dead for good.