Activation Function
The little non-linear function each neuron applies to its input, letting a network learn curved, complex patterns instead of just straight lines. Without it, stacking layers would be pointless.
Think of It Like This
The bend in the wire — without a kink, ten straight segments are still just one straight line.
Each neuron sums its weighted inputs and passes the result through an activation like ReLU, sigmoid, or tanh. That non-linearity is exactly what gives a deep network its power: remove it and any number of layers collapses into a single linear step. ReLU — keep positives, zero out negatives — is the modern default because it's cheap to compute and trains well.
import numpy as np
def relu(x: np.ndarray) -> np.ndarray: """The most common activation: pass positives through, clamp negatives to 0.""" return np.maximum(0, x)