Activation Function
A mathematical operation applied to a neural network node's output that introduces the crucial non-linearity required to learn highly complex patterns.
Think of It Like This
Like a gatekeeper deciding how loudly a message should be shouted to the next room based on the importance of the incoming signal.
Without these non-linear transformations, deep networks would simply collapse into a single giant linear regression model, regardless of their depth. Functions like ReLU, GELU, and Swish dictate how aggressively a neuron fires based on its inputs, effectively deciding which information flows forward. The choice of activation function directly impacts a network's training stability and convergence speed.