GELU
A widely used activation function in modern transformers that weights neural inputs by their value, smoothing decision boundaries and improving model training.
Think of It Like This
Like a bouncer who strictly denies people with terrible attitudes, lets great people in immediately, and takes a nuanced approach to everyone in between.
The Gaussian Error Linear Unit (GELU) merges the properties of dropout, zoneout, and ReLUs. Unlike ReLU, which sharply cuts off negative values at zero, GELU applies a smooth, probabilistic transformation based on the standard normal distribution. This smoothness helps gradients flow more reliably during backpropagation, leading to faster convergence in large networks.