SwiGLU
A gated feed-forward activation: one linear projection of the input is scaled elementwise by a Swish-activated projection before a final output projection.
Written , SwiGLU replaces a plain ReLU or GELU feed-forward activation with a gated one. Two separate projections of the same input multiply elementwise — one runs through Swish and acts as a learned valve, the other carries the content that valve lets through.
It costs a third weight matrix compared to a plain feed-forward block, which models typically offset by shrinking the expansion width. Most large language models built after 2022 use it, because it beats plain activations by a small but consistent margin at the same parameter count.