Feed-Forward Networks

After attention mixes information across positions, each position still needs its own nonlinear update. A feed-forward network widens it, bends it, then squeezes it back.

One position with values 2 and minus 1 widens to 2, 0, 0 and 5, then projects to 4.5 and 1.0 with the same weights used at every position

Why Does This Exist?

Self-attention is a weighted average of value vectors. However clever the weights, it only routes and blends information that already exists across positions. It cannot bend one position into something new on its own.

That gap is why every transformer block has a second sublayer. Attention decides what to gather from where. The feed-forward network decides what to do with what was gathered, one position at a time.

One prerequisite keeps this page short. An activation function bends a number, such as ReLU keeping positives and zeroing negatives. The skip wiring around the block lives in residual connections. This page covers the widen, bend, and squeeze in between.

Think of It Like This

A prep cook with one bowl per order

Dinner service shares ingredients across the kitchen. One cook walks the line tasting pots and gathering a spoon of this and a spoon of that for each order. That gathering round is attention.

Then the cook returns to the station, closes off one bowl at a time, chops, seasons, and reduces it into something the raw spoons were not. Every bowl gets the same recipe steps, but no bowl borrows from another bowl during this part. That solo work is the feed-forward network.

Each gathered spoon mix is one token vector, the wider counter space is the hidden dimension, chopping and seasoning is the nonlinearity, and plating back into the same bowl size is the down projection.

Where it stops: a cook tastes and improvises per bowl. A feed-forward network applies the same fixed matrices at every position with no judgment about the content.

How It Actually Works

Call the token vector $x$ with dimension $d$ . Call $W_1$ the up matrix with shape $d$ by $h$ , where $h$ is the wider hidden size, often four times $d$ . Call $b_1$ its bias. Call $\sigma$ the elementwise bend such as ReLU or GELU. Call $W_2$ the down matrix with shape $h$ by $d$ and $b_2$ its bias. The sublayer is:

\text{FFN}(x) = W_2 \, \sigma(W_1 x + b_1) + b_2

The same $W_1$ , $b_1$ , $W_2$ , $b_2$ run at every position. One row never reads another row here. Mixing across positions belongs only to self-attention. Transforming within a position belongs only here. The block output then adds this result back to its input through residual connections, which is why shapes must match at $d$ in and $d$ out.

Worked example: 2 values widen to 4 then return to 2

Take $x = [2, -1]$ so $d = 2$ and $h = 4$ . Pick $W_1$ with columns $[1, 0]$ , $[0, 1]$ , $[-1, 1]$ , $[2, -1]$ written as rows $[[1, 0, -1, 2], [0, 1, 1, -1]]$ . Pick $b_1 = [0, 1, 3, 0]$ .

Multiply and add:

x W_1 + b_1 = [2, -1, -3, 5] + [0, 1, 3, 0] = [2, 0, 0, 5]

Apply ReLU, which keeps positives and zeroes the rest:

\sigma([2, 0, 0, 5]) = [2, 0, 0, 5]

Pick $W_2$ with rows $[1, 0]$ , $[0, 1]$ , $[1, 1]$ , $[0.5, 0]$ and $b_2 = [0, 1]$ . Project down:

[2, 0, 0, 5] W_2 + b_2 = [4.5, 0] + [0, 1] = [4.5, 1.0]

That pair is the diagram headline: input $2$ and $-1$ on the left, hidden $2$ , $0$ , $0$ and $5$ in the middle, output $4.5$ and $1.0$ on the right.

Why the same weights at every position

Sharing keeps the parameter count fixed as sequences grow and forces one clear job per sublayer. A multi-layer perceptron trained alone sees whole rows. Here the same small perceptron sees each row separately. Change row two of the input and only row two of the output moves.

Code

Rebuild the worked example with plain Python and confirm $4.5$ and $1.0$ .

1def relu_vec(v: list[float]) -> list[float]:2    return [a if a > 0.0 else 0.0 for a in v]3
4
5def ffn_step(x: list[float], w1: list[list[float]], b1: list[float], w2: list[list[float]], b2: list[float]) -> list[float]:6    pre: list[float] = [x[0] * w1[0][j] + x[1] * w1[1][j] + b1[j] for j in range(4)]7    hid: list[float] = relu_vec(pre)8    out: list[float] = [hid[0] * w2[0][k] + hid[1] * w2[1][k] + hid[2] * w2[2][k] + hid[3] * w2[3][k] + b2[k] for k in range(2)]9    return out10
11
12x: list[float] = [2.0, -1.0]13w1: list[list[float]] = [[1.0, 0.0, -1.0, 2.0], [0.0, 1.0, 1.0, -1.0]]14b1: list[float] = [0.0, 1.0, 3.0, 0.0]15w2: list[list[float]] = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0], [0.5, 0.0]]16b2: list[float] = [0.0, 1.0]17
18print(ffn_step(x, w1, b1, w2, b2))  # -> [4.5, 1.0]

The printout matches the hand arithmetic. Change $x$ to $[2.0, 0.0]$ and only the same row result moves, which proves no cross position read.

Watch Out For

Assuming this sublayer mixes positions

Attention sits right before it, so it is easy to credit this step with cross token work. By construction $W_1$ and $W_2$ touch one row at a time with shared weights.

Symptom: an edit to token three is expected to shift token five through this sublayer alone. Fix: trace the matrices row by row as in self-attention for mixing versus here for solo transforms, and test by changing one input row.

Starving the hidden width to save parameters

The $4d$ width looks wasteful, so a custom build cuts it to $d$ . Accuracy drops because the bend has no room to reshape the content.

Symptom: training loss stalls while attention maps look sensible. Fix: keep the expansion for working space and save parameters in residual connections or fewer layers instead, where the cut hurts less.

The Quick Version

A feed-forward network transforms one position at a time with no reads across positions.
Our input 2 and minus 1 widens to 2, 0, 0 and 5 after adding bias and applying ReLU.
Those four hidden values project down to 4.5 and 1.0 through the second matrix plus bias.
The same two matrices run at every position, so longer sequences add no new weights here.
Attention routes between positions while this sublayer bends within a position.
Most transformer parameters sit in these two matrices, not in the attention projections.