gMLP Gates Tokens Without Attention

gMLP splits channels and gates one half by a spatial projection of the other, adding location sense to MLPs cheaply.

Channel split plus 196 by 196 spatial gate adds order for 38K weights

Why Does This Exist?

Mixer mixes all tokens equally with no order bias. Language and vision both need order without full attention.

gMLP gates instead. It follows mlp mixer baselines, aids image classification scaling, and keeps pretrained models simple.

Think of It Like This

A dimmer wired to a neighbor room

Two rooms share power. One room dial sets brightness in the other, so activity couples without a central controller.

Split halves are rooms, spatial projection is wiring. Where it stops: fixed wiring cannot adapt per input, just as static gates lack input dependence.

How It Actually Works

Split $X$ into $U,V$ halves of $256$ channels. Spatial gate $S(V) = W V$ with $W$ shape $S \times S = 196 \times 196$ . Output $U \odot S(V)$ , where $\odot$ multiplies elementwise. Add channel MLPs around it.

Worked example

$S = 196$ . $W$ holds $196^2 = 38{,}416$ weights. Gate for token $5$ is $\sum_{j=1}^{196} W_{5,j} V_j$ . If row $5$ peaks $0.6$ on token $5$ plus $0.2$ on $6$ , output keeps local order. Cost $38$ K versus attention $QK$ per head per layer dynamic.

Watch Out For

Expecting input adaptive routing from static gates

$W$ is fixed after training, unlike attention. Symptom: poor handling of shuffled layouts. Fix: use attention or dynamic gates when order varies strongly per input.

The Quick Version

gMLP splits channels and gates half by spatial projection.
Gate matrix $196 \times 196$ holds $38$ K static weights.
Order emerges without attention or positions.
Static gates cannot adapt per input like attention.
Pair with channel MLPs for full blocks.