MLP-Mixer Mixes Patches Without Attention

MLP-Mixer alternates token mixing across patches with channel mixing per patch, proving MLPs alone classify well.

Transpose mixing across 196 patches plus channels replaces attention

Why Does This Exist?

Attention and convs were assumed required. Mixer tests the floor with only MLPs and transposes.

It frames image classification baselines, contrasts convolutional layers bias, and precedes later MLP variants.

Think of It Like This

Sorting a spreadsheet by rows then columns

You sort student rows to compare across class, then sort subject columns per student. Two one-dimensional sorts cover the table.

Token MLP sorts rows, channel MLP sorts columns. Where it stops: sorts ignore distance, just as Mixer ignores geometry.

How It Actually Works

Patches $X$ shape $S \times C$ with $S = 196$ , $C = 512$ . Token mix: $U = X + W_2 GELU(W_1 LayerNorm(X)^T)^T$ across $S$ . Channel mix: $Y = U + W_4 GELU(W_3 LayerNorm(U))$ across $C$ . Repeat $12$ times, pool, classify.

Worked example

$S = 196$ , hidden token dim $128$ . Token MLP uses $196 \times 128 + 128 \times 196 = 25{,}088 + 25{,}088 = 50{,}176$ weights. Channel MLP with $C = 512$ , hidden $2048$ uses $512 \times 2048 + 2048 \times 512 = 2{,}097{,}152$ . Token step is $42$ times smaller yet carries spatial talk.

Watch Out For

Training Mixer on small data from scratch

Mixer has no locality bias so 1M photos underfit. Symptom: 10 point gap to CNNs on small sets. Fix: pretrain large or distill before judging small runs.

The Quick Version

Mixer alternates token MLPs across patches and channel MLPs per patch.
No attention, convolution, or positions beyond patches.
Fixture token step 50K weights versus 2.09M channel step.
Needs large pretraining to match biased models.
Serves as the clean MLP baseline.