ResMLP Stabilizes MLP Vision Stacks

ResMLP adds residuals plus simple scaling to token and channel MLPs, so deep attention free stacks train stably.

Residuals plus 0.1 scaling train 24 MLP layers stably

Why Does This Exist?

Deep MLPs without attention drift: activations explode and early layers stall.

ResMLP borrows residuals. It follows mlp mixer mixing, steadies like gmlp gating needs, and serves image classification depth.

Think of It Like This

Handrails plus soft steps on stairs

Steep stairs need rails plus shallow risers. You hold steady while climbing higher safely.

Residuals are rails, LayerScale 0.1 is shallow steps. Where it stops: rails cannot fix wrong direction, just as residuals cannot fix bad mixing.

How It Actually Works

Block is $x + LayerScale(MLP(LayerNorm(x)))$ for token then channel steps. LayerScale multiplies each channel by learned $\lambda$ init $0.1$ . Train $12$ - $24$ layers with heavy augmentation plus distill.

Worked example

Activation norm $10.0$ through MLP gives $12.0$ . Without scale, $x + 12.0$ jumps. With $\lambda = 0.1$ , add $1.2$ for $11.2$ , a $12$ percent step not $120$ percent. Over $24$ layers totals stay bounded and gradients flow.

Watch Out For

Dropping LayerScale on 24 layer runs

Plain residuals still spike deep. Symptom: loss NaN after epoch 5. Fix: keep LayerScale init near $0.1$ and warm up before raising rates.

The Quick Version

ResMLP is Mixer plus residuals plus per-channel LayerScale.
Init scale $0.1$ keeps deep steps small and stable.
Fixture $10.0$ to $11.2$ shows 12 percent steps.
Trains $24$ layers to about 81 percent with help.
Keep scaling and warmup for deep stacks.