S4 Brings Structured States to Vision

S4 parameterizes state matrices for long memory with fast kernels, giving vision a linear long-range option before selection.

Structured 64 dim states remember 16K pixels with stable gradients

Why Does This Exist?

RNNs vanish on $16$ K pixel strings. Attention handles them but costs $256$ M pairs.

S4 structures memory. It precedes vision mamba selection, grounds state spaces, and informs convolutional layers long-range debates.

Think of It Like This

Tuned echo chambers instead of plain rooms

Plain rooms echo muddy. Shaped chambers hold tones for seconds with clear decay.

HiPPO init shapes chambers, discretization tunes decay. Where it stops: fixed shapes suit all inputs alike, just as S4 lacks per-input selection.

How It Actually Works

Discretize $A,B$ with step $\Delta$ to $\bar{A},\bar{B}$ . HiPPO sets $A$ to memorize history optimally in $64$ dims. Run as recurrence or long conv kernel $K = (C\bar{B}, C\bar{A}\bar{B}, ...)$ .

Worked example

$128 \times 128 = 16{,}384$ pixels. Attention $16{,}384^2 = 268$ M pairs. S4 kernel length $16{,}384$ with state $64$ runs $O(N \log N)$ near $16{,}384 \times 14 \approx 229$ K steps. Decay $0.99^{1000} = 0.00004$ versus HiPPO $0.42$ retention shows long memory kept.

Watch Out For

Using S4 where selection matters most

Fixed kernels average clutter with targets. Symptom: fine detail washes on busy scenes. Fix: move to selective Mamba variants for cluttered dense tasks.

The Quick Version

S4 structures $64$ dim states with HiPPO for long memory.
Runs as recurrence or $16$ K long conv kernels.
Fixture $229$ K steps replace $268$ M pairs.
Fixed after training, unlike selective scans.
Use selective successors for cluttered scenes.