State Spaces Model Long Vision Context

State space layers scan patches with linear recurrence, so global context costs linear steps instead of quadratic attention.

Linear scans replace 9.8M pairs with 3136 steps for global context

Why Does This Exist?

Attention on $56 \times 56$ costs $9.8$ M pairs. High-res plus video explode faster than GPUs grow.

Recurrence scans linearly. This overview frames vision mamba, vmamba, and S4 vision details.

Think of It Like This

A running tally instead of all pairs polling

Polling $3136$ friends pairwise needs $9.8$ M chats. Keeping one running tally needs $3136$ updates with memory of past.

Attention is polling, state scan is tally. Where it stops: tallies compress history, just as states forget far detail without selection.

How It Actually Works

State $h_t = A h_{t-1} + B x_t$ , output $y_t = C h_t$ , where $A,B,C$ are learned with discretization step $\Delta$ . Selective variants make $B,C,\Delta$ input dependent. Scan patches in rows plus columns for 2D.

Worked example

$N = 3136$ . Attention pairs $3136^2 = 9{,}834{,}496$ . Scan steps $3136$ with state $64$ , mults $3136 \times 64 \times 2 \approx 401$ K, about $24$ times fewer. Scalar run $h = 0.9 h + 0.5 x$ : inputs $[1,0,1]$ give $h_1 = 0.5$ , $h_2 = 0.45$ , $h_3 = 0.905$ , showing memory decay plus update.

Watch Out For

Scanning one direction only on 2D scenes

Causal row scans miss right-to-left context. Symptom: left bias in masks. Fix: use bidirectional plus cross scans as VMamba does.

The Quick Version

State layers recur $h = Ah + Bx$ and read $y = Ch$ linearly.
Fixture $401$ K mults replace $9.8$ M pairs, 24 times cheaper.
Selection makes $B,C$ input dependent to keep key detail.
Scan multiple directions for 2D fairness.
Details live in Vision Mamba, VMamba, and S4 pages.