Mamba
Mamba lets each token turn its own dial on memory. Important tokens rewrite the state strongly while filler tokens pass through, so long sequences cost grows in a straight line.
Why Does This Exist?
Plain state space models run in linear time by squeezing the past into a fixed-size state. Their weakness is that the squeeze is blind. The matrices that decide how much each token rewrites memory are fixed, so a filler word like the pushes just as hard as a name or a date. Old facts fade at the same rate no matter how much they mattered.
Attention fixes the blindness by letting every step reread every earlier step directly, but pairs grow with the square of length, so a 4,096-token page needs millions of pairs. Mamba asks for both: keep the linear budget of a state model while letting each token control its own update. This page covers that dial, what it costs, and where it still loses to direct lookup. The dense baseline it competes with lives in transformers. The IO-aware speed trick on the attention side lives in flash attention.
One basic makes the rest click. A state is a short running summary, updated per token as . The step size sets how much the new input matters against old memory. Small keeps memory, large rewrites it.
Think of It Like This
A notebook grip that presses harder on key lines
You take notes while someone reads a report aloud. Most sentences get a light pencil tick that barely marks the page. When a sentence holds a name, a date or a decision, you press hard so it bites into the paper and stays readable later.
The notebook is the state, your hand pressure is , light ticks are filler tokens and hard presses are salient tokens. The page ends up with key lines prominent and filler already faded, without rereading the whole report.
Where it stops: paper keeps every hard press exactly, while a fixed-size state still blends presses together. Press hard on two conflicting facts in a row and the second smudges the first. The notebook never runs out of pages, but the state does.
How It Actually Works
Call the input at step by , meaning the current token vector. Call the step size for that token, the fixed memory decay with negative entries for stability, and the fixed input map. Mamba makes a function of the input:
Here and are a small learned projection, and softplus keeps the result positive. Discretize with that :
The update is . Large pushes toward zero, so old memory fades and the new token dominates. Small keeps near one, so memory survives and the token barely registers. The same also gates the output, so filler stays quiet downstream too.
Worked example: one hard press and one light tick
Take scalar decay for clarity. A key token produces :
Only 64 percent of old memory survives, so the new token writes strongly. A filler token produces :
Here 95 percent of memory survives, so the token passes through. If , the two writes contribute about and respectively, a nine-fold gap from the same machinery. That gap is the diagram headline: one thick arrow absorbing input, one thin arrow preserving memory.
Why training needs a new scan
Fixed-step models reuse one kernel for every position, so training parallelizes as one convolution. Input-dependent breaks that sharing because the kernel differs per position. Mamba replaces the convolution with a hardware-aware parallel scan that respects the per-step while still using fast on-chip memory, in the same spirit as linear attention trading exactness for linear cost.
Code
The point is the check: rebuild both memory factors and confirm 0.64 against 0.95 from the worked example.
import math
A: float = -0.5delta_key: float = 0.90delta_filler: float = 0.10
a_key: float = math.exp(delta_key * A)a_filler: float = math.exp(delta_filler * A)
print(round(a_key, 2)) # -> 0.64print(round(a_filler, 2)) # -> 0.95
Bx: float = 1.0print(round(delta_key * Bx, 2)) # -> 0.9print(round(delta_filler * Bx, 2)) # -> 0.1The printout matches the hand computation: key token keeps 0.64 of memory and writes 0.90, filler keeps 0.95 and writes 0.10. Raise the filler to 0.90 and both rows print the same pair.
Watch Out For
Expecting exact recall from a compressed state
Mamba still stores the past in a fixed-size vector. Plant one passcode 3,000 tokens back and ask for it verbatim, and the state may have blended it away under later writes even with good gating.
Symptom: strong reasoning but missed needles on long retrieval tests. Fix: pair selective layers with periodic full attention mechanism layers for needle tasks instead of trusting compression alone.
Porting convolution training code unchanged
Standard state models train with one fast convolution because the kernel is fixed. Input-dependent steps need the parallel scan path, or training falls back to a slow serial loop.
Symptom: tiny runs look right but large runs take far longer than the linear promise. Fix: use the selective scan kernel for Mamba blocks and keep convolution code only for fixed-step baselines in state space models.
The Quick Version
- Mamba makes the step size a learned function of each token through softplus of a small projection.
- With decay minus 0.50, Delta 0.90 gives memory factor about 0.64 while Delta 0.10 gives about 0.95.
- The same Delta scales the write, so the key token contributes about 0.90 against 0.10 for filler, a nine-fold gap.
- Input dependence breaks single-kernel convolution training, so Mamba uses a hardware-aware parallel scan instead.
- Fixed-size memory still blends old facts, so exact long-range retrieval favors hybrids with attention layers.
- Cost per token stays linear in length, unlike the quadratic pair cost of full attention.