Skip to content
AI360XpertBeta

Mamba

Mamba lets each token turn its own dial on memory. Important tokens rewrite the state strongly while filler tokens pass through, so long sequences cost grows in a straight line.

A key token with Delta 0.90 shrinks memory to 0.64 and absorbs input while filler with Delta 0.10 keeps 0.95 of memory
A key token with Delta 0.90 shrinks memory to 0.64 and absorbs input while filler with Delta 0.10 keeps 0.95 of memory

Why Does This Exist?

Plain state space models run in linear time by squeezing the past into a fixed-size state. Their weakness is that the squeeze is blind. The matrices that decide how much each token rewrites memory are fixed, so a filler word like the pushes just as hard as a name or a date. Old facts fade at the same rate no matter how much they mattered.

Attention fixes the blindness by letting every step reread every earlier step directly, but pairs grow with the square of length, so a 4,096-token page needs millions of pairs. Mamba asks for both: keep the linear budget of a state model while letting each token control its own update. This page covers that dial, what it costs, and where it still loses to direct lookup. The dense baseline it competes with lives in transformers. The IO-aware speed trick on the attention side lives in flash attention.

One basic makes the rest click. A state hh is a short running summary, updated per token as h=Ah+Bxh = A h + B x. The step size Δ\Delta sets how much the new input xx matters against old memory. Small Δ\Delta keeps memory, large Δ\Delta rewrites it.

Think of It Like This

A notebook grip that presses harder on key lines

You take notes while someone reads a report aloud. Most sentences get a light pencil tick that barely marks the page. When a sentence holds a name, a date or a decision, you press hard so it bites into the paper and stays readable later.

The notebook is the state, your hand pressure is Δ\Delta, light ticks are filler tokens and hard presses are salient tokens. The page ends up with key lines prominent and filler already faded, without rereading the whole report.

Where it stops: paper keeps every hard press exactly, while a fixed-size state still blends presses together. Press hard on two conflicting facts in a row and the second smudges the first. The notebook never runs out of pages, but the state does.

How It Actually Works

Call the input at step tt by xtx_t, meaning the current token vector. Call Δt\Delta_t the step size for that token, AA the fixed memory decay with negative entries for stability, and BB the fixed input map. Mamba makes Δt\Delta_t a function of the input:

Δt=softplus(Wxt+b)\Delta_t = \mathrm{softplus}(W x_t + b)

Here WW and bb are a small learned projection, and softplus keeps the result positive. Discretize with that Δt\Delta_t:

Aˉt=exp⁡(ΔtA),Bˉt≈ΔtB\bar{A}_t = \exp(\Delta_t A), \quad \bar{B}_t \approx \Delta_t B

The update is ht=Aˉtht−1+Bˉtxth_t = \bar{A}_t h_{t-1} + \bar{B}_t x_t. Large Δt\Delta_t pushes Aˉt\bar{A}_t toward zero, so old memory fades and the new token dominates. Small Δt\Delta_t keeps Aˉt\bar{A}_t near one, so memory survives and the token barely registers. The same Δt\Delta_t also gates the output, so filler stays quiet downstream too.

Worked example: one hard press and one light tick

Take scalar decay A=−0.5A = -0.5 for clarity. A key token produces Δ=0.90\Delta = 0.90:

Aˉ=exp⁡(0.90×−0.5)=exp⁡(−0.45)≈0.64\bar{A} = \exp(0.90 \times -0.5) = \exp(-0.45) \approx 0.64

Only 64 percent of old memory survives, so the new token writes strongly. A filler token produces Δ=0.10\Delta = 0.10:

Aˉ=exp⁡(0.10×−0.5)=exp⁡(−0.05)≈0.95\bar{A} = \exp(0.10 \times -0.5) = \exp(-0.05) \approx 0.95

Here 95 percent of memory survives, so the token passes through. If Bxt=1.00B x_t = 1.00, the two writes contribute about 0.900.90 and 0.100.10 respectively, a nine-fold gap from the same machinery. That gap is the diagram headline: one thick arrow absorbing input, one thin arrow preserving memory.

Why training needs a new scan

Fixed-step models reuse one kernel for every position, so training parallelizes as one convolution. Input-dependent Δt\Delta_t breaks that sharing because the kernel differs per position. Mamba replaces the convolution with a hardware-aware parallel scan that respects the per-step Aˉt\bar{A}_t while still using fast on-chip memory, in the same spirit as linear attention trading exactness for linear cost.

Code

The point is the check: rebuild both memory factors and confirm 0.64 against 0.95 from the worked example.

import math
A: float = -0.5delta_key: float = 0.90delta_filler: float = 0.10
a_key: float = math.exp(delta_key * A)a_filler: float = math.exp(delta_filler * A)
print(round(a_key, 2))  # -> 0.64print(round(a_filler, 2))  # -> 0.95
Bx: float = 1.0print(round(delta_key * Bx, 2))  # -> 0.9print(round(delta_filler * Bx, 2))  # -> 0.1

The printout matches the hand computation: key token keeps 0.64 of memory and writes 0.90, filler keeps 0.95 and writes 0.10. Raise the filler Δ\Delta to 0.90 and both rows print the same pair.

Watch Out For

Expecting exact recall from a compressed state

Mamba still stores the past in a fixed-size vector. Plant one passcode 3,000 tokens back and ask for it verbatim, and the state may have blended it away under later writes even with good gating.

Symptom: strong reasoning but missed needles on long retrieval tests. Fix: pair selective layers with periodic full attention mechanism layers for needle tasks instead of trusting compression alone.

Porting convolution training code unchanged

Standard state models train with one fast convolution because the kernel is fixed. Input-dependent steps need the parallel scan path, or training falls back to a slow serial loop.

Symptom: tiny runs look right but large runs take far longer than the linear promise. Fix: use the selective scan kernel for Mamba blocks and keep convolution code only for fixed-step baselines in state space models.

The Quick Version

  • Mamba makes the step size Δt\Delta_t a learned function of each token through softplus of a small projection.
  • With decay minus 0.50, Delta 0.90 gives memory factor about 0.64 while Delta 0.10 gives about 0.95.
  • The same Delta scales the write, so the key token contributes about 0.90 against 0.10 for filler, a nine-fold gap.
  • Input dependence breaks single-kernel convolution training, so Mamba uses a hardware-aware parallel scan instead.
  • Fixed-size memory still blends old facts, so exact long-range retrieval favors hybrids with attention layers.
  • Cost per token stays linear in length, unlike the quadratic pair cost of full attention.