Agent Memory Systems That Persist State

Why Does This Exist?

A model call is stateless. It reads exactly the tokens in its window and keeps none afterward. For one question that hardly matters. For an agent serving the same user for weeks, it breaks everything: preferences restated every Monday, fixes relearned every sprint, addresses re-asked after a move.

The naive fix pastes old chats into every prompt until the window bursts. Six past sessions at 500 tokens each plus 400 of system and 700 of task want $400 + 700 + 3000 = 4100$ tokens against a 4000 window, over by 100. Memory systems replace that paste with a file and a pull: write distilled facts to durable storage once, retrieve the top match per task, and leave the rest filed.

This page covers the stores, the write decision, and the retrieve step. What each type is for lives in agent memory types. The loop that spends the assembled prompt lives in agent architectures, and the plans that survive across sessions live in long-horizon planning. The general short versus long-term split lives in memory systems, while the episode-to-fact distillation lives in episodic and semantic memory.

Basics first: the context window is the maximum tokens one call can read, prompt plus reply together. Retrieval usually ranks by similarity over embeddings. If windows are new, the worked example shows the token arithmetic before any jargon arrives.

Think of It Like This

How It Actually Works

Call $W$ the window: the most tokens one request may hold. Call $S$ the live contents: system instructions, recent turns, and whatever retrieval just added. Call $M$ the durable pile: one row per saved fact with its source. A write moves a fact from $S$ into $M$. A retrieve moves a few rows from $M$ back into $S$ for one request. Saved and used are two steps, and skipping the second builds a store nobody reads.

The write is a durability judgment

A one-off detail, like renaming one variable today, belongs in $S$ and dies there. A repeated preference, like pytest over unittest stated twice and confirmed by edits, earns a row in $M$ with its source episode. When in doubt, do not write. Every extra row taxes every future search. The step loop that produces these facts lives in ReAct, and the milestone state it must survive lives in long-horizon planning.

Retrieval ranks facts against today

Call the current task $q$: the query for what you need right now. Call each stored fact $m_i$: fact number $i$ as a vector. Call $s_i$ its score against $q$. Call $k$ how many you pull back: usually one or two. You rank by $s_i$ and copy only the top $k$ rows into $S$.

Worked example: three facts ranked, one budget checked

Take three stored facts as two-dimensional vectors so you can check the math. Real stores use hundreds of dimensions. The arithmetic is the same. Here $q = [2, 1]$ stands in for writing a pytest fixture, $m_1 = [2, 1]$ is prefers pytest, $m_2 = [1, 2]$ is deploys on Fridays, and $m_3 = [0, 2]$ is likes demo GIFs.

Score with cosine similarity, which is alignment divided by lengths. The length of $[a, b]$ is $\sqrt{a^2 + b^2}$. Lengths: $\lVert q \rVert = \sqrt{4 + 1} = \sqrt{5} \approx 2.2361$, same for $m_1$ and $m_2$. $\lVert m_3 \rVert = \sqrt{0 + 4} = 2$. Dots: $q \cdot m_1 = 4 + 1 = 5$. $q \cdot m_2 = 2 + 2 = 4$. $q \cdot m_3 = 0 + 2 = 2$. Divide: $s_1 = 5 / (2.2361 \times 2.2361) = 5 / 5 = 1.0$. $s_2 = 4 / 5 = 0.8$. $s_3 = 2 / (2.2361 \times 2) = 2 / 4.4721 \approx 0.4472$. Ranked, $1.0$ beats $0.8$ beats $0.4472$. With $k = 2$ you pull back the first two. The diagram shows these three scores and the two returning arrows.

Now the token reason this matters. Name $W = 4000$. System needs 400 tokens and today's task needs 700. Pasting six old sessions at 500 tokens each costs $6 \times 500 = 3000$, so the request wants $400 + 700 + 3000 = 4100$. Overflow is $4100 - 4000 = 100$. The store instead holds 20 facts at 12 tokens each: $20 \times 12 = 240$ tokens total. Pulling the top 2 costs 24. The request becomes $400 + 700 + 24 = 1124$, leaving $4000 - 1124 = 2876$ spare. Same history, two shapes: one never fits, one barely registers.

That trade assumes past choices predict future ones. When the user switches runners or moves cities, old rows lie until a new episode overwrites them.

Code

The snippet reruns the ranking and the token fit, so you can check both numbers yourself:

1import math2
3
4def cosine(u: list[float], v: list[float]) -> float:5    dot = sum(x * y for x, y in zip(u, v))6    nu = math.sqrt(sum(x * x for x in u))7    nv = math.sqrt(sum(x * x for x in v))8    return dot / (nu * nv)9
10
11query = [2.0, 1.0]12mems = [[2.0, 1.0], [1.0, 2.0], [0.0, 2.0]]13print([round(cosine(query, m), 4) for m in mems])  # -> [1.0, 0.8, 0.4472]14print(400 + 700 + 6 * 500 - 4000)  # -> 10015print(20 * 12)  # -> 24016print(400 + 700 + 24)  # -> 1124

Four outputs match the worked example: the three scores, the 100-token overflow, the 240-token store, and the 1124-token request.

Watch Out For

The Quick Version

• Short-term memory is the live window. It costs nothing to read and it vanishes when the session ends.
• Long-term memory is facts filed on purpose outside the window. Filed facts do nothing until retrieval copies them back.
• Three facts scoring 1.0, 0.8, and 0.4472 retrieve the top two, turning a 3000-token paste into a 24-token pull.
• A 4100-token paste overflows a 4000 window by 100, while an 1124-token memory request fits with 2876 spare.
• Twenty 12-token facts cost 240 tokens total, a fraction of one pasted session.
• Writes should record repeated preferences, not one-off details, and old rows must be overwritten when the world changes.

What to Read Next

• Agent Memory Types
• Long-Horizon Planning for AI Agents
• Planning in Agents: Goals to Steps