Memory Systems

Why Does This Exist?

Say you use a coding assistant for weeks. In week one you tell it you test with pytest and you want type hints everywhere. In week eleven you expect type hinted code and pytest fixtures without repeating yourself. That only happens if something from week one survived after its window closed.

A model call is stateless. It reads exactly the tokens in its context window and keeps none of them afterward. Context engineering decides what fits inside that window right now. Memory systems decide what stays true next time, without pasting every old chat into every new prompt.

This page covers that system: what counts as short term, what earns a durable write, and how retrieval pulls facts back. The split inside durable storage lives in episodic and semantic memory. The loop that spends memory on real tasks lives in AI agent architecture.

Think of It Like This

How It Actually Works

Model calls forget by design, so an agent needs two kinds of memory. Short-term memory is whatever sits in the window right now. Long-term memory is facts written somewhere durable on purpose, plus the two decisions that make them useful: what earns a write, and what gets retrieved.

Short-term ends when the window closes

Call $W$ the window: the most tokens one request may hold, input plus output. Call the live contents $S$: system instructions, recent turns, and whatever retrieval just added. $S$ is free because it is already there, and temporary because the next session starts with an empty $S$. Nothing is saved by staying in $S$.

If the frame itself is fuzzy, start with context windows and the token math in context budgeting.

Long-term needs a write and a retrieve

Call the durable pile $M$: one row per saved fact, each with its source. A write moves a fact from $S$ into $M$. A retrieve moves a few rows from $M$ back into $S$ for one request. Saved and used are two steps, and skipping the second is how teams end up with a store nobody reads.

The write is a durability judgment. A one off instruction, like renaming one variable today, belongs in $S$ and dies there. A repeated preference, like pytest over unittest stated twice and confirmed by your edits, earns a row in $M$. When in doubt, do not write. Every extra row taxes every future search.

Retrieval is usually similarity search over embeddings. Call the current task $q$: the query vector for what you need right now. Call each stored fact $m_i$: fact number $i$ written as a vector. Call $s_i$ its score against $q$: how closely fact $i$ matches this task. Call $k$ how many you pull back: usually one or two. You rank by $s_i$ and copy only the top $k$ rows into $S$. The rest stay filed.

Worked example: three facts ranked, one pulled back

Take three stored facts, each as a two dimensional vector so you can check the math. Real stores use hundreds of dimensions. The arithmetic is the same.

Here $q = [3, 1]$ stands in for writing a pytest fixture, $m_1 = [3, 1]$ is prefers pytest, $m_2 = [2, 2]$ is deploys on Fridays, and $m_3 = [0, 3]$ is likes demo GIFs.

Score with cosine similarity, which is alignment divided by lengths. The length of a vector $[a, b]$ is $\sqrt{a^2 + b^2}$, the straight line distance from the origin.

Lengths first: $\lVert q \rVert = \sqrt{3^2 + 1^2} = \sqrt{10} \approx 3.1623$, and the same for $m_1$. $\lVert m_2 \rVert = \sqrt{2^2 + 2^2} = \sqrt{8} \approx 2.8284$. $\lVert m_3 \rVert = \sqrt{0^2 + 3^2} = 3$.

Dots next: $q \cdot m_1 = 3 \times 3 + 1 \times 1 = 10$. $q \cdot m_2 = 3 \times 2 + 1 \times 2 = 8$. $q \cdot m_3 = 3 \times 0 + 1 \times 3 = 3$.

Divide: $s_1 = 10 / (3.1623 \times 3.1623) = 10 / 10 = 1.0$. $s_2 = 8 / (3.1623 \times 2.8284) = 8 / 8.9443 \approx 0.8944$. $s_3 = 3 / (3.1623 \times 3) = 3 / 9.4868 \approx 0.3162$. Ranked, $1.0$ beats $0.8944$ beats $0.3162$. With $k = 1$ you pull back $m_1$ only. The diagram shows these three scores and the one returning arrow.

Now the token reason this matters. Name $W = 4000$ for the fixture window. System needs $400$ tokens and today's task needs $800$. Pasting six old sessions at $500$ tokens each costs $6 \times 500 = 3000$, so the request wants $400 + 800 + 3000 = 4200$. Overflow is $4200 - 4000 = 200$. Something must give.

The store holds twelve facts at $15$ tokens each: $12 \times 15 = 180$ tokens total, which is $180 / 4000 = 4.5\%$ of the window. Pulling the winner costs $15$. The request becomes $400 + 800 + 15 = 1215$, leaving $4000 - 1215 = 2785$ spare. Same history, two shapes: one never fits, one barely registers.

That is the trade. Persistence costs judgment at write time and a search at read time, and it buys recall across sessions for a fraction of the window. Skip it for single session tools like a summarizer, where yesterday says nothing about today and the machinery never pays off. It assumes past choices predict future ones. When you switch runners or tastes, old rows lie until a new episode overwrites them, which is why the overwrite rule in episodic and semantic memory is load bearing.

Code

The snippet reruns the ranking and the token fit, so you can check both numbers yourself.

1import math2
3
4def norm(v: list[float]) -> float:5    """Straight-line length from the origin."""6    return math.sqrt(sum(x * x for x in v))7
8
9def cosine(u: list[float], v: list[float]) -> float:10    """Alignment divided by lengths, between -1 and 1."""11    return sum(x * y for x, y in zip(u, v)) / (norm(u) * norm(v))12
13
14query: list[float] = [3.0, 1.0]15memories: list[list[float]] = [[3.0, 1.0], [2.0, 2.0], [0.0, 3.0]]16
17scores: list[float] = [round(cosine(query, m), 4) for m in memories]18print(scores)  # -> [1.0, 0.8944, 0.3162]19print(scores.index(max(scores)))  # -> 020
21store_total: int = 12 * 1522paste_cost: int = 6 * 50023request_paste: int = 400 + 800 + paste_cost24request_memory: int = 400 + 800 + 1525print(store_total)  # -> 18026print(request_paste - 4000)  # -> 20027print(request_memory)  # -> 1215

Watch Out For

The Quick Version

• Short-term memory is the live context window. It costs nothing to read and it vanishes when the session ends.
• Long-term memory is facts saved on purpose in durable storage. Saved facts do nothing until retrieval copies them back into the window.
• Writes should record repeated preferences, not one off task details. A fact confirmed twice earns storage.
• Retrieval ranks stored facts against the current task and pulls back the top one or two, leaving the rest filed.
• Three facts scoring 1.0, 0.8944, and 0.3162 retrieve the 1.0 match, turning a 3000-token paste into a 15-token pull.
• Twelve 15-token facts cost 180 tokens total, about 4.5 percent of a 4000-token window.

What to Read Next

• Episodic and Semantic Memory
• Context Engineering
• Embeddings
• AI Agent Architecture
• Memory-Augmented Architectures