Gen AI

Memory Systems

Short-term memory is just what's sitting in the context window right now; long-term memory is saved somewhere durable so it survives after the session ends.

Short-term memory lives in the context window and empties when the session ends, while long-term memory sits in durable storage and only matters once something retrieves it back in

Why Does This Exist?

Say you're using a coding assistant across weeks, not one sitting. In session one, you mention you want type hints on everything and you test with pytest, not unittest. By session eleven, a good assistant writes type-hinted functions and pytest fixtures without being told again. That only happens if something from session one survived past the moment its context window closed.

Nothing about a language model guarantees that survival. A model call is stateless — it sees exactly the tokens in its context window and nothing else. Once that window is gone, whatever wasn't written down elsewhere is gone too. Context engineering manages what fits in the window right now; memory systems decide what stays true next time.

That's the problem memory systems solve: giving an agent something that outlives a single window, without cramming every past conversation into every new one. Paste the eleven-session transcript into session twelve's prompt and you've recreated the budget problem context engineering exists to manage — most of those sessions have nothing to do with today's task anyway.

Think of It Like This

A whiteboard versus a filing cabinet

The context window is a whiteboard. Anything on it is right there, free to read — and the second someone wipes it for the next meeting, it's gone.

Long-term memory is the filing cabinet down the hall. Putting something in it takes an actual decision — you don't file every napkin sketch, only what's worth keeping. A filed note does nothing for today's meeting unless somebody walks down the hall, pulls the folder, and brings it back to the whiteboard. A cabinet nobody opens is indistinguishable from no cabinet at all.

How It Actually Works

Short-term vs long-term

Short-term memory isn't a separate system — it's just a name for whatever's sitting in the context window: the system prompt, the conversation so far, the last few tool results. It's free in the sense that it's already there, and temporary in the same sense: once the session ends, it disappears with the window. There's no separate act of "saving" short-term memory, because it was never anywhere durable to begin with.

Long-term memory is the opposite on both counts. Someone — a developer, or the agent itself — writes it somewhere outside the window on purpose: a file, a database row, a vector store entry. That write is what makes it survive; a fact in long-term storage is still there next week, in a session on a different machine entirely. But durability is only half the story. Being saved somewhere doesn't put a fact back in front of the model — it has to be explicitly pulled back into the window later, or it might as well not exist for that conversation. Saved and used are two different steps, and conflating them is where "the agent has memory now" quietly fails.

The write decision

Not everything that happens in a session deserves a permanent write. Ask the coding assistant to rename one variable in one file, and that's a fact about this task, useful for thirty seconds, worth nothing in session twelve. Writing it to long-term storage anyway just means more junk to search through later, for zero benefit.

A standing preference is different. The first time you mention you prefer type hints, that's one data point — maybe you meant it for this file only. The second or third time you say the same thing, or the assistant notices you consistently adding hints to code it wrote without them, that's no longer a one-off. It's a pattern stable enough to bet on across sessions. That's the moment worth a write: not "user said X" logged from every message, but a distilled fact — user prefers pytest and type hints — that holds whether the current task is a script or a one-line fix. The write decision is really a durability judgment: true today, or still true in a month?

Retrieval and forgetting

A write with no retrieval path is a fact rotting in a database nobody queries. Something has to bring the right fact back at the right moment — usually by comparing the current task against stored memories with a similarity search over embeddings, so "write a function to parse a CSV" pulls back "prefers type hints and pytest" instead of an unrelated note about a deployment script from weeks ago.

The harder discipline runs the other way: not retrieving everything. It's tempting to think more memory is strictly better, so why not surface every past preference and fact ever written, every session, going forward. That instinct rebuilds the exact problem long-term memory was invented to dodge — a window stuffed with irrelevant history, competing for space with what today's task needs. Forgetting here rarely means deleting anything. It means retrieval staying selective: pulling back the two or three memories that are actually relevant and leaving the rest filed away, unread, until something makes them relevant again.

Show Me the Code

A tiny heuristic for the write decision: a fact earns a permanent write once it's been stated, or confirmed by behavior, more than once.

1from collections import Counter2
3
4def facts_worth_writing(5    observations: list[str], min_confirmations: int = 26) -> list[str]:7    """Only facts seen at least `min_confirmations` times are stable8    enough to write to long-term storage -- a one-off isn't a preference."""9    counts = Counter(observations)10    return [fact for fact, seen in counts.items() if seen >= min_confirmations]11
12
13session_notes = [14    "prefers pytest",15    "renamed variable in utils.py",   # one-off, task-specific16    "prefers type hints",17    "prefers pytest",                 # confirmed a second time18    "prefers type hints",             # confirmed a second time19]20
21print(facts_worth_writing(session_notes))22# -> ['prefers pytest', 'prefers type hints']

Watch Out For

Writing every conversational detail to long-term storage

Logging each message verbatim "just in case" fills long-term storage with one-off details indistinguishable from real preferences. Retrieval later can't tell a stable fact from noise, so a similarity search surfaces whatever matches the wording, not what matters. Write what's confirmed as durable, not everything that was said.

Retrieving too much long-term memory back into every session

Pulling every stored fact into context "to be safe" defeats the point of having long-term memory at all. If session twelve's prompt includes all forty facts filed since session one, you're paying the full context budget context engineering exists to manage — just with stale preferences instead of a raw transcript.

The Quick Version

Short-term memory is just whatever's in the current context window; it's free, and it vanishes when the session ends.
Long-term memory is written somewhere durable on purpose, so it survives across sessions — but it does nothing until it's retrieved back into context.
Write to long-term storage when a fact is confirmed as stable (a repeated preference), not for every one-off detail from a single conversation.
Retrieval, usually similarity-based, decides what comes back into context and when.
Not retrieving everything is deliberate, not an oversight — dumping all stored memory back in every session recreates the budget problem long-term memory was supposed to fix.