Memory Systems in Agents

Agent memory systems extend the limited context window with persistent storage — enabling agents to remember past interactions, accumulate knowledge, and maintain state across long-running tasks that span multiple sessions.

Why Does This Exist?

An LLM's context window is finite — typically 8K to 200K tokens. For a task that spans hours or days (writing a research report, managing a project, serving a repeat customer), the context window fills up and older information gets dropped. Memory systems solve this by selectively persisting important information outside the context window and retrieving it when needed.

Without memory, every agent interaction starts from zero. With memory, agents can accumulate knowledge, learn user preferences, reference past decisions, and maintain coherence across long tasks.

Think of It Like This

A researcher's system of notes, notebooks, and habits

A researcher has several types of memory: their immediate working memory (what they're thinking about right now), their lab notebook (a written record of past experiments), their accumulated domain knowledge (years of reading papers), and their practiced experimental techniques (procedural skill). Agent memory systems map directly to these: working memory = context window; episodic = conversation history; semantic = knowledge base; procedural = learned strategies.

How It Actually Works

The four memory types

1. Working memory (in-context) The current context window — the agent's immediate awareness. Everything in the context window is "active" but limited to the window size. Efficiently managing what goes into working memory (via summarization and selection) is the core challenge.

2. Episodic memory A log of past interactions, observations, and decisions. Stored externally (database, file system) and retrieved by recency or semantic similarity. Enables the agent to say "When I encountered this situation last week, here's what I tried and what happened."

3. Semantic memory A knowledge base of facts, documents, and domain knowledge. Typically implemented as a vector store with embedding-based retrieval (RAG). The agent queries this when it needs information it doesn't have in-context.

4. Procedural memory Learned action patterns — successful strategies and workflows the agent has used before. Stored as prompt templates, few-shot examples, or even fine-tuned model weights.

Memory retrieval strategies

Strategy	When to use	Mechanism
Recency	Need recent history	Sliding window over time-ordered events
Semantic similarity	Need relevant past experiences	Embedding cosine similarity via vector DB
Exact lookup	Need specific fact	Key-value store with deterministic keys
Hybrid	Complex queries	Combine semantic search + recency filter

Memory management: what to store

Not everything is worth storing. Selection criteria:

Surprise/novelty: New information that contradicts or extends existing knowledge
Goal relevance: Information directly relevant to the current or future task
User feedback: Explicit corrections and preferences
Failure cases: What went wrong and why (the most valuable for improvement)

Code

1from langchain_openai import OpenAIEmbeddings, ChatOpenAI2from langchain_community.vectorstores import FAISS3from langchain.memory import ConversationSummaryBufferMemory4from langchain.schema import Document5import json, time6
7# ── 1. Working memory: conversation summary buffer ────────────────────────────8llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)9
10# Keeps recent messages in full; summarizes older messages to save tokens11working_memory = ConversationSummaryBufferMemory(12    llm=llm,13    max_token_limit=1000,14    memory_key="chat_history",15    return_messages=True,16)17
18# Simulate adding conversation turns19working_memory.save_context(20    {"input": "What is the capital of France?"},21    {"output": "Paris is the capital of France."}22)23working_memory.save_context(24    {"input": "What is its population?"},25    {"output": "Paris has approximately 2.1 million people in the city proper."}26)27
28print("Working memory contents:")29print(working_memory.load_memory_variables({})["chat_history"])30
31# ── 2. Semantic memory: vector store (RAG) ────────────────────────────────────32embeddings = OpenAIEmbeddings()33
34# Build knowledge base from documents35knowledge_docs = [36    Document(page_content="Python was created by Guido van Rossum in 1991.", metadata={"source": "history"}),37    Document(page_content="The Eiffel Tower is 330 meters tall.", metadata={"source": "facts"}),38    Document(page_content="FAISS is a library for efficient similarity search.", metadata={"source": "tech"}),39]40
41vectorstore = FAISS.from_documents(knowledge_docs, embeddings)42
43def retrieve_memories(query: str, k: int = 2) -> list[str]:44    """Retrieve semantically similar memories."""45    docs = vectorstore.similarity_search(query, k=k)46    return [doc.page_content for doc in docs]47
48memories = retrieve_memories("Who invented Python?")49print(f"\nRetrieved memories for 'Who invented Python?':")50for m in memories:51    print(f"  - {m}")52
53# ── 3. Episodic memory: persistent event log ──────────────────────────────────54class EpisodicMemory:55    """Simple episodic memory: stores events with timestamps."""56    def __init__(self):57        self.episodes = []58    59    def record(self, event_type: str, content: dict):60        self.episodes.append({61            "timestamp": time.time(),62            "type": event_type,63            "content": content,64        })65    66    def recall_recent(self, n: int = 5) -> list:67        return self.episodes[-n:]68    69    def recall_by_type(self, event_type: str) -> list:70        return [e for e in self.episodes if e["type"] == event_type]71
72episodic = EpisodicMemory()73episodic.record("user_feedback", {"rating": 4, "comment": "Good answer but too verbose"})74episodic.record("tool_call", {"tool": "search", "query": "FAISS", "success": True})75episodic.record("task_completed", {"task": "research report", "duration_min": 15})76
77print(f"\nRecent episodes:")78for episode in episodic.recall_recent(3):79    print(f"  {episode['type']}: {episode['content']}")

Watch Out For

Retrieval hallucination from irrelevant memories

Semantic similarity retrieval can surface memories that are superficially similar but contextually irrelevant — contaminating the agent's context with misleading information. Always include the retrieved memory's source and timestamp so the agent can reason about relevance. Filter retrieved memories through a relevance check before injecting them into the context.

Memory bloat and retrieval latency

Episodic memory that records every token exchanged grows rapidly. After thousands of interactions, retrieval becomes slow and results degrade as the signal is diluted by noise. Implement memory pruning: score old memories by a combination of recency, access frequency, and relevance to current goals, and archive or delete low-scoring entries.

The Quick Version

Agent memory extends the finite context window with persistent external storage.
Four types: working memory (context window), episodic (interaction log), semantic (knowledge base via RAG), procedural (learned strategies).
Retrieval strategies: recency (sliding window), semantic similarity (vector DB), exact lookup (key-value), hybrid.
Store selectively: prioritize novel, goal-relevant, and failure-case information.
Manage memory size with summarization and pruning — raw episode logs don't scale.