Chapter 3: Agent Memory Systems — How AI Remembers Across Time

Introduction: The Statefulness Problem

There is a fundamental asymmetry at the heart of how large language models work. A model’s weights encode billions of facts about the world — history, science, code, language — learned during training from enormous datasets. But within any single conversation, that model begins fresh. It has no memory of the conversation you had with it yesterday. It does not know that you corrected it three turns ago. It cannot track that the user it is talking to is a senior engineer who prefers terse answers, not a newcomer who needs every step explained.

This is not a limitation that larger models or longer context windows fully resolve. A model with a million-token context window can, in principle, hold an enormous amount of prior interaction in its active memory — but only for the duration of a single session. Close the browser tab, and the context is gone. More subtly, even within a session, models struggle to reliably use long context: experiments with models at 12K and 16K context windows have shown that performance on temporal reasoning tasks can actually decline as context length grows, because the model’s ability to locate and reason about relevant information does not scale with its storage capacity.

For agentic AI systems — systems that operate across long time horizons, multiple sessions, and potentially many parallel threads — this statefulness problem is existential. An agent that cannot remember what it did yesterday is not truly autonomous. It is a sophisticated stateless function: impressive at individual tasks, but incapable of the kind of persistent, evolving engagement that makes an agent genuinely useful over time.

This chapter is about how researchers and engineers have approached that problem. We will cover the taxonomy of memory types in agentic systems, the architectural patterns that have emerged to implement them, the benchmarks that measure how well they work, the performance numbers that reveal how large the human-AI gap remains, and the practical approaches used in production systems today. By the end, you will have both a theoretical framework for thinking about agent memory and a concrete set of tools for implementing it.

1. A Taxonomy of Agent Memory

1.1 The Fundamental Split: Textual vs. Parametric

The clearest high-level taxonomy for agent memory comes from a 2024 survey on the memory mechanisms of LLM-based agents (arXiv:2404.13501). It divides memory into two primary forms, each with distinct tradeoffs:

Textual memory is anything stored as text that can be placed into the model’s context window — complete interaction logs, recent conversation turns, retrieved passages from an external store, or structured knowledge injected at query time. Textual memory is interpretable: you can read it, edit it, and verify that it says what you think it says. It is also updatable: facts that change can be revised without retraining the model. The cost is context consumption. Every token you devote to memory is a token unavailable for the current task.

Parametric memory is knowledge encoded directly in model weights during training or fine-tuning. A model that has been fine-tuned on a company’s internal documentation “knows” that documentation in a parametric sense — the knowledge is distributed across its parameters, not stored as text. Parametric memory consumes no context tokens, but it has a critical limitation: it cannot be easily inspected, edited, or updated. Correcting a parametric memory error requires retraining.

This distinction is not merely academic. It drives a core architectural decision every agent system builder faces: which facts should live in the model’s weights, which should live in the context window, and which should live in an external store that the agent retrieves from as needed? There is no universal right answer, but the tradeoffs are clear enough to reason about systematically.

1.2 The Cognitive Architecture View

A complementary taxonomy comes from CoALA (“Cognitive Architectures for Language Agents,” arXiv:2309.02427), a 2023 paper from Princeton and MIT that drew explicit parallels between cognitive science’s model of human memory and the components an AI agent needs. CoALA distinguishes several memory types:

• Working memory — the information currently active in the agent’s context window. Analogous to human working memory, it is fast and flexible but strictly capacity-limited.
• Episodic memory — records of specific past experiences: what the agent did, when, with what outcome. This is the memory type most directly relevant to learning from experience and avoiding repeated mistakes.
• Semantic memory — general world knowledge and factual associations, corresponding roughly to what the model learned during pretraining, plus any additional factual stores it retrieves from.
• Procedural memory — knowledge of how to do things, encoded either implicitly in the model’s weights or explicitly as retrievable code, tool descriptions, or reusable skill libraries.

CoALA was notable not just for its taxonomy but for its diagnosis. Writing in September 2023, the authors identified adaptive, context-specific recall as the critical under-studied gap in language agents — the ability to retrieve exactly the memories most relevant to the current situation, rather than retrieving broadly or relying on recency alone. The research wave that followed — Mem0, Graphiti, hierarchical memory architectures — represents the field’s direct response to this diagnosis.

1.3 Why Multiple Taxonomies Exist

It is worth pausing to note that no single taxonomy has become universal. Different papers frame the landscape differently, and all of the frameworks above are valid, non-canonical perspectives on the same underlying design space. What matters for practice is not adopting the correct taxonomy but using some taxonomy as a design tool: thinking explicitly about what your agent needs to remember, for how long, with what precision, and whether retrieval or parametric encoding is the right vehicle for each type.

The two taxonomies map onto each other more cleanly than they first appear — the diagram below places them side by side:

2. The Context Window Problem

2.1 A Constraint That Scales Poorly

Every agent memory architecture begins with the same constraint: the context window. Whatever memory mechanism you choose, the information that a model can act on at any given moment is bounded by how much text fits in its active context. In 2023, state-of-the-art models had context windows in the 8K–32K token range. By 2025–2026, frontier models offered 128K to 1M tokens. This seems like it should make the memory problem tractable.

It does not, for two reasons.

First, filling a large context window is expensive. Inference cost scales roughly linearly with context length. An agent that stuffs its entire conversation history into every prompt is paying for context it may not need. At production scale, this cost adds up quickly.

Second — and more importantly — models do not reliably exploit large contexts. This is one of the most well-documented and counterintuitive findings in recent agentic research. A 2024 benchmark study (LoCoMo, arXiv:2402.17753) showed that GPT-3.5-turbo-16K’s score on temporal reasoning tasks actually declined when given 16K context compared to 12K context — 20.3% F1 at 16K versus 25.0% at 12K. More context did not produce better recall; it produced noise that overwhelmed the signal.

This “Lost in the Middle” effect — where models struggle to use information positioned in the middle of long contexts, even when that information is directly relevant — has been independently documented across multiple studies. The implication is uncomfortable: you cannot simply expand the context window and call the memory problem solved. You need architectures that actively manage what goes into that window.

2.2 MemGPT: The Operating System Analogy

The clearest early response to this constraint was MemGPT, introduced in October 2023 (arXiv:2310.08560) and later rebranded as the Letta platform. MemGPT drew an explicit analogy to operating system memory management: just as an OS manages a small pool of fast RAM and a large pool of slow disk storage, MemGPT manages a small active context window (fast, directly accessible) and a large external storage tier (slower, requiring explicit retrieval).

In MemGPT’s model, the agent has explicit control over what moves between tiers. Information that is immediately relevant stays in-context. Information that may be relevant later gets written to external storage and retrieved when needed. The model itself is aware of this architecture — it can issue “memory management” function calls to page information in and out, similar to how a program issues system calls for memory allocation.

This architecture made MemGPT particularly effective for tasks involving very long documents or extended multi-session interactions, where the amount of relevant information vastly exceeded what could fit in any context window. It also introduced a new kind of failure mode: if the agent’s memory management decisions are wrong — if it pages out information that turns out to be critical, or fails to retrieve information it needed — the entire task can fail in ways that are hard to diagnose.

3. Production Memory Architectures

3.1 Graphiti and Zep: Temporal Knowledge Graphs

One of the most architecturally interesting approaches to agent memory to emerge from the 2024–2026 period is Graphiti, the open-source engine underlying the Zep platform (arXiv:2501.13956). Graphiti takes a fundamentally different approach from flat vector stores: rather than storing memories as text blobs indexed by embedding similarity, it constructs a temporal knowledge graph where facts are stored as typed relationships between entities, each with validity windows.

The key innovation is what Graphiti calls bi-temporal fact tracking. When information changes — when a user’s job title changes, when a project’s status updates, when a preference shifts — the old fact is not deleted. It is invalidated, with its valid-until timestamp set to the moment the new fact was established. This means the system can answer not just “what is true now?” but “what was true at any point in the past?” — a capability that proves surprisingly important for agents with long operational histories.

For retrieval, Graphiti uses a hybrid approach that combines three distinct signals:

1. Semantic embeddings — cosine similarity between query and stored fact representations, capturing conceptual relevance.
2. Keyword search (BM25) — lexical matching that catches specific names, identifiers, and technical terms that embedding similarity can miss.
3. Graph traversal (BFS) — following entity relationships to surface facts that are not directly similar to the query but are connected through shared entities.

These three signals are combined using Reciprocal Rank Fusion (RRF), which merges ranked lists without requiring any of the signals to be calibrated to a common scale. Critically, Graphiti performs no LLM summarization at query time — the retrieval pipeline is deterministic and fast, with the LLM only involved at write time (when new information is being integrated into the graph) and at generation time (when the retrieved facts are used to inform a response).

3.2 Mem0: Flat Memory with Hybrid Retrieval

Mem0 (github.com/mem0ai/mem0) takes a different architectural philosophy. Where Graphiti models memory as a graph of typed entities and their relationships, Mem0 stores each memory as a self-contained record — a short natural-language fact (“prefers window seats,” “is allergic to shellfish”) paired with its vector embedding and metadata — in a flat store rather than a linked structure. There is no graph to traverse; retrieval is primarily nearest-neighbor search over the embeddings, optionally combined with keyword and metadata filtering, which keeps both writes and reads simple and fast.

What “multi-level” refers to is not the storage layout but the scoping of those records. Every memory is tagged with the level it belongs to, so the same store can serve three different lifetimes at once:

• User scope — durable facts about a specific end user that should persist across every conversation they ever have (dietary restrictions, preferred name, recurring goals).
• Session scope — context relevant only to the current conversation or task, discarded or aged out when the session ends.
• Agent scope — knowledge that belongs to the agent itself rather than any one user, such as learned procedures or organization-wide facts shared across all the users it serves.

At retrieval time, Mem0 filters by the relevant scope before running the similarity search, so a query about one user never surfaces another user’s memories, while agent-level knowledge remains available to everyone. This scoping is what lets a single deployment personalize per user without either leaking data between users or re-learning shared facts for each one. When a new memory arrives, Mem0 also runs an LLM-driven step that decides whether to add it, update an existing record, or discard it as redundant — the consolidation logic that keeps a flat store from simply accumulating near-duplicate facts forever.

The April 2026 algorithm update produced benchmark-leading results on the two most demanding public evaluation suites, with self-reported performance gains that are striking in magnitude:

Retrieval latency at p50 is approximately 1.09s for LongMemEval and 0.88s for LoCoMo, using roughly 6.8K–7.0K tokens per call. The benchmark code is open-sourced at github.com/mem0ai/memory-benchmarks, allowing independent reproduction — though as of writing, independent peer-reviewed replication does not yet exist, and slight numerical discrepancies between Mem0’s GitHub README and research page suggest some measurement variance between infrastructure configurations.

The tradeoff between Mem0’s flat-store approach and Graphiti’s graph approach is not yet empirically settled. Flat stores are simpler to implement and reason about; knowledge graphs preserve relational structure that may matter for complex multi-hop reasoning. No controlled head-to-head comparison on a shared benchmark exists in the confirmed literature.

4. How the Major Platforms Handle Memory

The open-source engines above are what you reach for when you want to build a memory layer yourself. But most teams building agents in 2025–2026 are building on a framework or a cloud platform that already ships some memory abstraction. It is worth understanding what those platforms give you out of the box — because the abstractions they expose reveal how the industry as a whole has come to think about the problem.

The striking finding, when you survey the six most widely used stacks — LangChain/LangGraph, AWS, Microsoft Azure, Google, OpenAI, and Anthropic — is how much they agree. Despite wildly different APIs, they have all converged on the same two-tier model this chapter has been building toward: a fast, automatically-managed short-term tier for the current session, and a persistent long-term tier that survives across sessions. And for that long-term tier, four of the six have independently arrived at the same architecture — the LLM-driven extract → consolidate → retrieve pipeline that §3’s research called the field’s answer to CoALA’s “adaptive recall” gap.

4.1 The Short-Term Tier: Automatic by Default

Every platform makes short-term memory nearly free. The vocabulary differs — LangGraph calls it a checkpointer keyed by thread_id; OpenAI and Azure call it a thread or conversation; Google’s ADK calls it a session — but the mechanism is the same: the platform automatically persists the running conversation and replays it on the next turn, so the developer does not hand-manage message history. (OpenAI’s own Agents SDK confusingly reuses the term session for the same idea.)

Where they differ is in how they cope with the context-window problem from §2. Azure’s threads auto-truncate to fit the model’s window. OpenAI’s Responses API offers server-side compaction past a configurable token threshold. Anthropic offers two distinct server-side mechanisms — context editing, which clears the oldest tool results once context crosses a trigger (default 100,000 input tokens, keeping the 3 most recent tool calls), and compaction, which summarizes older content into a concise block (default trigger 150,000 input tokens). All of these are the production-grade descendants of the MemGPT paging idea: keep the active window small, move the rest somewhere cheaper.

4.2 The Long-Term Tier: The Extract–Consolidate–Retrieve Consensus

For memory that outlives a session, four platforms have converged on a nearly identical managed pipeline:

• AWS Bedrock AgentCore Memory (GA since ~October 2025) runs an asynchronous background process that extracts insights across sessions using four built-in strategies — User Preferences, Semantic, Session Summaries, and Episodic — notably one of the only products to use the cognitive-science vocabulary from §1 directly.
• Microsoft Azure Foundry Agent Service Memory (preview) runs an explicit three-phase pipeline the docs literally label Extraction → Consolidation → Retrieval, with an LLM merging duplicates and resolving conflicts (a new allergy overrides an old fact) into per-user memory stores.
• Google Vertex AI Agent Engine Memory Bank (public preview since July 2025) uses Gemini to asynchronously extract facts and preferences from session history, persist them by scope (e.g., user ID), and consolidate them while resolving contradictions.
• LangChain’s LangMem SDK (launched February 2025) provides a background memory manager that “extracts, consolidates, and updates agent knowledge asynchronously,” built on LangGraph’s long-term Store.

The convergence is not a coincidence. It is the direct engineering expression of the research consensus from earlier in this chapter: raw storage is not the hard part — deciding what is worth remembering, keeping it current, and retrieving the right piece at the right time is. Every one of these pipelines is an attempt to automate that judgment.

Anthropic takes a deliberately different path. Rather than a managed service, Claude’s memory tool exposes memory as a tool the model drives: Claude issues file commands (create, view, str_replace, insert, delete, rename) against a /memories directory, and the developer’s own application executes them against storage the developer controls. There is no server-side extraction pipeline — Claude itself decides what to write and when to read it, and the persistence guarantees are the developer’s responsibility. Maximum control, maximum flexibility, but you own the backend.

OpenAI occupies a middle ground. Its durable Conversation objects (created via the Conversations API) persist across sessions, devices, and jobs without the 30-day expiry that standalone stored responses have — thread-like persistence, but of raw conversation items rather than distilled long-term facts. Distillation, if you want it, is left to you or to the ChatGPT consumer product’s separate memory feature.

4.3 A Comparison

Two practical lessons fall out of this table. First, the short-term tier is a solved, commoditized problem — pick any platform and you get automatic conversation persistence with some window-management strategy. Second, the long-term tier is still stabilizing: as of this writing, only AWS AgentCore Memory and Anthropic’s memory tool are generally available; Azure, Google, and LangMem’s offerings are preview or beta with APIs the vendors explicitly warn may change. If you are building on managed long-term memory today, budget for churn.

5. Benchmarks and the Human-AI Gap

5.1 LoCoMo: The Hardest Conversational Memory Benchmark

The most demanding public benchmark for agent memory is LoCoMo (“Long Conversational Memory,” arXiv:2402.17753, February 2024). Its design was motivated by the inadequacy of prior conversational benchmarks, which typically covered only a handful of sessions and a few hundred tokens per conversation. LoCoMo’s 50 conversations average 304.9 turns and 9,209 tokens each, spanning up to 35 sessions — an order of magnitude more demanding than the prior state of the art.

The human-AI performance gap on LoCoMo is the clearest single data point for how far agent memory systems have to go:

The ~50-point overall gap is striking enough. The temporal reasoning gap — where the best-evaluated model scores 20.3% against human performance of 92.6% — is more alarming. Agents do not just struggle to retrieve facts from long histories; they struggle to place those facts in the correct temporal order, which is often essential for understanding what is currently true versus what was once true.

The context window non-result discussed earlier comes from this benchmark: GPT-3.5-turbo-16K’s temporal reasoning score declined from 25.0% at 12K context to 20.3% at 16K. The authors describe this directly: “long-context models may not be proficient at utilizing their context appropriately.” More storage is not the solution.

5.2 LongMemEval: A Five-Ability Framework

Where LoCoMo measures overall performance on long-conversation QA, LongMemEval (arXiv:2410.10813, October 2024) was designed to be diagnostic — to break memory performance down into the distinct cognitive abilities that an agent’s memory system must support.

LongMemEval defines five abilities, evaluated across 500 curated questions embedded in freely scalable conversation histories:

1. Information extraction — Can the agent locate and recall a specific fact stated earlier? This is the simplest memory operation: straightforward retrieval of a stored value.
2. Multi-session reasoning — Can the agent synthesize information mentioned across multiple separate sessions? This requires not just retrieval but integration of facts from different temporal contexts.
3. Temporal reasoning — Can the agent correctly understand when facts were established and whether they are still current? This is the hardest ability, and consistently the biggest source of AI underperformance.
4. Knowledge updates — Can the agent correctly handle cases where a fact changes over time — where the answer to “what is Alice’s job title?” is different in session 5 than in session 1?
5. Abstention — Can the agent correctly decline to answer when the required information is not present in its memory? This tests a different failure mode from incorrect recall: confabulation of an answer the agent should know it does not have.

The five-ability framework is valuable precisely because it disaggregates performance. A system that scores well on information extraction but poorly on knowledge updates has a different problem than one that scores well on knowledge updates but poorly on abstention. The diagnostic granularity makes LongMemEval more actionable than a single aggregate score — which is why it has become the standard evaluation suite for production memory systems like Mem0.

A follow-up, LongMemEval-V2 (arXiv:2605.12493, 451 questions), extends the original; results are broadly consistent with V1.

6. Open Problems and Honest Gaps

6.1 What the Research Does Not Yet Settle

The confirmed findings in this chapter tell a clear story about what works and what is hard. But several important questions remain unresolved in the primary literature.

Updated frontier model performance. The LoCoMo and LongMemEval baselines were established with 2023–2024-era models. Frontier models in mid-2026 — with substantially improved instruction following and, in some cases, million-token context windows — have not been systematically evaluated on these benchmarks in a comparable controlled study. The ~50-point human-AI gap documented for GPT-3.5-turbo may have narrowed; how much is unknown.

Memory security. Memory poisoning — the injection of false or malicious memories into an agent’s long-term store — and hallucinated memory — where an agent confabulates recall of events that never occurred — are both documented as concerns in the literature. Neither has a confirmed quantitative benchmark or validated mitigation in the primary research as of this writing. An agent that can be made to “remember” things that never happened, or that spontaneously generates false memories, has a fundamental trust problem that benchmark performance scores do not capture.

Cross-architecture comparison. Each memory framework in this chapter — Mem0, Graphiti/Zep, MemGPT — reports its own performance against its own benchmark configurations. No controlled head-to-head study on a common evaluation suite exists in the confirmed literature. This makes direct comparison impossible, which is a significant gap for practitioners trying to make architecture decisions.

Forgetting versus retention. Graphiti preserves invalidated facts permanently via its bi-temporal model; other systems consolidate, compress, or discard older memories. The right tradeoff depends on the use case, but no empirical study comparing these strategies on downstream task performance has been confirmed. This is an open design question.

6.2 The Temporal Reasoning Gap as the Central Challenge

If there is a single finding from this chapter that should inform how you think about agent memory systems, it is the temporal reasoning gap. Agents do not just fail to remember facts from long interaction histories — they fail specifically at placing those facts in the correct temporal context. They struggle to answer questions like “is this still true?” and “when did this change?” These are questions that humans answer almost reflexively, by maintaining implicit timelines of when facts were established and when they were superseded.

This gap is why the bi-temporal architectures — systems that explicitly track when facts were valid, not just what they are — represent the most conceptually principled approach to agent memory. If the model itself cannot reliably reason about fact temporality from raw stored text, the memory architecture needs to encode temporality explicitly and present it in a form the model can use.

Whether that approach translates to production-scale performance improvements is one of the most important open empirical questions in the agentic AI field today.

Summary

Memory in agentic AI systems divides into two fundamental forms — textual (in-context or retrieved) and parametric (encoded in weights) — with different tradeoffs along interpretability, updatability, and context cost. Production architectures have converged on hybrid retrieval systems that combine semantic search, keyword search, and structured traversal, because no single retrieval signal is sufficient.

Benchmarks reveal a substantial human-AI gap: on LoCoMo, the best-evaluated LLM scores 37.8% F1 overall and 20.3% on temporal reasoning against human performance of 87.9% and 92.6% respectively. Critically, larger context windows do not resolve this gap — in some configurations, they make it worse.

The most active architectural response to these limitations is explicit temporal fact management: systems that track not just what is true but when it became true and when it stopped being true. CoALA’s 2023 diagnosis — that adaptive, context-specific recall was the critical under-studied problem — has driven the primary research direction since, producing frameworks that integrate retrieval with structured representation rather than treating them as separate concerns.

The next chapter turns to Retrieval-Augmented Generation: how agents retrieve and ground knowledge from large document corpora, the evaluation frameworks for measuring retrieval quality, and the robustness gaps that standard metrics systematically miss.