Memory and State for Agents
Open in the interactive app: practice, quiz, mock interview →In one minute
- Models are stateless. Everything an agent "remembers" is something your code chose to store and put back into a later prompt. Memory is a data design problem, not a model feature.
- Separate the kinds: working memory (this task's context), session state (this conversation), long-term memory (facts and preferences that outlive sessions) and durable task state (progress of a long job).
- Long-term memory needs the same care as any user data: what to store, how to retrieve it, when to forget it, who can see it, and how users can view and delete it.
- Retrieval decides usefulness: score memories by relevance to the current task, importance and recency, and fit them into a token budget.
- Compaction keeps long sessions alive: summarise older history into a short record of decisions and facts, keep recent turns verbatim, and store details outside the context where the agent can look them up.
How it works
Four kinds of state
The split follows the usual agent taxonomy: short-term memory is in-context, bounded by the finite context window, while long-term memory is an external store queried at retrieval time (Weng).
- Working memory: the current context window: instructions, recent turns, tool results. Fast and expensive, limited by tokens.
- Session state: the conversation so far, stored by your app and replayed or summarised on each call.
- Long-term memory: facts that should persist across sessions: "prefers metric units", "team uses Postgres 16", "customer is on the enterprise plan". Stored in a database or files, retrieved when relevant.
- Task state: for long-running agents, the plan, completed steps, tool-call results and idempotency keys, persisted so the task can resume after a crash.
Writing memories
Decide what is worth remembering. Good candidates: stable preferences, decisions with their reasons, facts learned from tools that are expensive to recompute. Bad candidates: everything (noise drowns signal), transient details, and sensitive data you do not need. Many systems let the model propose memories through a tool ("remember: user prefers short answers"), then apply rules in code before storing: deduplicate, attach source and timestamp, classify sensitivity.
Reading memories
At the start of a turn, retrieve memories relevant to the current request. Common scoring combines:
- Relevance: keyword overlap or embedding similarity with the request.
- Importance: a score given when the memory was written ("allergy" outranks "liked the blue theme").
- Recency: newer memories usually matter more, and stale ones may be wrong.
Fit the top memories into a fixed token budget (Huyen's AI Engineering, Ch 6, treats memory as a first-class part of agents next to planning and tools), and show where each came from so the model can weigh it. MemGPT framed this like an operating system: a main context (read-only instructions, a small read/write working-context block and a FIFO queue of recent messages) plus external recall and archival storage that the model pages in and out through function calls. It warns the model at about 70% of the window so it can save what matters, and at the limit evicts a set count of messages (for example 50% of the context window) into a recursive summary while the full messages stay searchable in recall storage (Packer et al.). The Generative Agents paper gave the scoring recipe most memory systems copy: min-max normalised recency (exponential decay, factor 0.995 per game hour), importance (an integer 1 to 10 rated by the model at write time) and relevance (embedding cosine similarity), summed with equal weights, plus periodic reflections triggered when recent importance sums past 150 (Park et al.).
Compaction
Long sessions hit the context limit and degrade before that. Compaction replaces old history with a summary: decisions made, facts established, open questions, and pointers to stored details (files, notes). Keep the most recent turns verbatim. Some agent harnesses do this automatically when the context reaches a threshold; others ask the agent to maintain a notes file it can reread. Anthropic describes context as a finite resource whose recall degrades as it fills ("context rot"), and Claude Code's compaction keeps architectural decisions and unresolved bugs, drops redundant tool output, and continues with the summary plus the five most recently accessed files; clearing old tool results is the lightest form (Anthropic). For multi-session work, the pattern is an initializer session that creates a progress file, a feature checklist and a git history, so every later session starts by reading them and ends by updating them (Anthropic).
Forgetting and privacy
Memories go stale ("works at Acme" after a job change) and can be wrong. Expire low-importance memories after some time, let newer facts supersede older ones, and give users a way to view, correct and delete what is stored. Anthropic's memory tool documentation tells implementers to cap file sizes, expire files that have not been accessed in a long time, and strip sensitive data before writing (Claude docs). Memories are personal data: apply retention limits, access control (one user's memories must never appear in another's context) and deletion on request.
A concrete implementation: the memory tool
Anthropic's memory tool is client-side: Claude emits tool calls (view, create, str_replace, insert, delete, rename) against a /memories directory, and your application executes them against storage you control, such as a per-user directory or database keys. The API adds a system instruction to view the memory directory first and to assume the context may be reset at any moment. Because the model chooses the paths, your handler must canonicalise and reject anything outside /memories, including ../ and URL-encoded variants. The docs also pair it with server-side compaction: compaction shrinks the active context, and memory keeps what must survive summarisation (Claude docs).
The same shapes as any system
Memory is a cache with eviction (what to keep in the expensive context), an index (finding relevant items fast) and a database with retention rules. Task state is a durable job record; a workflow engine such as Temporal rebuilds it by replaying a recorded event history rather than restoring a snapshot (Temporal docs). Nothing about it is unique to AI except that the reader is a model.
Down the stack
Layer 1The contract
An agent's memory system promises: relevant facts from earlier sessions are available when needed, within a token budget; memories are scoped to their owner; users can see and delete them; and long tasks resume from their last completed step without repeating side effects.
Layer 2The mechanism
Long-term memories are rows with fields such as owner, text, embedding, importance, created and last-used timestamps, source and sensitivity. Retrieval runs a filtered similarity search by owner, then re-scores with importance and recency, then truncates to the budget. Supersession is handled by keys ("preference:units") so a new value replaces the old. Compaction is a model call that summarises a span of history into a structured note, stored and inserted in place of that span. Task state is stored per step, often in a workflow engine or a table keyed by task ID and step number.
Layer 3The machine
Every remembered token placed into the prompt is processed again on every call: a 2,000-token memory block in a 20-turn conversation is 40,000 input tokens of cost and prefill latency. That is why budgets and ranking matter more than storage size; storing a million memories in a database is cheap, putting a thousand of them into each prompt is not. Prompt caching helps only if the memory block stays identical across calls, so systems often fix the memory block at the start of a session rather than changing it every turn.
Why it's built this way
- Models forget everything between calls, so state lives in your systems.
- Context is expensive and attention is finite, so only relevant memories should enter it.
- Relevance alone misses importance and staleness, so scoring combines several signals.
- Memories are user data, so ownership, retention and deletion apply.
- Long tasks fail midway, so their state must be durable and idempotent.
Diagram
flowchart TD U["User request"] --> R["Retrieve memories: owner filter, relevance, importance, recency"] R --> B["Fit to token budget"] B --> P["Prompt: instructions, memories, recent turns"] P --> M["Model"] M --> W["Proposed memory writes"] W --> RULES["Rules: dedupe, supersede, sensitivity"] RULES --> DB["Memory store"] DB --> R H["Long history"] --> C["Compaction summary"] C --> P
Worked example
A personal coding assistant remembers project facts across sessions. Version one stores every message in a memory table and inserts the 30 most similar ones into each prompt. After two months, users report the assistant suggesting a library the team removed months ago, and prompts have grown to 9,000 memory tokens.
Version two changes three things. Writes go through rules: only facts and decisions, each with a key ("db:engine", "lint:tool"), importance from 1 to 10 and a source; a new fact with the same key replaces the old one. Reads score each memory as three times keyword or semantic overlap, plus importance, minus age in days for low-importance items, and take the top 8 within a 1,000-token budget. A weekly job forgets memories older than 90 days with importance below 4 that have not been used. Users get a "what do you remember about this project?" command that lists memories with delete buttons.
Results: memory tokens per prompt drop from 9,000 to under 1,000, the stale-library complaints stop because the newer "lint:tool" fact replaced the old one, and cost per session falls by about 40 percent.
In the wild
- Anthropic, 2025: long-running coding agents lose the thread across context windows. Without scaffolding, agents tried to build the whole app in one pass and ran out of context mid-feature, later sessions declared the project finished too early, and features were marked done after unit tests without end-to-end checks. The fix was an initializer session, a progress file, a JSON feature list (over 200 items in the example) and git commits as recovery points (Anthropic).
- Anthropic, 2025: memory-tool path traversal is a documented attack surface. The docs warn that a path such as
/memories/../../secrets.envcan escape the memory directory unless every command validates paths, and list canonical-path checks and rejection of encoded traversal as required safeguards (Claude docs). - Greshake et al., 2023: poisoned retrieved content persists as attack payload. Instructions planted in data an LLM app later retrieves can take over its behaviour, so anything an agent writes to long-term memory from untrusted tool output can re-inject itself in later sessions (arXiv).
Trade-offs
| Approach | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| Replay full history | Nothing lost | Grows without bound | Short conversations |
| Summary plus recent turns | Bounded size | Summaries can drop details | Long sessions |
| Retrieved long-term memories | Personalised, scalable | Retrieval errors, staleness | Repeat users and projects |
| Notes file the agent maintains | Transparent, editable | Agent must keep it current | Coding and research agents |
| Durable task state in a workflow engine | Resumable, auditable | Infrastructure | Long-running, side-effecting tasks |
Pitfalls in production
- Storing everything and retrieving by similarity alone. Fix: write rules, keys for supersession, importance and recency in scoring.
- Memories leaking across users or tenants. Fix: owner filters enforced in the store, tested.
- Stale facts overriding new ones. Fix: keyed memories, timestamps, expiry.
- No user visibility or deletion. Fix: list, edit and delete controls, and retention limits.
- Memory blocks that change every turn and defeat caching. Fix: stable memory block per session.
Leadership lens
- Treat memory as a product feature with a privacy review, not an implementation detail.
- Define memory types, write rules and retention per type in the design doc.
- Measure memory usefulness: how often retrieved memories are referenced, and complaints about stale or wrong recall.
- Build task state on durable infrastructure from the start; retrofitting resumability is painful.
Check yourself
Why is "the model remembers" a misleading way to think about agent memory?
Show answer
The model keeps nothing between calls; your code stores information and chooses what to put back into each prompt.
Why combine relevance with importance and recency when retrieving memories?
Show answer
Relevance alone can surface trivial or outdated items; importance and recency help prefer critical and current facts.
What should compaction preserve?
Show answer
Decisions, established facts, open questions and pointers to stored details, while recent turns stay verbatim.
Why do memories need deletion and retention rules?
Show answer
They are user data that can be wrong, stale or sensitive, so users and laws require the ability to correct and remove them.
Sources and further study
Primary
- MemGPT: Towards LLMs as Operating Systems (Packer et al.): tiered memory, memory-pressure warnings and recursive summaries.
- Generative Agents: Interactive Simulacra of Human Behavior (Park et al.): recency, importance and relevance retrieval and reflection.
Systems docs
- Memory tool (Claude docs): commands,
/memorieslayout and security considerations. - Compaction (Claude docs): server-side summarisation as it nears the window limit.
- Effective context engineering for AI agents (Anthropic): context rot, compaction, note-taking and sub-agents.
- Effective harnesses for long-running agents (Anthropic): progress files and git across sessions.
- Temporal workflows: durable task state via event history replay.
Books
- AI Engineering (Chip Huyen, O'Reilly 2025), Ch 6 "RAG and Agents" (Agents; Memory): short-term versus long-term memory and retrieval.
Other curricula
- LLM Powered Autonomous Agents (Lilian Weng): frames agent memory as short-term (in-context) and long-term (external retrieval) components.
Free resources
- MemGPT: Towards LLMs as Operating Systems: paging information between context and external storage.
- Memory tool (Anthropic docs): letting an agent store and retrieve information across conversations.
- Managing context on the Claude Developer Platform (Anthropic): context editing and memory for long-running agents.
- Effective context engineering for AI agents (Anthropic): compaction, notes and sub-agents.
- Temporal workflows: durable state for long-running tasks.