AI Agent Memory Explained: Short-Term, Long-Term, and How It Actually Works

Short-term memory: the context window as working memory

Short-term memory is the information an agent holds while it works on a single task. Think of it as working memory in the cognitive sense: a small, fast scratchpad that keeps the immediate job coherent. For an LLM agent, that scratchpad is the context window, the fixed budget of tokens the model can read in one pass.

Inside that budget sits everything the model needs for the current turn. The system prompt and the agent's instructions. The running conversation. The tool calls it has made and the data those tools returned. A few notes the agent jotted to itself mid-task, like a plan or a partial result. When the model decides its next move, it is reading all of this at once. There is no separate place it can look.

The catch is that the budget is finite, and modern context windows being large does not make the problem go away. A long support thread plus a 40-row query result plus a knowledge-base article will fill space fast, and stuffing more in is not free. Larger prompts cost more, run slower, and bury the important tokens in the middle where models attend to them least. So short-term memory is really an exercise in curation: keep what the current task needs, drop what it does not, and do it on every turn.

When a task runs long enough that the window can no longer hold the full history, the agent has to compress, which is where summarization comes in. We will get to that. The key idea here is that short-term memory is scoped to one job. Close the session, and unless something deliberately writes it down, it is gone.

Long-term memory: facts that survive across sessions

Long-term memory is what lets an agent remember across conversations, days, and tasks. It lives outside the model in a persistent store, and it is the difference between an agent that greets a returning customer by name and one that asks for their account number for the fourth time this month.

Practitioners usually break long-term memory into a few kinds, borrowing loosely from how psychologists describe human memory. The categories are not rigid, but they are useful for deciding what to store and how.

What unites these types is that none of them sit in the model by default. They are written to a store during or after a task, and retrieved later when something signals they are relevant. The retrieval step is where most of the engineering effort goes, because a memory you cannot find at the right moment may as well not exist.

• Semantic memory holds facts and stable knowledge: the customer is on the Enterprise plan, the company's fiscal year ends in March, this teammate wants reports in their local timezone. It changes slowly and is queried often, so it is worth storing cleanly.
• Episodic memory holds specific past events: this user asked about refund policy on May 3rd, that deal stalled after the security review. It is what lets an agent say "last time we tried this, it failed," which is often more valuable than any general fact.
• Procedural memory holds how-to knowledge the agent has learned: the exact steps to escalate a billing dispute, or the format your team expects for a weekly pipeline summary. It tends to be encoded as instructions or reusable skills rather than raw records.
• Profile or entity memory holds durable attributes about people and accounts: names, roles, preferences, relationships. It is small, high-value, and almost always worth keeping in fast, structured storage rather than burying it in a vector index.

How memory is stored and retrieved

Storing a memory means turning a piece of information into something you can find later. Two storage shapes dominate, and most serious systems use both. The first is structured storage: ordinary database rows and key-value records for facts you can name, like a user's preferred language or an account's tier. You retrieve these with exact lookups, fast and unambiguous. The second is vector storage, used when you want to find things by meaning rather than exact match.

Vector retrieval works by converting text into an embedding, a long list of numbers that captures its meaning, and storing that in a vector database. When the agent needs context, it embeds the current question the same way and asks the store for the records whose embeddings sit closest. The win is semantic search: a memory written as "customer wants invoices in euros" can surface for a question phrased as "what currency does this account bill in," even though the words barely overlap. This is the same retrieval idea behind RAG, applied to the agent's own history instead of a document corpus.

Pure vector similarity has a known weakness: it finds things that sound alike but can miss things connected by relationship. "Who reports to the VP this contact mentioned" is a relationship question, not a similarity question. That is why many 2026-era systems pair vector memory with graph-style memory that links entities and the relationships between them, then retrieve through those links. Neither approach is sufficient alone; the pairing covers more of what real questions actually ask.

Retrieval is rarely just one query. A good pipeline pulls a handful of candidate memories, ranks them for relevance, drops near-duplicates, and trims the set to what fits the budget before injecting it into the prompt. Pull too little and the agent acts uninformed. Pull too much and you are back to a bloated, expensive, confused prompt. The art is in the filtering.

Summarization and consolidation: compressing without forgetting

When a task outgrows the context window, the agent compresses older content into a shorter summary and keeps working. Done well, this preserves the thread. Done carelessly, it quietly corrupts the agent's understanding of what happened.

Consolidation is the broader process of moving information from short-term to long-term storage and deciding what is worth keeping. At session end, an agent might distill a long conversation into a few durable facts ("customer confirmed new billing address," "prefers email over phone") and write only those to long-term memory. Periodic jobs can then merge related episodic records into cleaner semantic facts and prune the noise. The cadence matters: too aggressive and you store junk, too conservative and you lose context the moment a session closes.

The risk to watch is summarization drift. Every time you compress history to fit a window, you throw away detail, and if you keep compressing summaries of summaries, you eventually hold a memory that no longer matches what actually happened. A number gets rounded, a caveat gets dropped, and three turns later the agent is confidently acting on a distorted version of the truth. Mitigations include keeping the raw record alongside the summary so you can re-derive it, summarizing from the original rather than from prior summaries, and consolidating on a schedule instead of compulsively on every turn.

A related design choice is deduplication and merging. If the same fact gets written five times across five sessions, naive memory grows linearly and retrieval gets noisier. Mature systems detect that "the CFO is Marcus Webb" and "our finance lead is Marcus Webb" are the same fact and store one canonical entry, updating it rather than appending.

Short-term vs long-term memory at a glance

The two layers solve different problems and have different costs. Short-term memory is about coherence inside one task; long-term memory is about continuity across many. Here is how they compare on the dimensions that matter when you design or evaluate an agent.

A worked example: a support agent that remembers

Picture a support agent connected to a help desk and a billing system. A customer, Dana, writes in: "My invoice looks wrong again." The word "again" only means something if the agent remembers.

On this turn, the agent embeds Dana's message and queries long-term memory. It pulls a profile fact (Dana is on the Enterprise plan, bills in euros) and an episodic record (two weeks ago, Dana reported a currency mismatch that support resolved by correcting the account region). Those few records get ranked, trimmed, and dropped into the context window alongside the live conversation and the current invoice the agent just fetched from billing. That assembled set is the short-term working memory for this turn.

Now the agent can act with continuity. It recognizes the pattern from the prior event, checks whether the region setting reverted, and proposes the same fix that worked last time, rather than asking Dana to re-explain her setup. Because this is a change to a billing record, the action routes through an approval step before anything is written, and the whole exchange lands in an audit log. After the turn, the agent consolidates: it writes a short episodic note ("second currency issue, same root cause, fix proposed") so the next person who touches this account inherits the history.

Strip out the memory and the same conversation becomes Groundhog Day. Dana re-explains, the agent re-investigates from scratch, and the recurring root cause never gets noticed. The memory is what turns a series of disconnected tickets into something that looks like a relationship.

The governance angle: what an agent should and should not remember

Memory is not just an engineering feature; it is a data-retention surface, and that changes the rules. The moment an agent writes a fact to long-term storage, you have created a record someone is accountable for. Treating memory as a free-for-all is how a helpful agent becomes a compliance incident.

Start with what not to remember. An agent should not persist payment card numbers, passwords, government IDs, health details, or anything else it does not have a clear reason to keep. The cleanest pattern is to redact or never write sensitive fields in the first place, rather than storing them and hoping to scrub later. Just because a value passed through the conversation does not mean it belongs in permanent memory.

Then there is scope. In a multi-tenant or multi-user deployment, memory must be partitioned so one user's facts can never surface in another user's context. A retrieval bug that mixes tenants is both a privacy breach and a credibility disaster. Memory should also respect the same least-privilege access controls as the rest of the system: who can read a given memory should follow who is allowed to see the underlying data.

Finally, retention and deletion. Memories should expire or get reviewed on a defined schedule, and when a user exercises a deletion right under a regime like GDPR, the erasure has to reach every tier, including vector index entries and backup snapshots, not just the obvious table. That is hard to retrofit, so it belongs in the design from day one alongside encryption at rest and in transit and an audit trail of what was stored, recalled, and removed.

Common mistakes teams make with agent memory

Most memory problems are not exotic. They are a handful of predictable errors that compound. Watching for these will save you the worst of the debugging.

• Remembering everything. Writing every message to long-term storage seems thorough, but it floods retrieval with noise and inflates cost. Store distilled facts and meaningful events, not raw transcripts.
• Compressing on summaries of summaries. Repeated lossy compression causes drift, where the agent's memory slowly diverges from reality. Summarize from the original record and keep the raw version recoverable.
• Confusing recall with relevance. Pulling the ten most similar memories is not the same as pulling the right ones. Without ranking and filtering, an agent will confidently act on a stale or tangential fact.
• Never forgetting. Memory with no expiry or review accumulates outdated facts, like a job title that changed a year ago. Build decay and refresh into the design, not as an afterthought.
• Ignoring tenant scope. If retrieval is not partitioned per user and per organization, you eventually leak one customer's data into another's session. This is a privacy bug, not just a quality bug.
• Skipping deletion plumbing. Storing memory across structured, vector, and backup tiers without a way to erase from all of them makes data-subject deletion requests nearly impossible to honor.

How much memory complexity do you actually need?

Not every agent needs a graph database and a five-tier memory architecture. Matching the memory design to the job avoids both under-building and over-engineering. Use this rough decision framework.

Start at the lightest tier that does the job and add complexity only when a real limitation forces it. The cost of memory is not just storage; it is the retrieval logic, the governance, and the failure modes you take on. The illustrative figures below show how a layered approach tends to shift agent behavior on the things teams usually measure.

• If your agent handles single-shot tasks with no follow-up (generate a report, answer one question), short-term memory in the context window is enough. Do not add a store you do not need.
• If conversations span multiple turns but not multiple sessions, focus on managing the context window well: trim, summarize, and order it. You still may not need persistent storage.
• If users return and expect to be remembered, add long-term memory, starting with structured profile facts and a vector store for episodic recall. This covers the large majority of business agents.
• If your questions are heavily relational (org charts, supply chains, who-knows-whom), add graph-style memory on top of vectors so the agent can traverse relationships, not just similarity.
• If you operate in a regulated or multi-tenant setting, treat governance (scope, retention, deletion, audit) as a first-class requirement from the start, regardless of how simple the rest of the memory is.

How memory connects to the rest of an agent stack

Memory does not live alone. It interacts with retrieval, tools, and guardrails, and understanding those seams helps you reason about the whole system. Retrieval-augmented generation and long-term memory share the same machinery; the difference is mostly what you are retrieving, a document corpus versus the agent's own accumulated facts. If you already run RAG, you already have most of the parts.

Tools and memory feed each other. A tool call to your CRM produces fresh data that becomes short-term memory for this turn, and the agent may consolidate a slice of it into long-term memory afterward. Conversely, long-term memory can make tool use smarter by recalling which tool worked last time for a similar request. Memory and observability also pair up: when you trace an agent run, the memories it retrieved and wrote are part of the evidence for why it did what it did.

For teams shipping agents into a real business, the practical move is to treat memory as one governed layer among several. An agent that takes action across your CRM, support desk, and data tools, then delivers finished work to Slack or Teams on a schedule, needs its memory wrapped in the same controls as its actions: approvals before sensitive writes, least-privilege access, and an audit trail. A platform like Onpilot is built around that governed model, so memory, action, and accountability stay in one place rather than bolted on separately.

The short version: memory is the substrate that makes an agent feel continuous, retrieval is how it reaches into that substrate, tools are how it changes the world, and governance is what keeps all of it trustworthy. Design them together.