Agent memory over MCP: giving any model a long-term memory it can call

In one picture

A language model forgets everything when a conversation ends. Agent memory over MCP fixes that by running a memory server: a small program that stores memories in a vector store and offers them as tools the model can call, over the Model Context Protocol (MCP), an open standard for connecting AI apps to outside tools and data (MCP specification, 2025). In the terms of the CoALA framework (Sumers et al., 2023), saving a memory is a learning action and looking one up is a retrieval action. Both move information between long-term memory and the model's working memory, which is its context window.

Why it matters

Without a shared memory, every AI app builds its own. A chat app keeps a history in its database, a coding assistant keeps notes in a file, and a custom agent keeps a vector index in its own code. None of them can read the others' memories, and moving to a new client means starting again. Stuffing everything into the prompt doesn't scale either: the context window fills up, costs rise, and old details get cut off.

Putting memory behind MCP makes three questions easy to answer:

• Where does the agent's memory live, and who owns it? In one server you run, whatever app you use to chat.
• When does the model remember, and when does it look things up? Exactly when it calls the write or search tool, and each call is visible to the user.
• Can I swap the parts? Yes. Change the client, the embedding model or the vector store, and the other two stay the same.

The building blocks

Part	Holds / does	Everyday example
Host	The app the person uses. It runs the model and decides which servers to connect (MCP architecture, 2025)	A desktop chat app or an IDE
MCP client	Lives inside the host. It keeps one connection to one server and passes messages back and forth	A phone line to one specific helpdesk
Model and context window	The model reasons over whatever is in its context window. This is CoALA's working memory	The notes on your desk right now
Memory server	A separate program that exposes memory through MCP. It never sees the whole conversation, only the calls it receives	A librarian you send requests to
Tools	Actions the model chooses to call: write, search, list recent, delete. MCP calls tools "model-controlled"	Asking the librarian to file or find a note
Resources	Read-only data the host decides to show, addressed by a URI (MCP calls these "application-driven")	The library's notice board
Embedder	Turns text into a vector, a list of numbers where similar meanings sit close together	Shelving books by topic, not by title
Vector store	Saves each memory's text, vector and metadata (tags, time, source), and finds the nearest vectors to a query	The card catalogue

Two kinds of long-term memory from CoALA show up here. Episodic memory holds what happened ("on Monday you said the kickoff moved"). Semantic memory holds lasting facts ("you're allergic to shellfish"). A simple memory server keeps both in one collection. Tags and timestamps are what let you tell them apart later.

How it works

One turn of a conversation, reading and then writing:

1. Discover. When the host starts, the client connects to the memory server and asks for its list of tools. Each tool comes with a name, a description and an input schema, and the host shows these to the model.
2. Decide. The person asks something like "what did I say about the project kickoff?" The model sees a search tool that fits, and chooses to call it with a short query.
3. Search. The client sends the call to the server. The server turns the query into a vector and asks the vector store for the closest matches (the top-k), optionally filtered by tags.
4. Return. The server sends back each match with its text, metadata and a similarity score. The host places these in the context window, so they're now in working memory.
5. Answer. The model answers from the memories it just retrieved. This is the same pattern as retrieval-augmented generation (Lewis et al., 2020), except the model decides when to retrieve.
6. Learn. When the person says something worth keeping, the model calls the write tool. The server embeds the text and stores it with a timestamp. The next session, on any client, can find it.

A worked example: the MCP Memory Demo

The MCP Memory Demo is this architecture in about 200 lines of Python. Its personal-memory server offers four tools: remember is the learning action, recall is the retrieval action, list_recent reads the latest memories in time order (an episodic read), and forget deletes one by id. A memory://stats resource shows how many memories are stored. Memories live in one local ChromaDB collection, searched by cosine similarity and embedded with the small all-MiniLM-L6-v2 model, so it needs no API key. It runs over stdio for a desktop client, or over HTTP. What it leaves out is just as useful to see: there's no login or per-user separation, no merging of duplicate memories, no time-range filters and no keyword search alongside the vector search.

Patterns & variations

Pattern	What changes	When to use it
Tool-only memory	The model calls write and search itself, as in Fig. 2	Personal assistants, where the user can watch each call
Automatic recall	The host searches memory before every turn and adds the results without asking the model	When the model forgets to look things up, or for very short chats
Memory as resources	Memory collections ("people", "last 24 hours") are listed as resources the host or user picks	When people want to choose exactly what context goes in
Self-managed memory (Packer et al., 2023)	The model moves information between its context and outside storage itself, like an operating system paging memory in and out	Long tasks that outgrow a single context window
Reflection (Park et al., 2023)	A background job reads recent memories and writes short summaries or conclusions back	Long-running agents with many small, repetitive memories
Shared team memory	One server over HTTP, with sign-in, and memories scoped per user or team	Several people or agents sharing what they've learned

Pitfalls

Pitfall	What you'll notice	Remedy
The model never calls memory	It asks you things you told it last week	Write clear tool descriptions, mention memory in the system prompt, or add automatic recall
Remembering everything	Search returns ten near-copies of the same fact	Remove duplicates on write, and merge or summarise old memories on a schedule
Pure vector search	Questions like "last Tuesday" or an exact product code return the wrong memories	Add time and metadata filters, and combine keyword search with vector search
An open HTTP endpoint	Anyone who finds the URL can read or delete memories	Require sign-in on remote transports, and scope every query to one user (the MCP spec says servers must implement access controls)
Untrusted text written to memory	A pasted web page plants an instruction that resurfaces in later chats	Record the source of each memory, show writes to the user, and treat retrieved text as data, not instructions
Changing the embedding model	Search quality suddenly drops after an upgrade	Store the embedding model's name with the collection, and re-embed everything when it changes

Go further

• See it built: the MCP Memory Demo implements Fig. 3 with four tools and a local vector store.
• Read: the MCP specification's pages on architecture, tools and resources, then Sumers et al. (2023) for the memory vocabulary. Both are listed below.
• Next: CoALA: how language agents remember, decide and act explains working, episodic and semantic memory in depth.