Building a Shared Memory for My AI Agents

I am building a local memory system that different AI agents can share. The reason is continuity: I want to change agents, start a new session, or return to a project without rebuilding all the context from the beginning.

I started with existing memory tools. Over time, I wanted a system shaped more closely around how I work. I also thought building one would teach me a lot about how agent memory actually works. That combination led to Agent Memory, an ongoing project that I am developing through reading, experiments, and use.

Keeping the work continuous across agents

Different agents can help with different parts of a project. One might investigate a problem, another might implement a change, and a later session might need to understand why that change was made.

The context I want to carry between them includes the decisions, their reasons, the sources behind them, and the questions still open. A new agent should be able to recover that context without depending on the previous agent's conversation history.

For example, imagine returning to a project after choosing a storage design. I want the next agent to find the decision, understand the tradeoff, and check the relevant project files before proposing another change. That is the kind of handoff I am trying to support.

The aim is to give different agents access to the same working context. Each agent still needs to read and interpret the relevant material.

Moving from existing tools to my own system

Using existing tools gave me a starting point. Working toward my own version made me think more carefully about what I needed to preserve and how I wanted to retrieve it.

My work includes project code, documents, research, and decisions that develop over time. I want an agent to find a useful passage, see where it came from, and distinguish a current decision from an earlier idea. I also want the memory to remain usable when I change the agent that accesses it.

Those needs gradually became design choices: keep the original files in place, maintain a local search index, and provide a common way for agents to search and read. The index can be rebuilt while the source material remains independent of the client. Early architecture notes

Building it myself also lets me inspect each choice. When a result is unhelpful, I can follow the path from the question to the source text and work out which part needs attention.

Learning from other people's designs

I have been reading other systems and borrowing ideas as the project takes shape. These references helped me ask different questions about the design:

I am adapting selected ideas to my workflow. For instance, the distinction between a navigation aid and original evidence shaped my search-then-read approach. My implementation uses search snippets as pointers; it does not implement the full automatic wiki or gist-generation workflow described in those references.

As the project grew, I also tried selected configurations of qmd, HippoRAG 2, Graphiti, Mem0 OSS, and Letta Code on a small set of synthetic and public sources. The useful question was what each approach could teach me about my own workflow. The comparison record keeps the configurations and limitations alongside the observations.

That process led me to improve source navigation and add optional semantic search, which looks for related meaning when wording differs. The implementation notes describe those changes separately from the earlier comparisons.

Building a small loop I can understand

The basic loop is straightforward: an agent searches for relevant material, reads a selected passage, and uses that evidence to continue the task. A shared command-line interface gives different clients access to that loop.

Keeping search and reading separate has been useful. A short result helps choose where to look. The source read provides the text and its location. The reader also checks whether the source still matches the reference returned by search.

Saving a new memory is a separate decision. In the current design, an explicit capture creates a candidate for review. A suggestion from a conversation should not quietly become a rule for every future agent. Memory lifecycle

What building it is teaching me

One lesson arrived through a small retrieval failure. An earlier renderer could show rollout instructions while hiding rollback conditions from the same document. The relevant file had been found, but part of the information needed to act was missing. Practical review

That made the difference between finding a document and recovering useful context much clearer to me. The next step might be another passage, a definition elsewhere in a file, or a second source named by the first. The early retrieval flows helped turn those cases into concrete checks.

I am also learning how much judgment sits around the search itself. What deserves to be saved? Which project does it apply to? When should an old decision be reviewed? How much text does an agent need to understand an exception?

Implementing these details is part of why I wanted to start. Each working version gives me something concrete to inspect, use, and question.

Still building toward continuity

There is already a working local engine, but the broader goal is still something I am testing. Sharing access to files does not automatically give different agents the same understanding.

The test I care about is practical: can another agent recover the reason for a decision, check its source, and continue the work with less repeated explanation from me?

I expect my idea of a useful memory system to change as I build it. For now, I want to keep that cycle going: study an approach, implement a manageable part, try it in my own work, and learn from what remains difficult.

This is a development note as of October 18, 2026. The linked project notes preserve earlier design steps; the external-system comparison is a small diagnostic, rather than a general ranking of memory systems.

NEXT NOTE ↗

Why I Built DBC Lens