After reading Anthropic's The components of agent memory, I wanted to try some of its ideas in my existing shared memory system. The possibility that interested me was learning across tasks: an agent encounters a problem, checks what happened, and leaves behind a useful rule for the next attempt.
I added a way to prepare and review lessons, then tested changes to how an agent gathered the evidence needed to answer questions. The implementation worked, but the corrected comparison showed no improvement in complete answers. Automatic follow-up reading also increased the input tokens and time required.
That result changed what I think I need to measure before calling a memory upgrade useful.
What I took from Anthropic's post
Anthropic describes memory notes, representations such as wikis, and an offline process it calls dreaming. An agent can revisit earlier records, correct a mistaken note, and organize information for later use. The part I wanted to explore was the learning loop: investigate a failure, test an explanation, and retain a rule with evidence for why it should work.
The post also reports that memory behavior depends on the model and its instructions. In its tests, restricting agents to recording only failures reduced some of the benefit. I took that as a reason to consider verified successful strategies as well as corrections. Anthropic's research
My project has a different starting point. It keeps original files in place, builds a local search index, and lets different agents search and read selected passages. New memories enter a review queue through an explicit save. I wanted to adapt the ideas within that design.
What I added to my system
The lesson workflow accepts a proposed rule, its source evidence, the conditions where it applies, and its limits. It can compare reviewed answers before and after applying that rule, including separate development and holdout tasks. An accepted comparison makes the candidate ready for review; it does not automatically turn it into trusted memory.
This gives me a process for evaluating lessons. It does not establish that a lesson improves future work. I also did not reproduce Anthropic's full transcript-based dreaming or wiki experiments.
For the actual model comparison discussed here, I tested a narrower prerequisite: could the agent recover the evidence it needed more effectively?
I made two changes. First, focused reads prefer complete lines or sentences when those units fit within the byte limit. This aims to preserve conditions and warnings that a fragment could lose. Second, I added evidence traces that retain separate reads of the same source, with each citation tied to the exact read that supplied it.
A private experiment then used those traces with a model controller that could choose the next read. The controller received the question and its actual observations, without the expected answers or grading rubric.
The measured change was in evidence gathering. Learning a rule and showing that it helps on a later task remains a separate test.
The comparison I ran
I used four demanding questions from two operational documents. They covered experiment interpretation, AI change review, external work agreements, and knowledge transfer. Each question required several conditions from its requested document.
Each of three approaches answered each question twice, producing 24 final answers:
| Approach | How it gathered evidence |
|---|---|
| Baseline | Focused reads of the top three visible search results |
| Boundary preservation | The same fixed reads, with complete-unit boundaries preferred |
| Adaptive reading | Boundary preservation, followed by controller-selected source actions |
All approaches had at most three source operations, each capped at 1,200 UTF-8 bytes, including output metadata. Locating a passage consumed an operation too. The adaptive approach could make up to two additional controller inference calls.
The answering model was gpt-6.1-sol, with low reasoning effort. It had to answer from the delivered excerpts, acknowledge missing evidence, and provide exact citations. Answers were requested in Korean within 700 characters.
I separated two outcomes: whether all specified evidence reached the model, and whether the final answer satisfied the full task rubric. A relevant document appearing in search was insufficient for either.
The improvement did not survive the corrected run
An earlier pilot appeared to raise complete answers from 0/8 to 2/8 with adaptive reading. That run contained an evaluation-driver bug: it rejected some valid line ranges around located text.
After repairing the driver, the experiment made 40 fresh inference calls: 24 answer calls and 16 controller calls. The questions, rubric, prompts, model, engine, and source-operation budget stayed the same. The apparent completion gain did not reproduce.
| Measure | Baseline | Boundary preservation | Adaptive reading |
|---|---|---|---|
| Complete answers under the rubric | 0/8 | 0/8 | 0/8 |
| All specified evidence delivered | 2/8 | 2/8 | 0/8 |
| Source output bytes, across eight trials | 28,240 | 25,734 | 24,928 |
| Total input tokens, including controllers | 77,838 | 76,098 | 221,616 |
| Median total time per trial | 12.96 s | 13.35 s | 28.69 s |
Boundary preservation reduced source output bytes by 8.87%, without improving completion. Adaptive reading used 2.85 times the input tokens and took 2.21 times as long as the baseline in this setup. Those token counts include the surrounding inference context; they are not billing totals.
All 24 answers were judged grounded and within scope in the recorded review. Their exact citations were valid. The answers often identified what the excerpts did not establish, but none satisfied the complete task rubric.
The public measurement summary contains the counts, protocol identifier, and limitations. It omits private source text and local paths.
These are exploratory results from four questions repeated twice per approach, not eight independent questions or a general accuracy estimate. The questions had already appeared in the diagnostic run. The rubric author also reviewed anonymized outputs, so this was not an independent human evaluation. The model's backend snapshot was unavailable.
What went wrong in the extra reads
The controller could often name the missing information correctly. Reaching that information was harder.
For the experiment question, it recognized that it still needed statistical interpretation rules. It located an acronym and read a nearby experiment template. The template listed a data-quality check, but did not provide the explanation the answer required.
For the external work agreement question, it identified missing change-request requirements, then repeated the same source reference and query. For knowledge transfer, it reread material about the handover test without reaching the paragraph about handling secrets. Earlier evidence remained available, but the extra operations did not close the gaps.
These cases made the next problem more concrete: track which passages have already been read, distinguish a template from an explanation, and choose a new source window that can supply the missing condition.
What I learned
Passing implementation checks does not demonstrate usefulness
The public source suite passed 255 tests when I rechecked it. That supports the tested implementation behavior: boundaries, evidence accounting, citation checks, and review gates.
The model comparison asks another question: does using the feature help finish a task? I need both kinds of checks. A working mechanism can still be an unhelpful default.
Valid citations can coexist with an incomplete answer
The system could produce an honest, cited explanation of what it had read while still missing a required approval condition or exception. Groundedness and completion therefore need separate measurements.
Acknowledging missing evidence is useful behavior. I also need to count the unfinished task it leaves behind.
Smaller source output can hide a larger total cost
The adaptive approach returned fewer source bytes, yet required much more inference context and time. Its controller calls were part of the feature's cost. Measuring only the final excerpts would have hidden most of that increase.
Boundary preservation also needs care: returning less text can remove useful adjacent context. Fewer bytes alone do not establish a better read.
A lesson needs evidence that it transfers
“Read again when something is missing” sounds reasonable. This experiment shows how easily that instruction can spend the remaining budget on overlapping or unhelpful text.
For a rule to become useful memory, I want to record when it applies, how it changes the agent's action, and whether it helps on a new task. The lesson workflow provides a place to check those claims. Its existence does not make them true.
What I will test next
First, I want to check whether a deliberately selected reference trace can deliver the complete evidence within the same reading budget. That would help separate a navigation problem from an insufficient budget.
Then I want to test a specific rule for choosing an unread passage, with the rule enabled and disabled on fresh tasks. I will measure completion, evidence coverage, repeated source windows, total inference tokens, and time. The corrected questions from this experiment will remain diagnostic examples.
For now, boundary preservation and explicit evidence tracing remain installed. The automatic controller stays outside the production default. My attempt to adapt Anthropic's ideas gave me a review process and clearer failure cases; it has not yet demonstrated better answers in this comparison.
Thanks for reading. Keep following your curiosity.