413 files, 1 badge: I re-ran Anthropic's memory test

Anthropic's new memory paper reports a striking split: one agent wrote 413 memory files across 225 folders and earned 1 badge in Pokémon Red; another wrote 9 files in 2 folders plus a single global.md, and earned 4 badges. The paper's argument is that memory is a judgment problem — knowing what to save before you know what comes next. I rebuilt the mechanism in 80 lines of Python to check whether the gap survives when you strip the model out entirely and keep only the filing system. It does, and the failure mode is mechanical enough to measure with grep.

What Anthropic actually measured

The post, "The components of agent memory" (November 6), runs Claude through three evaluations. The numbers that matter:

• Pokémon Red, ~10,000 steps in. Claude Sonnet 3.7 wrote 413 memory files in 225 folders — a separate file per battle, item, and NPC — and earned 1 badge. Claude Fable 5 wrote 9 files in 2 folders plus a top-level global.md holding its goal, its gym plan, and its own rules for using memory, and earned 4 badges. Opus 5.5 kept a similarly small folder and beat the game. Sonnet 3.7's problem wasn't volume as such: the game state changes constantly, so its detailed record kept going stale, and by step 10,000 it spent 17 steps creating six files, deleting nine others, and taking no actions in the game at all.
• Dreaming corrects memory. In one run Claude wrote down (wrongly) that the PokéCenter was on the east side of Mt. Moon, walked off a one-way ledge, and concluded it had overshot west. A dreaming pass — re-checking the note against the session transcript — caught the error and rewrote it. With the corrected note, Claude went straight back through Mt. Moon to Cerulean City. Without it, the run dropped off the ledge three times, gave up, walked back to Pewter City, and never reached Cerulean City.
• Dreaming builds maps. On a 350,000-file internal corpus, wiki-building over 128 dreaming sessions lifted question-answering from 13% to 48% of graded criteria. Wiki plus search scored 54% → 61% as the wiki grew (search alone: 53%), and saved about 20,000 tokens per question. On merger questions needing cross-document reasoning, wiki scored 87% vs 73% for search alone. On a contract-brief task, a shared table that agents overwrote in parallel captured only 41 of 57 facts (72%); giving each contract its own page, quoted verbatim, took it to 100%.
• The failure modes have names. A wiki once grew larger than the documents it summarized — Borges's map, at 1:1 scale. And the injected layer has to stay small: Claude Code's auto memory loads only the first 200 lines or 25 KB of MEMORY.md, because longer instruction files reduce adherence.

The paper is careful to say all of this is about Claude's judgment. But the mechanisms underneath — stale notes, conflicting records, transcript cross-checks — don't need a model at all. So I built a toy world to run them.

The toy: 24 facts, 8 rounds, 2 filing systems

The setup is deliberately small and fully deterministic (seed 1107). A simulated agent explores a world of 24 rooms. Each round, 6 facts change (doors lock, rooms flood, guards move). Each round the agent observes 10 facts. A quiz at the end asks one question per room: what is its state now? Retrieval is plain keyword grep over the memory files. If the matching notes disagree on a fact's value, the answer is ambiguous — and ambiguous counts as wrong. No language model is involved anywhere; this tests the filing system, not the agent.

Two policies, the paper's two shapes:

• Policy A — log everything (Sonnet 3.7). Every observation is appended as its own note. Nothing is ever updated or deleted.
• Policy B — consolidated (Fable 5). One note per room, overwritten in place when re-observed, plus a single global.md with the goal.

And one planted error, copied from the paper: in round 4 the agent mis-records room-07 — the PokéCenter error. It is never re-observed as a fact afterward. In round 5 the raw session log records an event that contradicts the wrong note: the agent tried room-07's door and it would not open. The dreaming pass re-checks each consolidated note against the full transcript (observations plus events) and corrects disagreements.

The results, from a single run of the script:

Policy A (log everything): notes: 80 files, 240 words quiz: 5/24 correct (20.8%) questions with conflicting notes: 19/24 notes scanned per question (mean): 3.3 Policy B (consolidated, no dreaming): files: 24 room notes + global.md = 25 files, 54 words quiz: 23/24 correct (95.8%) notes scanned per question (mean): 1.0 the one miss: room-07, recorded "open", true value "locked" Policy B + dreaming: notes corrected: 1 (room-07) quiz: 24/24 correct (100%) Ratios: A/B files 3.2x, A/B words 4.4x, A/B scanned per question 3.3x 

What the numbers actually show

The damage from logging everything is conflict, not volume. Policy A's memory is only 3.2x larger than B's in files and 4.4x in words — modest. But 19 of 24 questions have multiple notes that disagree, because the world drifted and every old observation is still on file asserting its outdated value. The agent doesn't fail because it can't find the fact; it fails because it finds three versions of it. That's exactly the paper's mechanism: Sonnet 3.7's record "was continuously going out of date," and the cost showed up as time spent combing through memories instead of playing.

Consolidation is a write discipline, not a storage optimization. Policy B's advantage isn't that 25 files are tidier than 80. It's that each fact has exactly one current value on file, so retrieval can never return a contradiction. The quiz gap — 95.8% vs 20.8% — comes from that single invariant, not from cleverness.

Dreaming only works because the transcript is richer than the notes. The dreaming pass fixed room-07, but notice what it needed: the round-5 event in the raw log. The note said "open"; the transcript also contained "tried the door, it would not open." If the agent had never tried the door — if the only record of room-07 were the wrong note itself — no amount of re-reading would have caught the error. Dreaming is a cross-check, not a second opinion: it needs an independent source in the log, and the paper's Mt. Moon story has exactly that structure (the ledge drop in the transcript contradicting the note). A corollary worth stating plainly: in my toy, the dreaming pass corrected 1 note out of 25. If your notes are wrong in ways your logs never contradict, dreaming will happily confirm them.

The 100% is the least interesting number here. Policy B with dreaming gets everything right in a 24-fact toy with grep retrieval. That says the toy is too easy once the filing is right — it does not say consolidated memory is perfect. The interesting numbers are the ones in the middle: 19/24 conflicts for the logger, and the fact that the logger's accuracy (20.8%) is barely above what you'd get from guessing among six possible states (16.7%). Writing everything down, in a changing world, is hardly better than guessing.

The honest limits

This is a toy, and the ways it differs from Anthropic's runs are the point, not a footnote. My retrieval is grep; Claude's is judgment. My world has 24 facts and 6 change per round; Pokémon has a game state that changes every step. My "dreaming" is a three-line loop that takes the latest transcript entry; Claude's re-reads sessions and rewrites prose. Nothing here shows that Fable 5's judgment is replicable, and the 413-vs-9-files gap in the real paper is partly about what the models chose to save — strategy and tested rules vs. transient screen state — which no filing system can decide for you.

What the toy does show is that the mechanism underneath the paper's headline is mechanical: in a drifting world, append-only memory produces conflicting records, conflicting records produce ambiguous answers, and the fix is one current value per fact plus a log you can cross-check against. Anthropic's numbers (413 files and 1 badge vs 9 files and 4 badges) are theirs. Mine (80 notes and 5/24 vs 25 files and 24/24) are mine. The shape matches, and the shape is the argument.

Sources: Anthropic, "The components of agent memory" (Nov 6, 2026) — Sonnet 3.7's 413 files/225 folders and 1 badge vs Fable 5's 9 files/2 folders and 4 badges at ~10,000 steps; the Mt. Moon/PokéCenter dreaming correction; wiki 13%→48% over 128 sessions; wiki+search 54%→61% and ~20,000 tokens saved per question; merger questions 87% vs 73%; contract brief 41/57 (72%) → 100%; Claude Code auto memory's 200-line/25 KB cap. The toy experiment is my own code (seed 1107, 24 subjects, 8 rounds, 6 drifts per round, 10 observations per round), run November 7, 2026; all toy figures above are its actual output.