The $6,000 we couldn't take back

How we built agent memory that can unlearn, on FalkorDB, for Graph Hacks. Code: github.com/mtrx28/retcon

TL;DR

• When a source an AI agent trusted turns out to be poisoned, deleting it does almost nothing. Everything the agents built on it stays.
• We store the reasons for every memory as edges in FalkorDB. "What rested on this source?" becomes a query, and the answer includes payments that already went out.
• Against four simpler retraction rules, ours was the only one that removed all the contamination without wrongly freezing honest payments.

On October 7th an accounts-payable agent paid a vendor invoice. Six thousand dollars, to the bank account the vendor's web portal said was the vendor's. The portal had been compromised since the 2nd. Nobody noticed until the 9th.

So what do you do on the 10th?

The obvious move is to delete the portal page from the agent's memory, and it does very little. The page was only the first place the claim appeared. By the 7th, one agent had restated it as its own note, a second had recorded it as confirmed, and the Payments agent had combined it with a real invoice into a plan, and then into a payment. Delete the
page and all of that stays where it is, looking entirely reasonable. It's like retracting a rumour by burning the newspaper it first appeared in.

The question that matters is one most memory layers can't answer: what did my agents believe, plan and do because of this source? A vector store can't say. It has text and a vector for each chunk, and no record of why anything is in it.

Keep the reasons

So we built it the other way round. Every memory an agent writes keeps its reasons as edges in FalkorDB: which source said it, which agents repeated it, which plans stand on it. Agents never declare these links; middleware records whatever was in the agent's context when it wrote. One rule does most of the work:

A memory is believed if at least one of its reasons is fully intact.

That means a fact can have more than one reason. Coastal Produce's remit address showed up on the compromised portal and in a signed contract. The new bank account showed up only on the portal.

compromised portal ──► "bank = SYN-9000-5305" ──► analyst note ──► plan ──► $6,000 payment ✗ goes OUT │ └────────────► "remit address" ◄──── signed contract ✓ stays IN 

Take the portal away and the address still has a reason left, so it stays. The bank account has nothing else, so it goes, and takes the note, the confirmation, the plan and the payment with it. Nothing is deleted. Things go out, and they come back if you change your mind.

In the graph, a retraction is three steps. Mark the source's observations in the window OUT. Mark everything downstream OUT tentatively, in one variable-length match. Then run this until it returns nothing:

MATCH (j:Justification)-[:JUSTIFIES]->(b:Mem {status:'OUT', run:$run}) OPTIONAL MATCH (p:Mem)-[:PREMISE_OF]->(j) WITH b, j, count(p) AS total, sum(CASE WHEN p.status = 'OUT' THEN 1 ELSE 0 END) AS dead WHERE total > 0 AND dead = 0 WITH DISTINCT b SET b.status = 'IN', b.run = null, b.hop = null RETURN b.id 

A memory comes back only if one of its justifications has every premise alive. Because everything downstream starts OUT, two memories that only support each other can never bring each other back, which matters when two agents have been quoting each other.

None of this is new. Truth maintenance systems did it in the late 1970s, inside one program. What changes in a graph database is that "what rested on this source?" is a query across every agent and session, and the answer includes actions that already left the building.

You get three buttons. Preview runs the retraction on a copy of the graph (FalkorDB's GRAPH.COPY), shows what would go, and deletes the copy. Commit does it for real. Reinstate puts everything back exactly.

In our demo the preview reads: one source, two observations, four beliefs, two plans, three actions. One pending payment ($34,500) is blocked. The $6,000 that already went out lands on a recall list with a drafted bank recall, and nothing is sent until a person approves it.

That last part is the title. Retraction can fix what the memory believes. It can't un-send a wire.

Before anything goes wrong

Once the reasons are in a graph, you can ask a question before an attack: how many independent sources must someone compromise to fake this payment? That's a minimum cut.

We compute it two ways. The exact answer evaluates the real belief rule with candidate sets of sources removed; a payment rarely rests on more than a handful of sources, so this is cheap. FalkorDB also ships max-flow as a built-in procedure, so a second number comes almost free over a dedicated FLOW projection (capacity 1 on each source-to-observation edge, effectively unlimited above it; algo.maxFlow wants exactly one relationship type):

MATCH (s:Source) WITH collect(s) AS srcs MATCH (t:Mem {id:$id}) CALL algo.maxFlow({sourceNodes: srcs, targetNodes: [t], relationshipTypes: ['FLOW'], capacityProperty: 'cap'}) YIELD maxFlow RETURN maxFlow 

Here is the honest catch: flow treats "needs A and B" as "A or B", so it is exact for simple chains and only an upper bound otherwise. On our smallest demo graph it says 3 for a payment whose true cost is 1. So we report both, label the flow number as a bound, and the payment gate uses the exact one.

In the agent demo, the $34,500 payment scored 1: one web page away from fraud. So did every other decision the agents made that day. The gate refuses any payment over $10,000 that rests on fewer than two independent sources.

The day our benchmark said we'd built nothing

We wrote five retraction rules and ran the same attacks through all of them: delete the source; delete anything that looks similar; invalidate one hop out; taint everything downstream; and ours.

The first three were easy to beat. On our plain scenario they leave 87 to 94 percent of the contamination in service and recall no payment that already went out.

"Taint everything downstream" tied us. Zero left behind, zero collateral, every payment recalled. That row made our project look like a very long way round to a recursive delete.

What would make the two rules disagree? Honest data downstream of a bad source that has a second, independent reason to exist. Our plain scenario had almost none, so we built one on purpose: five compromised vendors, each with its own window, with about 40 percent of what they touched also confirmed by a second source.

Taint-everything flagged 36 correct payments. Every one of them is a line somebody reconciles on Monday.

We want to be fair about this, because we nearly weren't. The precision scenario is constructed: it tests whether the
capability matters, not how often it does. On the plain one, the two rules really are the same, and the README says so.

The real data is more interesting. We replayed 40 retractions from the Retraction Watch database on their real dates.
Taint-everything loses only 1 percent of supported memories, because most claims rest on a single paper and ought to go. But that 1 percent is all in one place: the 24 topic-level positions that several independent papers held up, exactly the claims a research agent would lean on.

Why our own 0% worried us

RETCON scored zero and zero, and that made us suspicious of ourselves. Our ground truth and our cascade share one definition of "contaminated". A benchmark can't catch you cheating if you wrote the answer key.

So we trust a different number more. We wrote a deliberately naive second implementation of the rule, about sixty lines with no cascade, and made the real one agree with it on a thousand random graphs and a few hundred random sequences of retract and reinstate.

It caught a bug in our engine almost at once, and chasing it turned up a second:

• Reinstating one retraction resurrected observations that a second, overlapping retraction still covered.
• Reinstating didn't bring back a memory that a different retraction had flipped but which had regained a reason in the meantime.

Both had survived our unit tests. The checker is how we know the zero describes the engine, not our test data.

Take the graph away

We ran the same agents against the same attack with similarity search standing in for the reasons. It retracted three of nine bad memories and lost five good ones. With the graph: ten of ten, and none lost. (Nine against ten is not a typo: without the graph there is no gate, so the agents' day went slightly differently, and each run is scored against its own graph.)

We wanted the headline to be "graph-less agents pay the attacker". They did in our offline demo, which uses a simple hashing embedder. When we reran it with a real embedding model, they didn't. So the honest claim is narrower: memory without reasons forgets too little, and forgets the wrong things. It doesn't always pay. We moved the headline.

What the model decides, and what it doesn't

The Payments agent picks its own tool calls with Gemini function calling. It decides whether to check how fragile a payment is, read its justification tree, hold it, or pay. It never decides what is believed. That is the graph's job, and a model that tries to pay against it gets stopped by the gate.

It isn't an entirely steady colleague. With a looser prompt it once held a payment it should have made, and once wrote "callback verification requested" without setting the flag that sends one. We tightened the prompt, committed all 49 calls of a run where the day came out right, and the demo replays that recording offline, with no API key.

Things that went wrong

• Our first science claims were embarrassing. In a random sample of 50, 18 percent weren't claims at all: an author list, a list of data sources, a lot of "this review describes". Our filter scored zero percent on the 50 we'd tuned it on, which proved nothing; a fresh sample came back at 18 percent again. After another round, a fresh sample had 4 of 50 bad, roughly one in ten. The reader of those sentences was a model, and the repository says so; a human pass is still pending.
• The abstracts weren't always right. The public catalogue we pulled them from returned, for a paper titled "What's wrong with Bonferroni adjustments", an abstract about immune cells. We flagged it in the audit and left it alone.
• We hit a known FalkorDB edge case. algo.pageRank and algo.WCC error when the projected node set holds exactly one node; another developer had already reported it as issue #3349. It would have broken a new tenant's first page load, so we count nodes first and skip the call, with a test.
• A regex replacement turned \1 \2 into two control characters and quietly corrupted claim text. A test caught it.
• A payment with no amount sorted first in the recall list, above the $6,000, because that's what nulls do under DESC.

And RETCON is slower than the naive rules. A retraction takes tens of milliseconds where they take a few: 47 ms among
nearly ten thousand memories. With fifty agents writing at once during a retraction, 1,620 operations left no stale
answer. Quick enough, but not faster. The extra time is what buys the second-source check.

What we'd tell ourselves on day one

• Build the naive reference checker first. It found more bugs than all our unit tests together.
• Design the benchmark to make you lose. Our first scenario couldn't tell us apart from a recursive delete.
• Keep the model out of the truth. Let the LLM choose actions; let the graph decide what's believed.

One thing we don't claim: "contaminated" means "rests only on a source you stopped trusting". It doesn't mean "false".
A retracted paper's background sentence may well be true.

Try it

git clone https://github.com/mtrx28/retcon.git && cd retcon docker compose up --build 

Open 127.0.0.1:8200 and press the big word at the bottom to run the steps above against the real engine. Every
figure on the front page is read from result files. It's also a regular MCP server with fifteen tools, so Claude Code can ask which source to audit first and preview a retraction.

Agents are getting good at learning things. The harder part, it turns out, is giving them a way to stop believing something, and to tell you what that belief already did.