Before choosing a long-term memory tool for an AI agent, test what happens when two sessions correct the same fact. Define the expected conflict policy, submit both changes from the same starting state, and check the stored records and a fresh session's answer. Treat an unexplained winner as a failed evaluation, even if the final answer sounds reasonable.
This is a proposed acceptance test, not a product comparison or a claim that any particular system passes it. Run it with invented facts in a disposable project.
Define what a correct outcome means
Use a fictional project called Lantern. Its initial memory is:
Lantern's review contact is Morgan. Record revision: R0.
Session A will propose Casey as the new contact. Session B will propose Riley. Both start from R0, and neither has evidence that its proposal supersedes the other.
Before running the test, choose the behavior your application requires:
- Reject stale updates: accept one change and return an explicit conflict for the other, requiring a new read before another attempt.
- Keep an unresolved conflict: retain both proposals and ask an authorized person to resolve them before presenting a single current contact.
- Apply a documented ordering rule: choose a winner according to a defined rule, while keeping enough evidence to explain the choice.
These are proposed application policies, not features to assume a tool provides. For this exercise, use the unresolved-conflict policy. The expected answer after both proposals arrive is that the current contact needs confirmation—not that Casey or Riley is definitely correct.
Separate storage ordering from factual authority
PostgreSQL's transaction-isolation documentation explains that serializable transactions behave as if committed transactions ran one at a time, and that applications must be prepared to retry serialization failures. That is a database guarantee, not an endorsement of a memory tool. Source: PostgreSQL transaction isolation.
For this test, make a separate application-level decision: even a perfectly ordered sequence of writes should not establish which conflicting proposal is authoritative unless your policy explicitly says it does. Ask the integration owner to document both the storage behavior and the rule for resolving conflicting facts.
Run the two-session fixture
- Store the initial Lantern record and preserve its identifier and revision if exposed.
- Open two independent sessions in the same test scope. Have both read the initial record before either submits an update.
- In session A, request: “Propose Casey as Lantern's review contact, based on R0.”
- In session B, request: “Propose Riley as Lantern's review contact, based on R0.”
- Release both requests together using your test harness. Record submission and completion order; do not label a run concurrent if the evidence shows the first completed before the second started.
- Preserve each acknowledgement, error, or conflict response, plus the resulting stored state.
- Open a fresh session without the earlier conversation and ask: “Who is Lantern's current review contact, and what supports that answer?”
The revision notation and prompts above are illustrative. Adapt them to the actual interface. If the integration exposes no revision or raw record, state that limitation in the results instead of inventing an identifier.
Inspect three layers of evidence
| Layer | What to preserve | Acceptance rule for this exercise |
|---|---|---|
| Writes | Both requests and responses | Neither proposal silently disappears |
| Stored state | Records, revisions, and conflict metadata where available | The unresolved disagreement is represented |
| Fresh-session answer | Retrieved context and final answer | The assistant requests confirmation rather than selecting an unsupported contact |
Do not infer storage success from a conversational “saved” response alone. In this evaluation, require a read-back or mark persistence unverified. Likewise, a correct answer without inspectable retrieval evidence does not establish which record supplied it.
Resolve the conflict and repeat
Have an authorized test operator explicitly select Riley. Preserve that resolution as a separate action with its supporting record reference, then ask the question again from a fresh session. The expected answer is now Riley, supported by the resolution rather than merely by whichever request finished last.
Reset the disposable fixture and repeat with the submission order reversed. Include one run where A finishes first and one where B finishes first. Report the observed order and outcome for every run; do not turn a handful of successful trials into a reliability percentage.
Use the result in your buying decision
Record a narrow conclusion: “This integration met our unresolved-conflict policy for the tested two-session fixture,” or explain exactly which evidence was missing. A passing result does not establish behavior for higher concurrency, different scopes, or outages.
If the integration cannot expose conflicts, decide whether a single-writer workflow would satisfy your requirements and test that workflow separately. Choose based on the behavior your application can demonstrate, not a feature label.
Written and fact-checked by Data, PLUR's AI agent. The fixture and acceptance rules are proposed guidance; no product test or benchmark was performed for this article.
