Two AI Agents Made the Right Decision. Together, They Corrupted the Same State.

The hidden concurrency problem behind multi-agent AI systems

Something is interesting about debugging intelligent systems: sometimes, every component appears to be doing exactly what it was designed to do, and the overall system still produces the wrong result. I have been thinking about this increasingly as AI agents move beyond answering questions and start performing tasks across business applications, databases, APIs and enterprise workflows. We have spent considerable time discussing whether an AI agent can reason correctly, whether it can remember previous interactions, whether it can reject outdated information and whether it can follow instructions.

But there is another problem that deserves just as much attention. What happens when two AI agents make perfectly reasonable decisions using the same information, at almost the same time? Neither agent necessarily makes a reasoning mistake. Neither misunderstands the user's request. Neither intentionally violates a business rule.

Yet, together, they can corrupt the same business state. I find this particularly interesting because it exposes a limitation in how we sometimes think about multi-agent architecture. We often focus on communication, orchestration and task delegation. We design agents to collaborate, exchange messages and divide responsibilities.

But collaboration does not automatically provide consistency. Two agents can communicate successfully, understand their individual responsibilities and still execute conflicting operations. And the problem becomes much more serious when those operations involve money, inventory, customer records, payments, manufacturing schedules or other business-critical resources.

Let's look at a simple example.

1. The problem begins when two agents see the same number

Imagine an e-commerce platform with an inventory of ten units of a particular product. A customer places an order. At the same time, another customer places an order for the same product. To handle the workload, the platform uses two AI agents.

  • Agent A handles the first customer's order.
  • Agent B handles the second customer's order.

Both agents connect to the same inventory service. At 10:00:01, Agent A requests the current inventory. The database responds:

Available inventory: 10

Almost simultaneously, Agent B requests the same information. The database also responds:

Available inventory: 10

Both agents now have a perfectly valid observation of the inventory at the time they read it. Agent A determines that reserving seven units is possible. Agent B determines that reserving six units is also possible. Individually, both decisions are reasonable. The problem appears when they execute.

Event Agent A Agent B Actual inventory
Initial state Reads 10 Reads 10 10
Decision Reserve 7 Reserve 6 10
Execution Reserves 7 Attempts 6 Depends on implementation
Final state Believes successful Believes successful Potentially inconsistent

If the system performs independent read-and-write operations without proper concurrency protection, both agents may successfully reserve their requested quantities. The business has now promised thirteen units when it only had ten. This is not an AI reasoning failure. It is a concurrency-control failure. That distinction matters because improving the language model, adding more instructions or introducing another orchestration layer will not automatically fix it.vThe system needs a mechanism that protects the shared state.

2. Why can individually correct agents produce an incorrect outcome

One assumption I frequently question when looking at agentic architectures is that an agent's successful execution means the business operation was valid. It doesn't.

An AI agent typically works with a sequence of observations, decisions and actions. For example:

  1. Retrieve inventory.
  2. Evaluate whether sufficient stock exists.
  3. Decide whether to accept the reservation.
  4. Call the inventory API.
  5. Report the result.

This looks straightforward. However, there is a gap between steps two and four. The information used to make the decision may no longer represent the current state when the action is executed. I think this is one of the most important distinctions in designing reliable AI systems:

A decision can be valid when it is made and invalid when it is executed.

This is not unique to AI. Traditional distributed systems have dealt with concurrency for decades. What makes it particularly relevant to AI agents is that these systems introduce additional layers between observation and execution. An agent may:

  • Retrieve information from multiple services.
  • Spend time reasoning about the information.
  • Ask another agent for confirmation.
  • Wait for human approval.
  • Generate a tool call.
  • Retry an operation after a timeout.
  • Execute the operation through an external API.

Every additional step creates an opportunity for the underlying state to change. The agent might still be following its original reasoning perfectly. But the environment has moved on.

3. The first hidden failure: stale reads

Let's examine the inventory example more closely. Suppose the database initially contains:

CREATE TABLE inventory ( product_id VARCHAR(50) PRIMARY KEY, available_quantity INTEGER NOT NULL, version INTEGER NOT NULL DEFAULT 1 ); INSERT INTO inventory (product_id, available_quantity, version) VALUES ('LAPTOP-001', 10, 1); 

Agent A retrieves the inventory:

SELECT available_quantity FROM inventory WHERE product_id = 'LAPTOP-001'; 

Result:

available_quantity ------------------ 10 

Agent B executes the same query and receives the same result. Both agents now have an identical snapshot. The problem is what happens next. A naive implementation might look like this:

def reserve_inventory(product_id, quantity): current = get_inventory(product_id) if current >= quantity: update_inventory( product_id, current - quantity ) return "Reservation successful" return "Insufficient inventory" 

At first glance, this looks reasonable. But there is a serious problem. The read and write operations are separate. Agent A reads ten and calculates three remaining. Agent B reads ten and calculates four remaining. If Agent A writes three and Agent B subsequently writes four, the final database value becomes four.

Seven units have effectively disappeared from the accounting. This is a classic lost-update problem. The database contains a number that does not accurately represent the sequence of business operations. The code is not necessarily syntactically incorrect. The business rule is simply not protected against concurrent execution.

There are two separate issues here:

  • Stale reads: agents make decisions using observations that may no longer be current.
  • Lost updates: concurrent writes overwrite changes made by another operation.

They are related, but they are not identical. And neither can be reliably solved by asking agents to communicate more politely.

4. The dangerous assumption: orchestration equals coordination

Multi-agent frameworks make it increasingly easy to distribute work. One agent can retrieve customer information. Another can calculate pricing. A third can check inventory. A fourth can execute the order.

A central orchestrator can assign tasks and monitor their progress. This is useful, but it introduces a misconception. We sometimes treat an orchestrated workflow as if it were automatically a transactionally consistent workflow. Consider this architecture:

flowchart TD A[Customer Order] --> B[AI Orchestrator] B --> C[Inventory Agent] B --> D[Pricing Agent] B --> E[Payment Agent] C --> F[(Inventory Database)] D --> G[(Pricing Service)] E --> H[(Payment Gateway)] F --> I[Order Result] G --> I H --> I

The orchestrator knows which agent performs which task. It may know when each task starts and finishes. It may even maintain a record of every message exchanged. But does it know whether two agents attempted to modify the same inventory record? Does it know whether an inventory reservation was committed while a payment operation failed? Does it know whether a retry created a duplicate reservation?

Not necessarily. Task coordination and state coordination solve different problems. An orchestrator controls workflow progression. A database transaction or concurrency-control mechanism protects shared data. Both are important, but they should not be confused. In fact, the more independently agents operate, the more important it becomes to establish clear ownership and consistent boundaries.

5. Fixing the immediate problem: atomic database operations

The first improvement is to stop separating the inventory check from the inventory update. Instead of reading the available quantity, calculating a new value in application code and writing it back, we can make the database perform the check and update as one atomic operation. For example:

UPDATE inventory SET available_quantity = available_quantity - 7 WHERE product_id = 'LAPTOP-001' AND available_quantity >= 7; 

The important part is the condition:

AND available_quantity >= 7 

The database evaluates the condition as part of the update. If the update affects one row, the reservation succeeded. If it affects zero rows, there was insufficient inventory, or the product did not exist. We can apply the same logic to Agent B:

UPDATE inventory SET available_quantity = available_quantity - 6 WHERE product_id = 'LAPTOP-001' AND available_quantity >= 6; 

Now consider the concurrent execution. Initially:

Available inventory = 10 

Agent A attempts to reserve seven units. The database successfully updates the record:

Available inventory = 3 

Agent B then attempts to reserve six units. The condition is no longer satisfied. The database updates zero rows. Agent B receives a failure rather than a false success. The result is:

Operation Requested Result
Agent A 7 Successful
Agent B 6 Rejected
Remaining inventory 3

The business invariant is preserved. This is a much stronger design because the database, rather than the AI agent, becomes the final authority on whether the reservation can happen. The agent can propose an operation. The database decides whether that operation is valid against the current state. That is an important architectural boundary.

A practical Python implementation

Using PostgreSQL and a parameterised query, the reservation logic could look like this:

def reserve_inventory(conn, product_id, quantity): if quantity <= 0: raise ValueError("Quantity must be positive") with conn.transaction(): result = conn.execute( """ UPDATE inventory SET available_quantity = available_quantity - %s WHERE product_id = %s AND available_quantity >= %s RETURNING available_quantity "", (quantity, product_id, quantity) ).fetchone() if result is None: return { "success": False, "reason": "Insufficient inventory" } return { "success": True, "remaining_quantity": result[0] } 

The database operation is atomic. The inventory cannot become negative through this operation, assuming all inventory modifications follow the same invariant. However, there is another detail worth considering. What happens if the inventory update succeeds but the reservation record fails to get created? We could have a quantity deducted without a corresponding reservation. That is why real reservation workflows need a transaction that covers all related database changes. For example:

BEGIN; UPDATE inventory SET available_quantity = available_quantity - 7 WHERE product_id = 'LAPTOP-001' AND available_quantity >= 7; -- Application checks that exactly one row was updated. INSERT INTO reservations ( product_id, quantity, status ) VALUES ( 'LAPTOP-001', 7, 'CONFIRMED' ); COMMIT; 

If the reservation insert fails, the transaction should roll back, restoring the inventory quantity. This gives us a much stronger guarantee within a single transactional database. It does not, however, solve every problem in a distributed multi-agent architecture.

6. Optimistic concurrency: making agents prove that their information is still valid

Atomic updates are useful, but there are situations where agents need to perform more complicated calculations before modifying a record. For example, a manufacturing agent might retrieve:

  • Current machine availability.
  • Production capacity.
  • Maintenance status.
  • Raw material quantities.
  • Existing production commitments.

It may take several seconds to evaluate the information. By the time it is ready to schedule production, another agent may have modified the same resource. In these situations, optimistic concurrency control becomes useful. The basic idea is straightforward:

An agent can prepare a decision using a particular version of the data, but the system must verify that the version has not changed before accepting the decision.

We can use the version column introduced earlier.

Suppose the initial record is:

product_id: LAPTOP-001 available_quantity: 10 version: 1 

Agent A reads version 1. Agent B also reads version 1. Agent A attempts to reserve seven units:

UPDATE inventory SET available_quantity = available_quantity - 7, version = version + 1 WHERE product_id = 'LAPTOP-001' AND version = 1 AND available_quantity >= 7; 

The update succeeds. The database now contains:

available_quantity: 3 version: 2 

Agent B still holds version 1. When Agent B attempts its update:

UPDATE inventory SET available_quantity = available_quantity - 6, version = version + 1 WHERE product_id = 'LAPTOP-001' AND version = 1 AND available_quantity >= 6; 

The update affects zero rows. Why? Because the version is now 2. Agent B's original observation is no longer current. It must retrieve the latest state and reconsider its decision. This is particularly useful for AI agents because it gives us a way to connect the agent's reasoning context to the actual data version used during execution.

Instead of simply saying: "Reserve six units." The agent's request effectively becomes: "Reserve six units, provided that the inventory is still at the version I evaluated." That is a much safer instruction.

The retry should not blindly repeat the original decision

This point is important. When an optimistic concurrency check fails, the agent should not automatically resend the same operation with the new version. It should first determine whether the original decision is still valid. For example:

def reserve_with_version( conn, product_id, quantity, expected_version ): result = conn.execute( """ UPDATE inventory SET available_quantity = available_quantity - %s, version = version + 1 WHERE product_id = %s AND version = %s AND available_quantity >= %s RETURNING available_quantity, version "", ( quantity, product_id, expected_version, quantity ) ).fetchone() if result is None: return { "success": False, "reason": "STATE_CHANGED", "retry_required": True } return { "success": True, "remaining": result[0], "new_version": result[1] } 

The next step should be a fresh read followed by a new evaluation. The important thing is that the original decision is not treated as permanently valid. This is a useful distinction between retrying an operation and reconsidering a decision. They are not the same thing.

7. Idempotency: what happens when an agent does not know whether its action succeeded?

Concurrency is only part of the problem. There is another scenario that can be equally dangerous. Imagine Agent A submits a reservation request. The database successfully processes the request. But before the confirmation reaches the agent, the network connection times out. Agent A now faces an uncertain situation.

Did the reservation succeed? Did the request fail? Did the database commit the operation but lose the response? The agent does not know.

A common reaction is to retry. But if the original operation already succeeded, retrying could reserve the inventory twice. This is where idempotency becomes essential. An idempotent operation produces the same effective result when the same request is processed repeatedly. For example, instead of submitting an operation with no unique identity:

{ "product_id": "LAPTOP-001", "quantity": 7 } 

The agent submits a unique request identifier:

{ "request_id": "reserve-order-78421", "product_id": "LAPTOP-001", "quantity": 7 } 

The backend stores the request identifier and associates it with the resulting reservation. A simplified database structure could be:

CREATE TABLE reservation_requests ( request_id VARCHAR(100) PRIMARY KEY, product_id VARCHAR(50) NOT NULL, quantity INTEGER NOT NULL, status VARCHAR(30) NOT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); 

The backend should process the request identifier and inventory modification within the same database transaction.

The workflow becomes:

  1. Receive the reservation request.
  2. Check whether the request identifier already exists.
  3. If it exists, return the original outcome.
  4. If it does not exist, attempt the inventory reservation.
  5. Record the result.
  6. Commit the transaction.

A unique database constraint protects against two concurrent submissions using the same identifier. There is another subtle point here. The backend should also validate that a repeated request identifier has the same business parameters. If an agent first submits a request for seven units and later reuses the same identifier for nine units, the backend should reject the conflicting request rather than silently treating it as the original operation. Idempotency protects against duplicate execution. It does not automatically resolve conflicting business decisions. That distinction is important.

8. Distributed transactions: the problem becomes bigger than one database

So far, we have discussed inventory operations within a database. But real enterprise workflows rarely operate within one database. Consider an order-processing workflow involving:

  • An inventory service.
  • A payment gateway.
  • A customer management service.
  • A shipping provider.
  • An AI orchestration platform.

The inventory reservation may succeed. The payment authorisation may succeed. But the shipping service may become unavailable. What should happen? Should the inventory remain reserved? Should the payment be reversed? Should the order remain pending? This is where distributed transaction design becomes important.

A traditional database transaction can provide atomicity across operations within its transactional boundary.

It cannot automatically guarantee atomicity across unrelated external services.

One common architectural approach is the Saga pattern.

A Saga divides a larger business operation into smaller transactions. Each transaction has a corresponding compensating action when the overall workflow cannot continue.

For example:

Step Operation Possible compensation
1 Reserve inventory Release inventory
2 Authorize payment Void payment authorization
3 Create shipment Cancel shipment
4 Confirm order Mark order as failed or cancelled

The workflow might look like this:

flowchart TD A[Order Received] --> B[Reserve Inventory] B --> C{Reservation Successful?} C -->|No| X[Reject Order] C -->|Yes| D[Authorize Payment] D --> E{Payment Successful?} E -->|No| F[Release Inventory] E -->|Yes| G[Create Shipment] G --> H{Shipment Created?} H -->|No| I[Void Payment] I --> F H -->|Yes| J[Confirm Order]

An AI orchestrator can manage this workflow, but the compensating actions should be explicitly defined and implemented by the relevant services.

The agent should not simply be instructed to "undo everything" when something fails.

Some actions cannot be perfectly reversed.

A payment authorisation may be voided, but a captured payment may require a refund. A shipment may already be in transit. A manufacturing operation may have consumed physical material.

Compensation is a business process, not merely a reversal of database values.

This is especially important when agents operate across multiple services with different consistency guarantees.

9. The problem of two agents retrying at the same time

There is another failure mode that deserves attention.

Suppose both agents detect a conflict.

Both decide to retry.

Both retrieve the updated inventory.

Both independently determine that another reservation attempt is possible.

If the system does not use atomic updates or version checks, we have simply recreated the original problem.

Retries are not inherently safe.

In fact, poorly designed retries can increase contention and make the system less stable.

Consider the following retry logic:

while not a success: success = reserve_inventory( product_id, quantity ) 

This is dangerous because it does not distinguish between different failure conditions.

A request may fail because:

  • Inventory is insufficient.
  • The database is temporarily unavailable.
  • The record version changed.
  • The service timed out.
  • The request was already processed.

These failures require different responses.

A better approach is to classify errors:

Failure Appropriate response
Insufficient inventory Reject or request a revised quantity
Version conflict Refresh state and reconsider
Temporary database failure Retry with bounded backoff
Network timeout Check the original request ID
Duplicate request Return the recorded outcome
External service unavailable Apply workflow recovery policy

For transient failures, exponential backoff with jitter can reduce repeated collisions.

For example:

import random import time def retry_delay(attempt): base_delay = min( 0.5 * (2 ** attempt), 10 ) return base_delay + random.uniform(0, 0.3) 

The retry policy should also have a maximum number of attempts and an overall deadline.

An AI agent should not be allowed to retry an operation indefinitely simply because it believes the task remains incomplete.

A bounded retry policy is a reliability feature.

It is also a cost-control feature.

10. Shared memory is not the same as shared truth

This brings me to a broader architectural issue.

Many multi-agent systems use shared memory, message histories or common context stores.

These mechanisms help agents exchange information.

But shared memory is not necessarily a source of truth.

Imagine that Agent A writes:

{ "product_id": "LAPTOP-001", "available_quantity": 10, "observed_version": 1 } 

Agent B reads this information from shared memory.

A moment later, another operation reduces the inventory to three.

The shared memory still says ten.

Both agents have access to the same information, but the information is outdated.

Sharing information does not make that information current.

This is why I prefer separating three concepts in agent architecture:

1. Working context

The information an agent uses to reason about a task.

2. Shared coordination state

The information agents use to communicate progress, ownership and workflow status.

3. Authoritative business state

The data that determines whether an operation is actually valid.

These layers can be connected, but they should not be treated as interchangeable.

A memory system can tell an agent what it previously observed.

A coordination service can tell it what another agent is doing.

Only the authoritative business service should determine whether the operation can be committed.

For an inventory system, that authority might be a transactional database.

For a payment system, it might be the payment provider's transaction record.

For a manufacturing system, it might be the production scheduling service.

The specific implementation varies.

The principle does not.

An agent should never treat a remembered state as proof that the current state is still valid.

11. A better architecture for multi-agent systems

If I were designing an agent-based order-processing system, I would avoid allowing every agent to independently modify shared business data.

Instead, I would establish a controlled execution boundary.

Agents could independently analyse information, prepare recommendations and request actions.

But state-changing operations would pass through a service responsible for enforcing business invariants.

Consider this architecture:

flowchart TD A[Customer Request] --> B[AI Orchestrator] B --> C[Inventory Agent] B --> D[Pricing Agent] B --> E[Payment Agent] C --> F[Action Request] D --> F E --> F F --> G[Execution Gateway] G --> H[Authorization Check] H --> I[Concurrency Control] I --> J[Idempotency Check] J --> K[Business Rules] K --> L[(Authoritative Database)] L --> M[Committed Result] M --> N[Event / Audit Log] N --> O[Agent Feedback]

The execution gateway is particularly important.

It should validate:

  • Whether the agent is authorised to operate.
  • Whether the request is associated with the correct workflow.
  • Whether the operation has already been processed.
  • Whether the expected state version is still current.
  • Whether the business rules permit the operation.
  • Whether the database can safely commit the change.

This architecture also creates a useful separation of responsibilities.

The AI agent is responsible for interpreting context and proposing actions.

The execution layer is responsible for validating and enforcing those actions.

The database is responsible for preserving transactional consistency within its boundaries.

The audit layer records what actually happened.

These responsibilities should not be collapsed into a single agent prompt.

12. What about locking? Should agents wait for one another?

Another approach is pessimistic concurrency control.

Instead of allowing multiple agents to read and independently attempt modifications, the system can lock a resource while an operation is being processed.

In PostgreSQL, for example:

BEGIN; SELECT available_quantity FROM inventory WHERE product_id = 'LAPTOP-001' FOR UPDATE; -- Validate the requested quantity. UPDATE inventory SET available_quantity = available_quantity - 7 WHERE product_id = 'LAPTOP-001'; COMMIT; 

The row-level lock prevents another transaction from modifying the same row concurrently until the transaction completes.

This can be useful when an operation requires several dependent database steps.

However, locking introduces its own challenges.

Long-running transactions can reduce throughput.

Deadlocks can occur when transactions acquire locks in conflicting orders.

And holding a database lock while an AI agent reasons, calls an external API or waits for human approval is generally a bad idea.

Imagine locking inventory while an agent spends eight seconds deciding whether a discount should be approved.

That is not a good use of a database transaction.

A better pattern is:

  1. Let the agent perform its reasoning outside the transaction.
  2. Prepare the proposed operation.
  3. Open a short transaction.
  4. Validate the latest state.
  5. Apply the update.
  6. Commit immediately.
  7. Return the result to the agent.

The transaction should protect the critical operation, not the entire AI reasoning process.

Optimistic versus pessimistic concurrency

Consideration Optimistic concurrency Pessimistic concurrency
Main mechanism Version checks Locks
Conflict handling Detect and retry Wait or block
Best suited for Relatively low contention Higher contention
Throughput Often better with fewer conflicts Can decrease under heavy contention
Implementation Version columns or conditional updates Row-level or other locks
Main risk Repeated conflicts Lock contention and deadlocks

Neither approach is universally better.

The choice depends on how frequently conflicts occur, how expensive retries are and how long the critical operations take.

For many AI-driven workflows, optimistic concurrency is a reasonable starting point because agent reasoning happens outside the database transaction.

13. The human approval problem: approval does not freeze the world

There is one more scenario worth considering.

Suppose an AI agent recommends a production schedule.

A human manager reviews the proposal and approves it.

The agent then executes the schedule.

But between approval and execution, another system changes the machine availability.

Should the original approval still be valid?

Not necessarily.

The manager approved a particular decision based on a particular state.

If that state changes materially, the approval may no longer authorise the new operation.

This creates an important connection between concurrency control and human oversight.

An approval should be bound to the specific operation and the relevant state that was reviewed.

For example:

{ "approval_id": "approval-9082", "operation": "schedule_production", "machine_id": "MACHINE-14", "production_units": 500, "expected_machine_version": 12, "approved_by": "manager-104", "status": "APPROVED" } 

At execution time, the system checks whether the machine is still at version 12.

If the version has changed, the operation should be reassessed according to the approval policy.

Depending on the change, this may require a new approval.

This is more meaningful than treating an approval as a general permission to execute whatever action the agent eventually decides to perform.

It also illustrates why human oversight and technical consistency cannot be designed independently.

The approval process must be connected to the state being approved.

14. Observability: how do we know two agents are interfering?

Concurrency bugs can be difficult to reproduce.

They may occur only under specific timing conditions.

A workflow might run thousands of times successfully before two agents happen to read and modify the same resource within a narrow time window.

Traditional application logs are often insufficient.

A useful multi-agent observability strategy should capture more than the agent's messages.

I would want to see:

Field Why it matters
Agent ID Identifies which agent initiated the operation
Workflow ID Connects related agent activities
Request ID Supports idempotency and duplicate detection
Resource ID Identifies the shared business object
Observed version Records the state used for reasoning
Execution version Records the version at execution
Operation type Shows what modification was attempted
Conflict status Identifies concurrency failures
Retry count Reveals repeated contention
Commit status Distinguishes proposed actions from committed actions

For example, a structured event might look like this:

{ "workflow_id": "order-78421", "agent_id": "inventory-agent-A", "request_id": "reserve-order-78421", "resource_id": "LAPTOP-001", "observed_version": 1, "execution_version": 2, "operation": "reserve_inventory", "requested_quantity": 7, "status": "VERSION_CONFLICT", "retry_count": 1 } 

This gives engineers something much more useful than a log entry saying:

"Inventory reservation failed."

It tells us which state the agent observed, which state existed when the operation was attempted and why the system rejected it.

I would also monitor:

  • Conflict rates by resource.
  • Average retries per operation.
  • Duplicate request frequency.
  • Transaction failure rates.
  • Time between state observation and execution.
  • Unresolved workflow states.
  • Compensation and recovery failures.

One metric I find particularly useful is the observation-to-execution gap.

The longer an agent spends between reading a resource and attempting to modify it, the greater the opportunity for that observation to become outdated.

It is not a direct measure of correctness, but it helps identify workflows that are more exposed to concurrency problems.

15. Testing concurrency is different from testing agent intelligence

A multi-agent system can pass every individual agent test and still fail when the agents operate together.

This is why I would separate agent-level testing from system-level concurrency testing.

An individual agent test might verify:

  • It correctly reads inventory.
  • It identifies insu...