The duplicate delivery problem
Imagine a Node.js worker processing a blockchain deposit confirmation. It receives an event, updates a swap to exchange, and schedules the next operation.
The worker commits the update—but crashes before acknowledging the message. The queue delivers the event again. Another observer might even publish a new event ID for the same deposit confirmation.
If the handler simply performs an update and queues another job on every delivery, one deposit can trigger multiple exchange operations.
This is why idempotency matters: processing the same logical action more than once should not create additional business effects.
In this tutorial, we'll model two separate protections:
- Event deduplication: recognize a delivery that has already been recorded.
- Business idempotency: recognize that a workflow has already performed the requested transition, even when a new event ID is used.
We'll add a transactional outbox so the state change and the intention to perform the next operation are committed together.
Scope: This is a standalone educational example, not CryptoDeepApp's production architecture. A dependency-free in-memory model lets us test the behavior; a separate MongoDB adapter illustrates durable multi-document transactions. We do not connect to blockchain nodes or send real funds. Confirmation events are assumed to come from a trusted verifier that already checked the chain, destination, asset, amount, and confirmation/finality policy.
Two identities, two different jobs
A delivery ID and a business-operation ID are not interchangeable.
| Identifier | Example | Purpose |
|---|---|---|
| Event ID | chain-watcher:event-001 | Detect replay of the same delivered event |
| Deposit reference | btc-mainnet:tx-abc:vout-0 | Identify the specific on-chain transfer |
| Workflow ID | swap-42 | Identify the business process being updated |
| Outbox operation ID | start-exchange:swap-42 | Prevent scheduling the same logical next operation twice |
A transaction hash alone may be insufficient to identify a deposit. Bitcoin transactions can have multiple outputs; other chains may require a log or instruction index. The example treats depositRef as a prevalidated, chain-specific identifier.
Consider this already-observed deposit:
const workflow = { id: 'swap-42', status: 'deposit', depositRef: 'btc-mainnet:tx-abc:vout-0', amountUnits: '12500000', // Integer base units, never a floating-point amount. revision: 3, }; const event = { eventId: 'chain-watcher:event-001', workflowId: 'swap-42', type: 'DEPOSIT_CONFIRMED', depositRef: 'btc-mainnet:tx-abc:vout-0', amountUnits: '12500000', }; The workflow has already reached deposit: funds were observed, but the application's confirmation policy was not yet satisfied. A verified DEPOSIT_CONFIRMED event may move it to exchange.
We deliberately do not allow that event to jump from waiting or overdue straight to exchange. If a late deposit appears, its observation and any required review belong to a separate step.
Why a processed-events check alone is not enough
This code has a race condition:
// Unsafe pseudocode — don't use it as an idempotency guarantee. if (!(await processedEvents.findOne({ eventId }))) { await updateTransaction(); await processedEvents.insertOne({ eventId }); } Two workers can both read "not processed" before either inserts anything. And if the first update succeeds but the process crashes before inserting the receipt, the next delivery repeats the update.
A durable design needs database-enforced uniqueness and an atomic relationship between receiving an event, changing the workflow, and recording the next action.
Build a testable model in plain Node.js
First, we'll make the behavior easy to test without MongoDB. Create idempotent-events.mjs:
// Educational in-memory implementation; NOT durable or distributed. export function validateEvent(event) { if (!event || typeof event !== 'object' || event.type !== 'DEPOSIT_CONFIRMED') { throw new TypeError('Expected a DEPOSIT_CONFIRMED event'); } for (const field of ['eventId', 'workflowId', 'depositRef']) { if (typeof event[field] !== 'string' || !event[field].trim()) { throw new TypeError(`${field} must be a non-empty string`); } } if (typeof event.amountUnits !== 'string' || !/^[1-9]\d*$/.test(event.amountUnits)) { throw new TypeError('amountUnits must be a positive integer string'); } } export function fingerprint(event) { return JSON.stringify([ event.type, event.workflowId, event.depositRef, event.amountUnits, ]); } export function planDepositConfirmation(workflow, event) { if (!workflow) throw new Error('Unknown workflow; reconcile before acknowledgement'); if (workflow.depositRef !== event.depositRef || workflow.amountUnits !== event.amountUnits) { throw new Error('Deposit identity or amount mismatch; needs reconciliation'); } if (workflow.status === 'deposit') return 'advance'; if (['exchange', 'sending', 'success'].includes(workflow.status)) { return 'already-applied'; } throw new Error(`Cannot confirm deposit from status ${workflow.status}`); } export class InMemoryStore { #data; #tail = Promise.resolve(); constructor(workflows = []) { const keyed = new Map(); for (const workflow of workflows) { if (keyed.has(workflow.id)) throw new Error('Duplicate workflow ID'); keyed.set(workflow.id, structuredClone(workflow)); } this.#data = { workflows: keyed, inbox: new Map(), outbox: new Map() }; } async transaction(fn) { // Serializes calls in THIS Node.js process. Not a real database transaction. const previous = this.#tail; let unlock; this.#tail = new Promise((resolve) => { unlock = resolve; }); await previous; try { const draft = structuredClone(this.#data); const result = await fn(draft); this.#data = draft; // Simulated all-or-nothing commit. return result; } finally { unlock(); } } snapshot() { return structuredClone(this.#data); } } export async function processDepositConfirmed(store, event) { validateEvent(event); const identity = fingerprint(event); return store.transaction(async (data) => { const previous = data.inbox.get(event.eventId); if (previous) { if (previous.fingerprint !== identity) { throw new Error('Event ID reused with different payload'); } return { outcome: 'duplicate-delivery' }; } const workflow = data.workflows.get(event.workflowId); const plan = planDepositConfirmation(workflow, event); data.inbox.set(event.eventId, { fingerprint: identity, outcome: plan }); if (plan === 'already-applied') return { outcome: plan }; const operationId = `start-exchange:${event.workflowId}`; if (data.outbox.has(operationId)) { throw new Error('Outbox operation already exists; reconcile conflict'); } data.workflows.set(event.workflowId, { ...workflow, status: 'exchange', revision: workflow.revision + 1, }); data.outbox.set(operationId, { id: operationId, type: 'START_EXCHANGE', workflowId: event.workflowId, status: 'pending', }); return { outcome: 'applied', operationId }; }); } There are three important decisions in this code:
- A repeated event ID returns
duplicate-delivery. If the same ID appears with different meaningful fields, the handler rejects it rather than silently trusting the new payload. - A different event ID describing a confirmation already reflected by the workflow returns
already-applied. This prevents a second business transition. - The workflow state and a uniquely named outbox record are written in the same simulated unit of work. Errors discard the draft instead of leaving half-applied changes.
The model serializes simulated transactions inside one process. It does not provide persistence, cross-process locking, crash recovery, or database isolation. Its purpose is to verify the rules before translating them to real storage.
Test duplicates and concurrent deliveries
Node.js includes a built-in test runner. Create idempotent-events.test.mjs:
import test from 'node:test'; import assert from 'node:assert/strict'; import { InMemoryStore, processDepositConfirmed } from './idempotent-events.mjs'; const workflow = { id: 'swap-42', status: 'deposit', revision: 3, depositRef: 'btc-mainnet:tx-abc:vout-0', amountUnits: '12500000', }; const event = { eventId: 'chain-watcher:event-001', workflowId: 'swap-42', type: 'DEPOSIT_CONFIRMED', depositRef: 'btc-mainnet:tx-abc:vout-0', amountUnits: '12500000', }; test('replayed event does not create another outbox operation', async () => { const store = new InMemoryStore([workflow]); const first = await processDepositConfirmed(store, event); const replay = await processDepositConfirmed(store, event); assert.equal(first.outcome, 'applied'); assert.equal(replay.outcome, 'duplicate-delivery'); assert.equal(store.snapshot().outbox.size, 1); }); test('a new event ID cannot repeat the business transition', async () => { const store = new InMemoryStore([workflow]); await processDepositConfirmed(store, event); const second = await processDepositConfirmed(store, { ...event, eventId: 'chain-watcher:event-002', }); assert.equal(second.outcome, 'already-applied'); assert.equal(store.snapshot().workflows.get('swap-42').revision, 4); }); test('25 concurrent deliveries still schedule one operation', async () => { const store = new InMemoryStore([workflow]); const results = await Promise.all(Array.from({ length: 25 }, (_, i) => processDepositConfirmed(store, { ...event, eventId: `source:event-${i}` }) )); assert.equal(results.filter(r => r.outcome === 'applied').length, 1); assert.equal(store.snapshot().outbox.size, 1); }); Run:
node --test idempotent-events.test.mjs The downloadable example includes 19 unit tests. All passed on Node.js 22.16.0. They also cover missing workflows, mismatched deposit references, changed amounts, invalid states, conflicting event IDs, and simulated rollback. These are in-memory behavioral tests—not MongoDB integration tests or throughput benchmarks.
Move the guarantee into MongoDB
For a real multi-worker service, the uniqueness and atomicity must be enforced by a shared durable store.
A possible MongoDB data model is:
workflows _id: swap-42 status: deposit revision: 3 depositRef: btc-mainnet:tx-abc:vout-0 amountUnits: "12500000" event_inbox _id: chain-watcher:event-001 fingerprint: [canonical event fields] event_outbox _id: start-exchange:swap-42 type: START_EXCHANGE status: pending MongoDB provides a unique index on _id by default. That gives us uniqueness for event receipts and outbox operation IDs. A conditional update can additionally check that the workflow still has the expected status and revision.
The official MongoDB Node.js driver supports withSession() and withTransaction() for operations that need to commit across multiple collections. Here is the core pattern, with validateEvent(), fingerprint(), and planDepositConfirmation() imported from the previous example:
export async function processDepositConfirmedMongo(client, db, event) { validateEvent(event); const fp = fingerprint(event); const inbox = db.collection('event_inbox'); const workflows = db.collection('workflows'); const outbox = db.collection('event_outbox'); try { return await client.withSession(async (session) => session.withTransaction(async () => { await inbox.insertOne({ _id: event.eventId, fingerprint: fp, workflowId: event.workflowId, }, { session }); const workflow = await workflows.findOne({ _id: event.workflowId }, { session }); const plan = planDepositConfirmation(workflow, event); if (plan === 'already-applied') return { outcome: plan }; const result = await workflows.updateOne({ _id: event.workflowId, status: 'deposit', revision: workflow.revision, depositRef: event.depositRef, amountUnits: event.amountUnits, }, { $set: { status: 'exchange' }, $inc: { revision: 1 }, }, { session }); if (result.modifiedCount !== 1) { throw new Error('Concurrent state change; retry or reconcile event'); } const operationId = `start-exchange:${event.workflowId}`; await outbox.insertOne({ _id: operationId, type: 'START_EXCHANGE', workflowId: event.workflowId, status: 'pending', }, { session }); return { outcome: 'applied', operationId }; }, { readConcern: { level: 'snapshot' }, writeConcern: { w: 'majority' }, }) ); } catch (error) { if (error?.code !== 11000) throw error; // A unique-outbox collision is NOT necessarily a duplicate event. const existing = await inbox.findOne({ _id: event.eventId }); if (!existing) throw error; if (existing.fingerprint !== fp) { throw new Error('Event ID reused with different payload'); } return { outcome: 'duplicate-delivery' }; } } This adapter is an architectural illustration; it was not integration-tested against a MongoDB instance. It requires a supported MongoDB deployment for multi-document transactions (typically a replica set or sharded cluster), appropriately provisioned collections, and a compatible official Node.js driver.
The important properties are:
- Insert an inbox receipt with a unique event ID.
- Validate the existing workflow and perform a compare-and-set update using its status and revision.
- Insert a uniquely identified outbox operation in the same transaction.
- Commit them together—or roll them all back.
The driver can retry transaction callbacks for certain transient errors. For this reason, do not call an external exchange, send a notification, or transfer funds inside the transaction callback. Such a side effect could run again even if the database work was rolled back and retried.
An outbox record is not an exactly-once payout
The outbox solves a specific failure window:
Database commit succeeds ↓ Process crashes before publishing next job Because the intended next action is already stored in event_ou...
