The Happy Path Is Lying to You: What Production Systems Taught Me About Idempotency, Reversals, and Financial Truth

There is a point where a software project stops being "an app."

The UI might still look the same.

There are forms, tables, buttons, APIs, and database rows.

But suddenly a bug does not mean:

The page displayed the wrong thing.

It means:

The system charged twice.

Or:

The accounting entry was duplicated.

Or:

Someone reversed a transaction they were not authorized to touch.

Or:

The balance shown to the user was stale, but the system treated it as authoritative.

That changes the way you have to build software.

Over the last few weeks, I have been working on two production systems from very different domains.

One handles business operations such as sales, inventory, accounting, reporting, staff permissions, and branch activity.

The other handles accounts, balances, transfers, pricing, reconciliation, and asynchronous financial events.

Different products.

Same engineering problem.

Once software starts carrying real consequences, the main job is no longer creating features.

The job is preserving truth when the system is under pressure.

This post is about some of the patterns that become important at that point.

CRUD works until your rows start meaning something

A typical application starts simply:

Create Read Update Delete 

You build a few resources.

Then you add authentication.

Then permissions.

Then reports.

Then background jobs.

Then integrations.

Eventually, one of those database rows starts representing something that matters outside the database.

A stock movement affects whether an item can be sold.

A payment affects receivables.

A journal entry changes financial statements.

A transfer affects someone's available money.

At that point, this becomes dangerous:

UPDATE transactions SET amount = 125000 WHERE id = 42; 

Why?

Because you may have just changed history.

That transaction might already have affected:

• an account balance
• a journal
• a report
• a notification
• an external transfer
• a reconciliation process

The system now needs something stronger than CRUD.

It needs invariants.

Invariants are the real feature

For a manual accounting entry, the screen can be simple.

The hard part is not rendering those inputs.

The hard part is guaranteeing that the data can never become financially impossible.

For example:

function validateJournal(lines: JournalLine[]) { if (lines.length < 2) { throw new Error("A journal requires at least two lines"); } let debit = 0n; let credit = 0n; for (const line of lines) { const hasDebit = line.debit > 0n; const hasCredit = line.credit > 0n; if (hasDebit === hasCredit) { throw new Error( "Each line must contain either a debit or a credit" ); } debit += line.debit; credit += line.credit; } if (debit === 0n || debit !== credit) { throw new Error("Journal is not balanced"); } } 

That validation is useful.

It is not enough.

The system still needs to know:

• Is the account active?
• Can this account receive manual postings?
• Is the accounting period open?
• Does the user belong to this branch?
• Has this exact operation already been posted?
• Did the account state change after the form loaded?

That last question is where things get interesting.

Validate again at commit time

Imagine a finance user opens a posting screen at 2:00 PM.

The accounting period is open.

They prepare a long journal.

At 2:05 PM, an administrator closes the accounting period.

At 2:07 PM, the user clicks Post.

If the application only validated the period when the form loaded, you now have a posting inside a closed period.

The rule needs to be enforced inside the transaction that performs the write.

Conceptually:

await db.transaction(async tx => { const period = await tx.accountingPeriods .where({ id: input.periodId }) .forUpdate() .first(); if (!period || period.status !== "OPEN") { throw new Error("Accounting period is closed"); } const accountIds = [ ...new Set(input.lines.map(line => line.accountId)) ].sort(); const accounts = await tx.accounts .whereIn("id", accountIds) .orderBy("id") .forUpdate(); for (const account of accounts) { if (!account.active || !account.manualPostingAllowed) { throw new Error("Account cannot accept this posting"); } } await createJournal(tx, input); }); 

The important idea is not the syntax.

It is this:

A rule that matters financially should be checked as close as possible to the write that makes the operation permanent.

The check-then-insert race

Here is one of my favorite production bugs because the code often looks completely reasonable.

const existing = await db.journals.findFirst({ where: { sourceType, sourceId, sourceEvent } }); if (existing) { return existing; } return db.journals.create({ data: { sourceType, sourceId, sourceEvent } }); 

Looks fine.

Until two requests arrive together.

Request A

Checks.

Nothing exists.

Request B

Checks.

Nothing exists.

Request A

Inserts.

Request B

Also tries to insert.

If the database has no uniqueness constraint, congratulations:

You now have two financial effects.

If the database does have a uniqueness constraint, that is much better.

But Request B may still receive an ugly database exception even though the operation actually succeeded.

The stronger implementation treats the constraint as part of the control flow.

try { return await db.journals.create({ data: { sourceType, sourceId, sourceEvent } }); } catch (error) { if (!isUniqueViolation(error)) { throw error; } const winner = await db.journals.findFirst({ where: { sourceType, sourceId, sourceEvent } }); if (!winner) { throw error; } return winner; } 

And at the database layer:

CREATE UNIQUE INDEX journal_source_identity_unique ON journals ( source_type, source_id, source_event ); 

Now the database decides who wins.

The application knows how to handle the loser.

Important: Application checks can reduce unnecessary conflicts. Database constraints enforce reality.

Idempotency is not a frontend feature

A common mistake is thinking this solves duplicate operations:

button.disabled = true; 

It helps UX.

It does not solve duplication.

A user can still:

• open two browser tabs
• resend an HTTP request
• refresh after a timeout
• trigger a mobile retry
• cause a queue redelivery
• receive the same webhook twice

A consequential operation should have a stable identity.

For example:

{ "operation_id": "01K7H4N9S9P1D9JYC0H7...", "amount_minor": 250000, "currency": "NGN" } 

Then persist that identity.

CREATE UNIQUE INDEX transfer_operation_unique ON transfers(operation_id); 

Retries can now become safe:

const previous = await transfers.findByOperationId(operationId); if (previous) { if (previous.requestHash !== hash(input)) { throw new ConflictError( "Operation ID was already used with different data" ); } return previous; } 

This gives you two useful behaviors:

same key + same payload => return previous result same key + different payload => reject 

That is much stronger than blindly returning:

Already exists 

Reversals are better than rewriting history

Suppose a journal entry is wrong.

The tempting solution:

UPDATE journal_lines SET debit = 0, credit = 100000 WHERE id = 9001; 

Now you have a clean-looking current state.

But you lost historical truth.

A better model:

Journal #123 Debit Cash 100,000 Credit Revenue 100,000 Reversal #456 Credit Cash 100,000 Debit Revenue 100,000 

The original event still exists.

The correction exists separately.

They are linked.

type Journal = { id: string; reversedBy?: string; reverses?: string; }; 

And the reversal command itself should be idempotent.

async function reverseJournal(id: string) { return db.transaction(async tx => { const original = await tx.journals .where({ id }) .forUpdate() .first(); if (!original) { throw new NotFoundError(); } if (original.reversedBy) { return tx.journals.find(original.reversedBy); } const reversal = await createOppositeJournal( tx, original ); await tx.journals.update(original.id, { reversedBy: reversal.id }); return reversal; }); } 

Retrying the reversal should not produce:

reverse reverse reverse reverse 

It should return the one reversal that already exists.

Financial history should be append-heavy

This pattern is useful far beyond journals.

Payments, refunds, voids, settlement corrections, and adjustments can all be represented as related events.

That makes the system much easier to explain later.

Why does this balance equal ₦42,300?

Instead of saying:

> Because that is the value currently stored in the balance column.

The system can explain:

Opening balance +100,000 Purchase -35,000 Transfer -20,000 Fee -1,500 Refund +5,000 Adjustment -6,200 -------------------------------- Current balance 42,300 

That explainability becomes extremely important once support, finance teams, auditors, and users start asking questions.

Your browser should not decide how much money moves

Suppose a transfer has:

Amount: 10,000 Network fee: 100 Service fee: 50 Total debit: 10,150 

Do not let the client invent the final debit.

This is weak:

const total = amount + networkFee + serviceFee; await api.post("/transfer", { amount, total }); 

Why trust total?

The client is not authoritative.

A stronger flow looks like this.

Step 1: Create a quote

POST /transfer-quotes Content-Type: application/json 

{ "amount_minor": 1000000, "destination_id": "acct_123" } 

Server response:

{ "quote_id": "quote_abc", "amount_minor": 1000000, "fee_minor": 15000, "total_debit_minor": 1015000, "expires_at": "2026-10-07T20:30:00Z" } 

Step 2: Display the quote

The UI renders what the server calculated.

Step 3: Authorize that exact debit

POST /transfers Content-Type: application/json 

{ "quote_id": "quote_abc", "authorization": { "type": "pin", "proof": "..." } } 

The backend verifies the quote itself.

Not a client-provided total.

The server should always be able to explain exactly what amount the user authorized.

Minor units deserve boring code

Money bugs are often surprisingly boring.

Consider:

const minimum = 50000; 

What does that mean?

₦50,000?

₦500?

50,000 kobo?

Without a convention, the code is ambiguous.

I prefer names like:

const minimumMinor = 50_000n; 

Formatting happens only at the presentation edge.

function formatMoney(minor: bigint): string { return new Intl.NumberFormat("en-NG", { style: "currency", currency: "NGN" }).format(Number(minor) / 100); } 

Arithmetic stays integer-based:

const totalDebitMinor = amountMinor + serviceFeeMinor + networkFeeMinor; 

Not:

const total = 100.1 + 20.2; 

You do not want floating-point surprises anywhere near money.

Reconciliation is not cleanup

External systems introduce an uncomfortable state.

You send a transfer request.

Then:

Your request times out. 

What happened?

You do not know.

That is different from failure.

This can be dangerous:

try { await sendTransfer(input); } catch { transfer.status = "FAILED"; } 

The external system may have accepted the transaction before the connection disappeared.

Now your local system says:

FAILED 

while the external system later says:

SUCCESS 

Then the user retries.

The original transfer eventually completes.

The retry completes too.

A better state model includes uncertainty.

type TransferStatus = | "PENDING" | "SUBMITTED" | "UNKNOWN" | "CONFIRMED" | "FAILED"; 

After an ambiguous timeout:

transfer.status = "UNKNOWN"; 

Then reconciliation decides the final state.

const external = await fetchTransferStatus( transfer.externalReference ); switch (external.status) { case "successful": await settleExactlyOnce(transfer, external); break; case "failed": await markFailed(transfer); break; default: await scheduleAnotherCheck(transfer); } 

UNKNOWN is a valid state.

Pretending uncertainty does not exist is not.

Webhooks can arrive twice

Actually, assume they will.

This is dangerous:

app.post("/webhook", async req => { await creditAccount(req.body); }); 

A webhook sender may retry because:

• your server returned 500
• the connection closed too early
• they did not receive your acknowledgement
• delivery is intentionally at-least-once

A safer approach:

await db.transaction(async tx => { const exists = await tx.webhookEvents.findUnique({ where: { eventId: event.id } }); if (exists) { return; } await applyFinancialEffect(tx, event); await tx.webhookEvents.create({ data: { eventId: event.id, processedAt: new Date() } }); }); 

Plus a database constraint:

CREATE UNIQUE INDEX webhook_event_unique ON webhook_events(event_id); 

Again, the database is part of your correctness model.

Stale data can be worse than no data

One particularly interesting class of bugs happens when the application already has a value.

For example:

{ "balance_minor": 0 } 

The application sees that and assumes:

Balance is zero. 

But that value may have been captured hours ago.

Meanwhile, a more recent confirmed event says funds exist.

The local value is not missing.

It is stale.

That can be more dangerous because it looks authoritative.

A better representation carries metadata.

type BalanceSnapshot = { amountMinor: bigint; observedAt: Date; source: "ledger" | "external-confirmation" | "cache"; }; 

Now the application can reason about evidence.

function chooseBalance( a: BalanceSnapshot, b: BalanceSnapshot ) { if ( a.source === "external-confirmation" && b.source === "cache" ) { return a; } return a.observedAt > b.observedAt ? a : b; } 

The real implementation may be more complicated.

The important lesson is simpler:

Presence does not imply authority.

Authorization belongs inside the domain

A UI might hide a reversal button.

That is not security.

This:

{user.canReverse && ( <button>Reverse</button> )} 

is purely presentation.

The real check must happen server-side.

async function reverseEntry(user, entryId) { const entry = await entries.find(entryId); if (!user.permissions.includes("journal.reverse")) { throw new ForbiddenError(); } if (entry.branchId !== user.branchId) { throw new ForbiddenError(); } return reversalService.reverse(entry); } 

Notice that there are two separate questions:

Can this user reverse entries? Can this user reverse this specific entry? 

The second one is where many authorization bugs appear.

Lock rows in a deterministic order

Concurrency can also create deadlocks.

Suppose one transaction locks:

Account 8 then Account 12 

while another locks:

Account 12 then Account 8 

Both can wait on each other.

One useful mitigation is deterministic ordering.

const ids = [...new Set(accountIds)] .sort((a, b) => a.localeCompare(b)); const accounts = await tx.accounts .whereIn("id", ids) .orderBy("id") .forUpdate(); 

Every transaction tries to acquire the same resources in the same order.

Not every deadlock disappears.

But you remove an entire class of avoidable ones.

UI correc...