Idempotency Keys - The Double-Charge Trap

The Problem

You click Pay. The server charges your card, but the response is lost on the way back (network drop, timeout, tab freeze). The UI thinks it failed, so you click Pay again and get charged twice.

A timeout means "I don't know what happened", not "payment failed".

The Idea

Idempotent = doing the same thing twice has the same effect as doing it once.

An idempotency key is a unique ID for one payment attempt. The UI sends it with the request. If the same key arrives again, the backend says "I've already handled this one" and returns the saved result instead of charging again.

• UI's job: create the key once, keep it, and resend the same key on every retry.
• Backend's job: remember each key and its result, so repeats are not executed again.

Full Flow Diagram

Example: user pays USD 50 for order O100.

sequenceDiagram actor User participant UI as Browser UI participant SS as sessionStorage participant API as Backend participant DB as Database participant PSP as Payment Provider User->>UI: Click "Pay" UI->>SS: Key for O100? SS-->>UI: None UI->>UI: key = crypto.randomUUID() → K1 UI->>SS: Save payment-key:O100 = K1 UI->>UI: Disable Pay button, show "Processing..." UI->>API: POST /orders/O100/pay<br/>Idempotency-Key: K1 API->>DB: Insert (user, K1, O100, processing) DB-->>API: Inserted (K1 is new) API->>PSP: Charge USD 50 (provider key derived from K1) PSP-->>API: Success, payment P900 API->>DB: Update K1 → succeeded, result P900 API--xUI: 200 OK (response LOST) UI->>UI: Timeout → show "Checking payment status..." UI->>SS: Read key for O100 SS-->>UI: K1 (same key) UI->>API: POST /orders/O100/pay<br/>Idempotency-Key: K1 (retry) API->>DB: Find K1 DB-->>API: K1 = succeeded, P900 Note over API,PSP: Provider NOT called again API-->>UI: 200 OK, payment P900 (saved result) UI->>SS: Remove payment-key:O100 UI-->>User: Show receipt ✅ (charged once)

Step-by-Step

Step 1: User clicks "Pay" — UI gets a key

The UI first checks whether this order already has a key (from an earlier attempt or before a refresh).

• Key found → reuse it. This is a retry of the same payment.
• No key → generate a new one with crypto.randomUUID().

payment-key:O100 → K1 (e.g. "6f13d95a-e6c1-4bf2-9459-99b502098dc1") 

Use crypto.randomUUID() — it is random and practically never collides. Do not use Date.now(), Math.random(), or the amount; two different payments could produce the same value.

Step 2: Save the key BEFORE sending the request

sessionStorage.setItem("payment-key:O100", key); 

Save first, then send. If the page crashes or reloads mid-request, the key is still there and the next attempt reuses it.

Where to store on the UI:

Also disable the Pay button while the request is in flight. This stops accidental double clicks, but it is only a UX helper; the key is the real protection.

Step 3: Send the payment request with the key

POST /orders/O100/pay Idempotency-Key: K1 Cookie: session=... (user is logged in) 

The UI sends only the order ID and key. The backend loads the amount from the order itself; never trust a price sent by the browser.

Step 4: Backend checks the key

Why does the backend store the key? Because the UI can't know whether the first request was charged — only the backend can remember.

The backend saves one row per key:

When a request arrives:

The insert must be atomic (a unique constraint on user + key). If two identical requests arrive at the same moment, only one insert succeeds, so only one charge happens.

Step 5: Backend charges and saves the result

The backend calls the payment provider and passes its own provider idempotency key derived from K1, saved before the call. If the backend crashes after the provider charged but before saving the result, a retry reuses that provider key, so the provider returns the original charge instead of creating a new one.

Then it updates the row to succeeded with result P900 and responds.

Step 6: UI handles the response

The key rule: only remove the key when you know the final outcome.

Step 7: Retry after timeout

The UI retries with the same K1 (a few times, with increasing delays like 1s, 2s, 4s).

Two possible cases, both safe:

• First request never reached the backend → K1 is new → backend charges once now.
• First request was already charged → K1 = succeeded → backend returns P900, no second charge.

That's the whole point: the UI doesn't need to know which case happened.

Step 8: User refreshes the page

flowchart TD A[Page loads / user clicks Pay] --> B{Key for O100 in sessionStorage?} B -- Yes --> C[Reuse the same key] B -- No --> D[Generate new key with crypto.randomUUID] D --> E[Save key in sessionStorage] C --> F[POST /orders/O100/pay with key] E --> F F --> G{Response} G -- Success --> H[Show receipt, remove key] G -- Declined --> I[Show error, remove key] G -- In progress / timeout / 5xx --> J[Show 'Checking...', keep key, retry later] J --> F

After refresh, the UI finds K1 in sessionStorage and sends the request with K1 again. The backend returns the saved result or "in progress" — never a second charge.

UI Code Example

Illustrative only; status codes depend on your API.

function getOrCreateKey(orderId) { const name = `payment-key:${orderId}`; let key = sessionStorage.getItem(name); if (!key) { key = crypto.randomUUID(); sessionStorage.setItem(name, key); // save BEFORE sending } return key; } async function pay(orderId) { const key = getOrCreateKey(orderId); try { const res = await fetch(`/orders/${orderId}/pay`, { method: "POST", headers: { "Idempotency-Key": key }, credentials: "include", signal: AbortSignal.timeout(15000), }); if (res.ok) { sessionStorage.removeItem(`payment-key:${orderId}`); return { status: "paid", payment: await res.json() }; } if (res.status === 402) { // Clear decline: not charged, a new attempt may use a new key sessionStorage.removeItem(`payment-key:${orderId}`); return { status: "declined" }; } // 409 in progress, 5xx, etc.: outcome unknown, keep the key return { status: "unknown" }; } catch { // Timeout or network error: outcome unknown, keep the key return { status: "unknown" }; } } 

On "unknown", show "Checking payment status..." and call pay(orderId) again after a delay — it automatically reuses the same key.

sessionStorage can throw (e.g. storage full or blocked). In that case, keep the key in memory for this page and avoid letting the user start a fresh payment while one is unresolved.

When Is a New Key Needed?

Simple rule: same intention → same key. New intention → new key.

Security Checklist

• Store only the key in sessionStorage. Never store card number, CVV, or tokens.
• The key is not a password. The backend must still check that the logged-in user owns order O100.
• The backend scopes keys per user, so one user's key can't read another user's payment.
• Never trust amount or status from the browser; the backend decides both.
• Clear payment keys on logout.

What If sessionStorage Is Lost?

New tab, cleared storage, or another device → the UI generates a new key for the same order. The idempotency key can't catch this, so the backend needs one more rule:

An order that is already paid, or has a payment in progress, cannot start another charge.

Keys stop duplicate requests. Order rules stop duplicate payments for the same order. You need both.

System Design: Key Creation, Change Detection, and Cancellation

So far, one key = one payment. Real checkouts are messier: the user changes the cart, applies a coupon, or switches cards while a payment is unclear. This section designs a setup that answers three questions:

1. When should the UI create a key?
2. How do UI and backend detect that the payment details changed?
3. How is an old attempt safely cancelled so a new key can be used?

Core idea: a key belongs to a checkout snapshot

A key represents "pay for exactly this checkout". The checkout is identified by a small snapshot:

Same snapshot → same intention → reuse the key Different snapshot → new intention → old attempt must be finished/cancelled, then a new key 

orderVersion is the important part: the UI doesn't have to compare cart items itself. The backend increases the version whenever anything affecting the price changes.

Architecture

flowchart LR subgraph Browser UI[Checkout UI] KM[Key Manager] SS[(sessionStorage)] UI --> KM KM <--> SS end KM -->|Pay / Cancel + Idempotency-Key| GW[API Gateway<br/>auth, rate limit] GW --> PS[Payment Service<br/>idempotency check] GW --> OS[Order Service<br/>cart, orderVersion, lock] PS <--> IDB[(Idempotency Store<br/>key, fingerprint, status, result)] PS <--> OS PS -->|charge with provider key| PSP[Payment Provider] PSP -->|webhook: final result| WH[Webhook Handler] WH --> PS RJ[Reconciliation Job] -->|resolve stuck attempts| PS CJ[Cleanup Job] -->|expire old finished keys| IDB

Data model

Idempotency store (one row per key):

Order (relevant columns):

The backend calculates the fingerprint from its own data (loaded order + amount), not from values the browser claims.

Attempt lifecycle

stateDiagram-v2 [*] --> processing: Pay request accepted (new key) [*] --> cancelled: Cancel arrives before Pay processing --> succeeded: Provider approved processing --> declined: Provider declined processing --> processing: Provider timeout, waiting for webhook / reconciliation processing --> failed: Reconciliation found no charge succeeded --> [*] declined --> [*] failed --> [*] cancelled --> [*]

• Cancel is only possible before charging starts. Once processing, the UI must wait for the final result.
• A succeeded payment cannot be "cancelled". Undoing it is a refund, a separate operation with its own key.

When should the UI create a key?

Rule: create the key at the moment of commitment (the Pay click), never earlier.

Backend rules on POST /orders/O100/pay

The request now carries the snapshot:

POST /orders/O100/pay Idempotency-Key: K1 Content-Type: application/json { "orderVersion": 3, "paymentMethodId": "pm_123" } 

The backend checks in this order:

1. Authenticate and confirm the user owns O100.
2. Look up the key. If it already exists:
   • Fingerprint differs → 422 KEY_REUSED_WITH_DIFFERENT_DETAILS. Nothing is charged.
   • succeeded / declined / failed → replay the saved result.
   • processing → 409 IN_PROGRESS.
   • cancelled → 409 ATTEMPT_CANCELLED. This blocks a late, delayed request from charging after the user cancelled.
3. New key. In one database transaction:
   • Request orderVersion ≠ current version → 409 ORDER_CHANGED (UI must refresh the checkout).
   • Order is paid or already has an active_attempt → 409 PAYMENT_ALREADY_ACTIVE.
   • Otherwise insert the key as processing and set order.active_attempt = K1.
4. Charge using the saved provider key, then save the result:
   • Success → key succeeded, order paid.
   • Decline → key declined, clear active_attempt.
   • Timeout → stays processing; webhook or reconciliation finishes it later.

Key lookup comes before the version check, so a retry of an already-paid attempt still gets its saved receipt.

Cart edits while a payment is active

The Order Service rejects cart changes while active_attempt is set:

PATCH /orders/O100/cart → 409 PAYMENT_IN_PROGRESS 

The UI also disables editing during payment. The backend check is what actually guarantees it (another tab or device could try to edit).

Cancel endpoint

POST /orders/O100/payment-attempts/cancel Content-Type: application/json { "key": "K1" } 

The key goes in the body, not the URL, so it doesn't end up in access logs or browser history.

Why insert a tombstone for an unknown key? Because the original Pay request might be stuck in the network and arrive later. The tombstone uses the same unique (user_id, key) slot, so whichever arrives first wins, and the late Pay is rejected instead of charging.

Scenario: user changes the cart after a timeout

sequenceDiagram actor User participant UI as Checkout UI participant SS as sessionStorage participant API as Backend User->>UI: Click Pay (cart v3, card pm_123) UI->>SS: Save {key: K1, snapshot: "v3:pm_123"} UI->>API: POST /pay, key K1, v3 API--xUI: Timeout (outcome unknown) UI-->>User: "Checking payment status..." User->>UI: "I want to change my cart" UI->>API: Cancel K1 alt K1 never reached backend API->>API: Insert K1 as cancelled (tombstone) API-->>UI: cancelled UI->>SS: Remove K1 UI-->>User: Cart unlocked User->>UI: Edit cart → backend bumps to v4 User->>UI: Click Pay UI->>SS: Save {key: K2, snapshot: "v4:pm_123"} UI->>API: POST /pay, key K2, v4 API-->>UI: Success ✅ Note over API: If delayed K1 arrives now → cancelled, no charge else K1 is processing API-->>UI: processing UI-->>User: "Payment in progress, please wait" else K1 already succeeded API-->>UI: succeeded, P900 UI->>SS: Remove K1 UI-->>User: Show receipt (refund if they still want changes) end

Scenario: backend crashes mid-payment (K1 stuck in processing)

The key reached the backend and was saved as processing, but the status was never updated. Now the user retries with K1. What happens?

Why K1 gets stuck:

1. UI sends Pay with K1 2. Backend saves K1 = processing (+ provider key PK1) ✅ saved 3. Backend calls the provider 4. One of these happens: a) Backend crashes before or during the provider call → maybe NOT charged b) Provider charged, backend crashed before saving → charged c) Provider is slow or timed out → unknown 5. K1 is still "processing" in the DB 

The backend itself doesn't know whether money moved.

What the retry does:

UI -> Backend: Pay O100 with K1 (retry) Backend -> DB: Find K1 DB -> Backend: K1 = processing Backend -> UI: 409 "Payment in progress" (provider NOT called) UI: shows "Checking payment status...", asks again in a few seconds 

Rule: a retry request never starts a charge for a key that is already processing. It could be case (b) or (c), where money already moved or is moving.

Who fixes the stuck status?

1. Provider webhook (usually first). When the provider finishes the charge, it notifies the backend's webhook handler, which updates K1 to succeeded or declined.
2. Reconciliation job (safety net). Every minute or so, it finds attempts stuck in processing for longer than a threshold (e.g. 2 minutes) and asks the provider about them using PK1, which was saved with K1 before calling the provider.

Retrying with the same PK1 is safe: if the charge already exists, the provider returns it instead of creating a second one.

If the provider can't be reached, or offers no idempotency or status lookup, K1 stays processing and the order stays locked. It is flagged for manual review, never charged blindly.

sequenceDiagram participant UI as Checkout UI participant API as Backend participant DB as Database participant PSP as Payment Provider participant RJ as Webhook / Reconciliation UI->>API: Pay with K1 API->>DB: K1 = processing, save PK1 API->>PSP: Charge with PK1 PSP->>PSP: Charge succeeds Note over API: 💥 Backend crashes. K1 still processing API--xUI: No response UI->>API: Retry with K1 API->>DB: Find K1 → processing API-->>UI: 409 In progress (no charge) UI->>UI: "Checking payment status..." RJ->>PSP: Status of PK1? PSP-->>RJ: Succeeded, P900 RJ->>DB: K1 = succeeded, P900, order paid UI->>API: Retry with K1 API->>DB: Find K1 → succeeded API-->>UI: 200 Paid, P900 (saved result) UI->>UI: Show receipt, remove K1

Timeline:

12:00:00 UI: Pay with K1 12:00:00 Backend: K1 = processing, PK1 saved 12:00:01 Backend calls provider with PK1 → provider charges 12:00:01 Backend CRASHES (K1 still processing) 12:00:15 UI timeout → "Checking payment status..." 12:00:20 UI retries K1 → 409 in progress (no charge) 12:00:30 UI retries K1 → 409 in progress (no charge) 12:00:40 Webhook arrives: PK1 succeeded → K1 = succeeded, P900 12:00:45 UI retries K1 → 200 "Paid, P900" (saved result) UI shows receipt, removes K1 from sessionStorage Customer charged ONCE ✅ 

If the webhook never arrives, the reconciliation job performs the same update around 12:02.

What the user sees:

• While processing: "Payment in progress, please don't close this page." Pay button and cart stay locked.
• After a long wait (e.g. 1-2 minutes): "We're confirming your payment. We'll email you when it's done." The UI can stop polling; the backend still finishes the work.
• On refresh: the UI reads K1 from sessionStorage and asks again, getting the latest status.
• If the attempt ends as failed (not charged), the backend replays it like a decline (e.g. 402 in the UI example), so the UI removes K1 and the next Pay creates a new key.

Why it's safe:

Key Manager code (UI)

Illustrative; extends the earlier example with a snapshot and cancel.

const nameFor = (orderId) => `payment-attempt:${orderId}`; const snapshotOf = (c) => `${c.orderVersion}:${c.paymentMethodId}`; // Called only when the user clicks Pay function getOrCreateAttempt(checkout) { const name = nameFor(checkout.orderId); const saved = JSON.parse(sessionStorage.getItem(name) ?? "null"); const snapshot = snapshotOf(checkout); if (saved && saved.snapshot === snapshot) { return saved; // same intention: reuse key } if (saved) { // Details changed but the old attempt is unresolved throw new Error("Resolve the previous payment attempt first"); } const attempt = { key: crypto.randomUUID(), snapshot }; sessionStorage.setItem(name, JSON.stringify(attempt)); // save BEFORE sending return attempt; } // Called when the user wants to change details during an unclear payment async function cancelAttempt(orderId) { const saved = JSON.parse(sessionStorage.getItem(nameFor(orderId)) ?? "null"); if (!saved) return { status: "none" }; try { const res = await fetch(`/orders/${orderId}/payment-attempts/cancel`, { method: "POST", headers: { "Content-Type": "application/json" }, credentials: "include", body: JSON.stringify({ key: saved.key }), }); if (!res.ok) return { status: "unknown" }; // keep key, try later const { status } = await res.json(); if (["cancelled", "declined", "failed", "succeeded"].includes(status)) { sessionStorage.removeItem(nameFor(orderId)); // final: key no longer needed } return { status }; // "processing" → keep key and wait } catch { return { status: "unknown" }; // network error: keep key } } 

UI reactions:

• cancelled / declined → unlock the cart; the next Pay creates a new key.
• processing / unknown → keep the cart locked, show "Payment in progress".
• succeeded → show the receipt; offer a refund if the user still wants changes.
• Pay returns 409 ORDER_CHANGED → the order changed elsewhere (another tab). Reload the checkout, show the new total, and let the user confirm. The old key was never accepted, so remove it; the confirmation Pay creates a new key.

Design summary

Summary

Click Pay → UI: reuse key, or create one with crypto.randomUUID() → UI: save key in sessionStorage → UI: POST with Idempotency-Key → Backend: new key? charge and save result. Known key? return saved result → UI: success/decline → remove key | unknown → keep key and retry 

More Details:

Get all articles related to system design
Hashtag: SystemDesignWithZeeshanAli

systemdesignwithzeeshanali

Git: https://github.com/ZeeshanAli-0704/front-end-system-design

⬆ Back to Top