A payment gateway has a 400 ms hiccup. You have 10,000 clients in flight.
Every one of them times out in the same instant, catches the same error, and enters the same retry loop. Your backoff is 1s, 2s, 4s, so all 10,000 requests come back at t=1000ms. Together. The gateway, which was just starting to recover, receives the load it had before — in a single burst — and falls over again.
You didn't add retries to make things worse. You added them because the docs said to. But a retry loop without jitter isn't error handling. It's a traffic amplifier with a delay.
Three things fix this, and they fix different problems: jitter breaks the correlation between clients, a circuit breaker stops sending traffic to something already dead, and an idempotency key makes the retries safe in the first place. Most codebases have one of the three.
Everything below runs — no dependencies, Node 24+, tests included.
Failures are correlated because your clients are synchronized
A transient failure — a dropped connection, a 429, a 504 from the acquirer — is not random noise. It hits every client at the same time, because they're all talking to the same degraded dependency.
Those clients then compute a backoff delay from the same formula, with the same attempt number, starting from the same instant:
delay = base * 2^(attempt - 1) Deterministic input, deterministic output. You didn't build a backoff. You built a scheduled stampede, and you scheduled it for every client simultaneously.
That's why gateway outages so often have a second, worse peak a few seconds after the first one. The retries cause it, not the original fault.
One more number worth internalizing: failure probability compounds across hops. Ten hops at 99.9% each gives 0.999^10 ≈ 99.0% end to end. At 10,000 requests per second, that "three nines" infrastructure is dropping 100 requests every second. Retries aren't optional at that volume. Naive retries are fatal.
Fix 1: full jitter
The delay stays exponential. The offset becomes random, so each client lands at a different point on the timeline.
export const sleep = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms)); /** Uniform random in [0, min(maxDelay, base * 2^(attempt-1))) */ export function fullJitter(attempt: number, baseDelayMs: number, maxDelayMs: number): number { const ceiling = Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1)); return Math.random() * ceiling; } export type RetryPolicy = { maxAttempts: number; baseDelayMs: number; maxDelayMs: number; isRetryable?: (error: unknown) => boolean; onRetry?: (info: { attempt: number; delayMs: number; error: unknown }) => void; }; export function isTransientError(error: unknown): boolean { const e = error as { status?: number; code?: string }; if (typeof e.status === 'number') { return e.status === 408 || e.status === 429 || e.status >= 500; } return e.code === 'ECONNRESET' || e.code === 'ETIMEDOUT' || e.code === 'EAI_AGAIN'; } export async function withRetry<T>(fn: () => Promise<T>, policy: RetryPolicy): Promise<T> { const isRetryable = policy.isRetryable ?? isTransientError; let lastError: unknown; for (let attempt = 1; attempt <= policy.maxAttempts; attempt++) { try { return await fn(); } catch (error) { lastError = error; // Never retry a 400. A declined card will be declined forever. if (attempt === policy.maxAttempts || !isRetryable(error)) throw error; const delayMs = fullJitter(attempt, policy.baseDelayMs, policy.maxDelayMs); policy.onRetry?.({ attempt, delayMs, error }); await sleep(delayMs); } } throw lastError; } Two details are load-bearing:
-
await sleep(...)inside anasyncfunction does not block the event loop. A busy-wait loop does, and it stalls every other request in the process. -
isRetryableis a hard boundary, not a nicety. Retrying acard_declinedburns quota, delays the user, and can trip fraud heuristics.
What that actually buys you
Same 10,000 clients, retrying once. Here's the whole simulation:
function histogram(delays: number[], bucketMs = 100): { at: number; count: number }[] { const buckets = new Map<number, number>(); for (const d of delays) { const b = Math.floor(d / bucketMs) * bucketMs; buckets.set(b, (buckets.get(b) ?? 0) + 1); } return [...buckets.entries()].sort((a, b) => a[0] - b[0]).map(([at, count]) => ({ at, count })); } /** Renders on a scale shared by both runs, so the blocks are comparable. */ function render(rows: { at: number; count: number }[], scaleMax: number, width = 50): string { return rows .map(({ at, count }) => { const bar = '█'.repeat(Math.max(1, Math.round((count / scaleMax) * width))); return ` ${String(at).padStart(6)}ms ${String(count).padStart(7)} ${bar}`; }) .join('\n'); } const CLIENTS = 10_000; const fixed = Array.from({ length: CLIENTS }, () => 1_000); // naive: everybody at 1000ms const jittered = Array.from({ length: CLIENTS }, () => fullJitter(1, 1_000, 30_000)); console.log(' naive — fixed 1000ms delay'); console.log(render(histogram(fixed), CLIENTS)); console.log('\n full jitter — random in [0, 1000ms)'); console.log(render(histogram(jittered), CLIENTS)); Real output, one run:
naive — fixed 1000ms delay when clients 1000ms 10000 ██████████████████████████████████████████████████ full jitter — random in [0, 1000ms) when clients 0ms 1003 █████ 100ms 970 █████ 200ms 980 █████ 300ms 1000 █████ 400ms 954 █████ 500ms 1048 █████ 600ms 1044 █████ 700ms 984 █████ 800ms 991 █████ 900ms 1026 █████ summary strategy peak/100ms retries fixed 1000ms 10000 10000 full jitter 1048 10000 9.5x lower peak, same number of retries Same 10,000 retries, same total load: the peak per 100 ms window drops from 10,000 to 1,048 for four lines of arithmetic. (Your numbers will differ by a few percent; it's Math.random().)
Which jitter? Measured over 20,000 samples at attempt 4 (base 1000 ms, cap 30 s):
| Variant | min | median | max |
|---|---|---|---|
| full | 1 ms | 3,973 ms | 7,999 ms |
| equal | 4,000 ms | 5,988 ms | 8,000 ms |
| decorrelated | 1,001 ms | 12,411 ms | 24,000 ms |
Full jitter has no floor — a few clients retry almost immediately, which feels wrong but is exactly what flattens the curve. If you need a guaranteed minimum gap, use equal jitter (ceiling/2 + random(0, ceiling/2)). Decorrelated jitter derives the next delay from the previous one (min(cap, random(base, previous * 3))), so it drifts well above the exponential ceiling and recovers more slowly — useful when you want successive failures to back off harder than a fixed schedule would.
The simulation, the breaker, and the tests below are in JSTypescriptProgrammingSeries / 30_fintech / 14_chapter / retry-sim. Two commands, no install: node src/sim.ts and node --test "src/*.test.ts".
Fix 2: a circuit breaker that fails fast
Jitter handles failures that resolve in milliseconds. It does nothing for a provider that's down for 40 minutes. In that case every worker exhausts its retries, waits out its backoff, and keeps knocking. You get slow failures, saturated connection pools, and a queue backing up into your own API.
A circuit breaker is a stateful wrapper that stops calling a dependency it has already proven broken.
export class CircuitOpenError extends Error { constructor(message = 'Circuit is open: downstream is failing fast') { super(message); this.name = 'CircuitOpenError'; } } export type BreakerState = 'closed' | 'open' | 'half-open'; export type BreakerOptions = { windowSize: number; // outcomes to consider failureRateThreshold: number; // 0..1 openDurationMs: number; // cooldown before probing probesToClose: number; // consecutive successes to close }; export class CircuitBreaker { private state: BreakerState = 'closed'; private window: boolean[] = []; // true = success private openedAt = 0; private probeSuccesses = 0; private readonly opts: BreakerOptions; constructor(opts: BreakerOptions) { this.opts = opts; } get currentState(): BreakerState { if (this.state === 'open' && Date.now() - this.openedAt >= this.opts.openDurationMs) { this.state = 'half-open'; this.probeSuccesses = 0; } return this.state; } async execute<T>(fn: () => Promise<T>, fallback?: () => Promise<T>): Promise<T> { if (this.currentState === 'open') { if (fallback) return fallback(); throw new CircuitOpenError(); } try { const result = await fn(); this.record(true); return result; } catch (error) { this.record(false); throw error; } } private record(success: boolean): void { if (this.state === 'half-open') { if (!success) return this.trip(); if (++this.probeSuccesses >= this.opts.probesToClose) this.reset(); return; } this.window.push(success); if (this.window.length > this.opts.windowSize) this.window.shift(); // Only judge once the window is full, otherwise the first error trips it. if (this.window.length === this.opts.windowSize) { const failureRate = this.window.filter((ok) => !ok).length / this.window.length; if (failureRate >= this.opts.failureRateThreshold) this.trip(); } } private trip(): void { this.state = 'open'; this.openedAt = Date.now(); this.window = []; this.probeSuccesses = 0; } private reset(): void { this.state = 'closed'; this.window = []; this.probeSuccesses = 0; } } The state machine is the part reviewers get wrong:
- Closed — traffic flows. It keeps a rolling window of the last N outcomes and trips when the rate crosses the threshold. A rate, not a count: five failures out of five requests is a dead provider, five out of 10,000 is a Tuesday.
- Open —
CircuitOpenErrorimmediately, zero sockets opened. That's the entire point: you fail in microseconds instead of after 30 seconds of timeouts. - Half-open — after the cooldown, one request is allowed through as a probe. Success closes the circuit; a single failure re-trips it. Don't let 500 requests be the probe, or the recovering provider gets the herd again.
If you run more than one pod, this in-memory breaker is wrong. Each replica learns about the outage independently, so you get 12 partial breakers instead of one. Share the state in Redis, or use a service mesh that does it for you.
Fix 3: idempotency — the part that makes retries legal
Retries are only safe if the gateway can recognize a duplicate. That's what the idempotency key is for: a client-generated, deterministic identifier that travels with every attempt of the same logical charge.
Note the word deterministic. crypto.randomUUID() inside your retry loop generates a new key on every attempt and defeats the whole mechanism. The key belongs to the charge, generated once at the API boundary — the tested pipeline below asserts that the key never changes between attempts.
import { isTransientError } from './retry.ts'; type IdempotencyRecord<T> = | { status: 'in-flight'; startedAt: number } | { status: 'done'; result: T } | { status: 'unknown'; error: string; startedAt: number }; export class IdempotencyStore<T> { private readonly records = new Map<string, IdempotencyRecord<T>>(); async run(key: string, fn: () => Promise<T>): Promise<T> { const existing = this.records.get(key); if (existing?.status === 'done') return existing.result; if (existing?.status === 'in-flight') { throw new Error(`Duplicate request for ${key} is already in flight`); } if (existing?.status === 'unknown') { throw new Error(`Outcome for ${key} is unknown — reconcile before retrying`); } this.records.set(key, { status: 'in-flight', startedAt: Date.now() }); try { const result = await fn(); this.records.set(key, { status: 'done', result }); return result; } catch (error) { // The distinction most implementations miss: if (isTransientError(error)) { // We do NOT know whether the money moved. Keep the key, block blind retries. this.records.set(key, { status: 'unknown', error: String(error), startedAt: Date.now(), }); } else { // Definitive rejection (400, declined). The charge never happened. this.records.delete(key); } throw error; } } /** Exposed so the tests can inspect the stored outcome. */ peek(key: string): IdempotencyRecord<T> | undefined { return this.records.get(key); } } That unknown state is the whole reason reconciliation exists as a subsystem. A timeout is not a failure — it's an absence of information. If a charge times out after the gateway debited the account, and you delete the key and let the user retry with a fresh one, you have charged them twice. The ledger's zero-sum invariant doesn't save you: both entries are internally balanced, the customer is just out $49.99.
The correct move is to park the charge in unknown, keep the key, and let a reconciliation job match it against the provider's settlement feed later. When the outcome is unknown, the answer is not another retry.
Putting the three layers together
Order matters: idempotency outside, breaker in the middle, retry inside.
import type { CircuitBreaker } from './breaker.ts'; import type { IdempotencyStore } from './idempotency.ts'; import { withRetry, type RetryPolicy } from './retry.ts'; export type Charge = { idempotencyKey: string; amountCents: number; currency: string; }; export type ChargeResult = { reference: string; amountCents: number; }; export type ChargeDeps = { store: IdempotencyStore<ChargeResult>; breaker: CircuitBreaker; gateway: (charge: Charge) => Promise<ChargeResult>; policy: RetryPolicy; }; export async functi...
