Fifty POSTs, one per second, each carrying its own idempotency key. At fire 21, I killed the receiver.
That was the whole experiment. A dispatcher on my laptop, a receiver on the same laptop, and a kill signal in the middle. I wanted a number, not an opinion: when the thing on the other end dies mid-run, how much of a cron job's work actually arrives?
The dispatcher is a small Node script. It fires 50 POSTs to localhost, one a second, keys evt-001 through evt-050. Every POST carries an Idempotency-Key header with its key. If an attempt fails, it waits 500 milliseconds and tries once more, then moves on. It never asks why anything failed. It fires, it logs, it moves on.
The receiver is even smaller. It takes each POST, writes one JSON line to a log file, and answers 200. In phase 1 it ignores the idempotency key completely. It has no memory, no database, no queue. It is the dumbest possible webhook endpoint, which is the point. Most receivers I have actually built were not much smarter than this.
A controller script ran the outage. It watched the dispatcher log, killed the receiver after fire 21's ACK landed, and started it again after fire 31's retry failed. The outage is defined by delivery count, not by the clock, so exactly ten deliveries fell inside it: fires 22 through 31. No stopwatch, no guessing.
Phase 1, the dispatcher log for the first lost key:
{"ts":"2026-11-11T21:06:01.251Z","key":"evt-022","n":22,"attempt":1,"result":"error","error":"ECONNREFUSED"} Ten keys, evt-022 to evt-031, each with two error lines: the first attempt, then the retry 500 ms later, also refused. Twenty error lines in total. Everything else got an ACK.
The receiver log, when the run finished: 40 lines, all "delivered", zero duplicates. First line evt-001, last line evt-050, with a hole in the middle where the outage was.
So the count is 40 delivered, 10 lost, 0 duplicated. The title of this post is the result, not a guess. Ten webhooks vanished, and nothing I had built noticed. The dispatcher logged the errors and carried on. The receiver had nothing to log because it was dead. If this had been a real cron job syncing orders or sending receipts, ten of them would simply not exist, and the only trace would be error lines in a log file nobody reads at 3 a.m.
The retry deserves a closer look, because it feels like it should have saved something. It saved nothing. The retry fires 500 ms after the failure, and the outage lasted about fifteen seconds of dispatcher time. A retry only helps when the outage is shorter than the retry delay. Mine was thirty times longer. A retry is a seatbelt for a stumble, not a parachute for a fall. I wrote it in because every tutorial says to add a retry, and it did exactly nothing for the failure I actually had.
Phase 2. Same fifty fires, same ten-delivery outage. Two changes: the receiver now honours the idempotency key, and after the run I replay the keys that never got an ACK.
First, the honest part. The header alone recovers nothing. I want to be plain about that, because the draft description I wrote before running this said "then added one header and ran it again", as if the header were the fix. It is not the fix. The header is a promise the receiver has to keep. Phase 2 before the replay looked exactly like phase 1: 40 delivered, the same ten keys lost, zero duplicates. The header changed nothing while the receiver was dead, because a dead receiver keeps no promises.
What changed the numbers was the replay. After the dispatcher finished, a second script read its log, picked out the ten keys with no ACK, and sent each one again. All ten came back {"status":"ok"}. Ten recovered, zero still lost.
Then I ran the replay a second time, with all fifty keys, to see what the header actually buys. Every single one came back {"status":"duplicate"}. Fifty out of fifty recognised, zero re-processed. The receiver log ended at 100 lines: 50 delivered, 50 duplicate entries, 50 unique keys. Every key delivered exactly once.
That is the whole story. The replay recovers the lost work, and the idempotency key makes the replay safe. Without the key, replaying all fifty would have delivered forty of them twice. With the key, the second pass is a no-op. The pieces only work together. A retry without a replay recovers nothing. A replay without an idempotency key doubles your deliveries. The header without either is decoration.
There are limits, and I should name them, because this ran on one laptop. Localhost has no real network latency, no half-delivered POSTs, no receiver that writes the log line and dies before answering. In production the failure modes are uglier: a POST can land and its response can still be lost, which is the one case where a retry does create a duplicate, and it is the case my setup could not produce. The numbers here are for this setup. They are not a promise about a queue in production.
What I would actually build, after watching this run, is boring. The dispatcher keeps an outbox: every event written down before it is sent, with its key. A sweeper re-sends anything with no ACK, on a timer, until it gets one. The receiver keeps every key it has seen in a real store, not a log file, and answers "duplicate" for repeats. That is three small pieces, and none of them is clever. The clever version is the one I started with: fire and forget, one quick retry, hope the other end stays up. It lost ten out of fifty.
Twenty-six posts in the last seven days on dev.to argued about cron webhook recovery patterns, and twenty-five more about rate-limited queues and dead-letter recovery. Most of them debate retry counts and backoff curves. After counting my own losses, I think the debate is pointed at the wrong piece. The retry count barely mattered. The replay sweep and the key were the whole game.
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam
