The payment provider sent an order.paid webhook. The handler created the order, generated a PDF invoice, emailed it and synced the CRM. That took 14 seconds. The provider's timeout was 10. It marked the delivery as failed and retried. Then again. Each customer got three invoices.

The handler never errored. It was just too slow to say "got it".

TL;DR

  • Webhook senders have short timeouts (often 5 to 30 seconds) and retry on anything other than a fast 2xx.
  • Do four things inline: verify, store, dedupe, acknowledge. Do everything else asynchronously.
  • Returning 200 means "I have it safely", not "I've finished processing it".
  • My opinion: any webhook handler that calls a third-party API before responding is a retry storm waiting for a slow day.

What senders actually do

Most providers follow the same pattern: POST the event, wait a few seconds, treat a timeout or non-2xx as failure, retry with backoff for hours or days. Some disable your endpoint after enough failures.

So a slow handler doesn't just risk one timeout. It creates duplicates during the retries and can get your endpoint switched off during an incident, which is exactly when you need the events most.

The four-step handler 

1. verify signature (cheap, rejects junk) 2. store raw event (durable, before anything else) 3. dedupe on event id (unique constraint) 4. return 2xx (target: < 500 ms) --- async --- 5. process from storage (your worker, your timeouts, your retries)

Node (Express) 

import express from 'express'; import crypto from 'node:crypto'; const app = express(); app.post('/webhooks/payments', express.raw({ type: 'application/json' }), async (req, res) => { if (!verify(req.headers['x-signature'], req.body)) return res.sendStatus(401); const event = JSON.parse(req.body); try { await db.query( `INSERT INTO inbound_events (id, type, payload, received_at, status) VALUES ($1, $2, $3, now(), 'pending') ON CONFLICT (id) DO NOTHING`, [event.id, event.type, req.body.toString()] ); } catch (e) { return res.sendStatus(500); // storage failed: let the sender retry } res.sendStatus(200); // stored (or already had it): acknowledge }); function verify(sig, raw) { const mac = crypto.createHmac('sha256', process.env.WEBHOOK_SECRET).update(raw).digest('hex'); return sig && mac.length === sig.length && crypto.timingSafeEqual(Buffer.from(mac), Buffer.from(sig)); }

Two details matter. Verification runs on the raw body, before JSON parsing, or the signature won't match. And a duplicate insert still returns 200, because telling the sender to retry something you already have just creates more duplicates. The signature details are covered in How to verify HMAC-SHA256 signed webhooks in Node and Python.

The worker 

import time def work_loop(): while True: rows = db.execute(""" UPDATE inbound_events SET status='processing', locked_at=now() WHERE id IN ( SELECT id FROM inbound_events WHERE status='pending' OR (status='processing' AND locked_at < now() - interval '10 minutes') ORDER BY received_at LIMIT 20 FOR UPDATE SKIP LOCKED) RETURNING id, type, payload""").fetchall() if not rows: time.sleep(1) continue for r in rows: try: handle(r.type, r.payload) # must be idempotent  db.execute("UPDATE inbound_events SET status='done' WHERE id=%s", (r.id,)) except Exception as e: db.execute("""UPDATE inbound_events SET status='pending', attempts=attempts+1, last_error=%s WHERE id=%s""", (str(e)[:500], r.id))

This assumes the table also has attempts, locked_at and last_error columns. FOR UPDATE SKIP LOCKED lets you run several workers on the same table without them grabbing the same event. Stuck "processing" rows get picked up again after ten minutes, which means handle can run twice for the same event. That's why it needs to be idempotent: see the previous post on at-least-once delivery.

Add an attempts limit and move poison events aside instead of retrying forever. A small dead-letter table is enough.

"But I don't want another table"

Fair. Alternatives, from simplest to heaviest:

  • Background task in the same process (e.g. a framework's background tasks): fast to build, but events are lost if the process dies before finishing. Fine for non-critical events.
  • A queue you already run (Redis, SQS, a job library): push the raw event, acknowledge, let workers consume.
  • An inbox table, as above: durable, queryable, replayable. My default for anything involving money.

The key property is the same: the event is durable before you return 200.

Ordering

Retries and parallel workers mean events can be processed out of order: subscription.updated before subscription.created. Don't trust arrival order. Either re-fetch the current state from the provider's API when processing, or compare a version or timestamp field and ignore stale events.

Testing it 

# time your handler's response, not your processing curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' \ -X POST https://example.com/webhooks/payments \ -H "x-signature: $SIG" --data-binary @event.json

Anything over a second deserves a look. Send the same event twice and confirm only one row and one side effect.

If you're on the sending side, for example firing scheduled HTTP callbacks from cron or a service like delay.run, the same rule applies to your receivers. Keep the endpoint you call fast and do the heavy lifting afterwards. How to schedule an HTTP request covers the sending side.

My take

A webhook endpoint is a mailbox, not an office. Its only job is to accept the letter safely and close the door. Everything you do before responding is a chance to time out and get the same letter again.

Over to you

Do you process webhooks inline or through an inbox/queue, and what made you choose? If you've been burned by a provider disabling your endpoint after timeouts, I'd like to hear how you recovered the missed events.