Webhook retries are one of those topics almost nobody thinks about until the first partner calls and asks why no orders have reached them since yesterday. A webhook is quick to build: an HTTP POST to someone else’s URL, done. What happens when the other side is in the middle of a deployment, returns 503 for a minute or simply does not answer is not covered in many implementations. Then the event is gone.
The topic matters to me because webhooks are where a cleanly built event-driven system leaves its own garden. Inside AWS, SQS, EventBridge and Lambda take care of retries. With a foreign URL, you are on your own. So in this eighth part of the series, I show you how I would build webhook retries today: with a queue, exponential backoff, signed messages and a dead-letter queue for the case where nothing works anymore.
In short
The best way to send webhooks is as jobs through an SQS queue: a Lambda function delivers them and retries failures with a growing delay via ChangeMessageVisibility. Every message carries a fixed ID and an HMAC signature so the receiver can spot duplicates and reject forgeries. The biggest pitfall is assuming that “send once” is enough: without retries, a DLQ and an alarm, you lose events without noticing.
What you will learn in this article
- How to deliver webhooks reliably through SQS with exponential backoff
- Which HTTP status codes trigger a new attempt and which do not
- How to sign and verify webhooks with HMAC in the Standard Webhooks format
- How to deploy and test everything with the Serverless Framework and TypeScript
Why webhooks get lost so easily
The obvious approach looks like this: an order gets paid, the code stores it and right after that calls every partner’s webhook URL. That always works in testing. In production, you run into receivers that are restarting, load balancers with their own timeouts and networks that simply drop a packet. A single attempt is therefore a bet that everything is fine on the receiver’s side at the exact moment of the event.
There is a second problem. If the call sits in the request path, your own customer waits for the response of a foreign system. If the partner answers slowly, your API becomes slow. Without any answer at all, your Lambda function runs into its timeout. That is exactly the coupling you wanted to avoid with an event-driven architecture.
A webhook is a message to a system you do not control. For messages like that, you need the same tools as for any other unreliable target: decoupling through a queue, retries with a delay and a place for messages that failed for good. I explained the basics of delays with a random component in detail in the article Exponential backoff with jitter. Here, the focus is on what that looks like for webhooks.
Webhook retries with SQS: the architecture at a glance
The example is once again a fictional ticket shop called “Kartenkiste”. When an order is paid or refunded, partners should learn about it, for example a point-of-sale system at the box office or an admission app. Each partner registers a URL and gets its own signing secret.
From event to partner: dispatch, delivery queue, deliverer and DLQ with alarm
The flow has two stages. The shop publishes an event such as OrderPaid on the EventBridge bus. A small dispatch function reads from a DynamoDB table who subscribed to this event and puts one delivery job per subscriber into an SQS queue. A second function, the deliverer, picks up the jobs from the queue, signs them and sends them to the partner via HTTP. If delivery fails, the job stays in the queue and comes back after a growing delay. After eight attempts, SQS moves it to a dead-letter queue (DLQ), a storage area for messages that cannot be processed. A CloudWatch alarm fires as soon as something lands there.
Why a queue instead of retries in code
You could simply write a loop with await sleep() in the deliverer. That has three drawbacks. First, Lambda bills you for every second the function waits. Second, the function stops after 15 minutes at the latest, because Lambda does not run longer. Third, the job is lost when the function crashes or gets redeployed. For webhook retries, a queue solves all three: the message sits safely in SQS while nobody pays for compute time, and the delay can be hours.
In addition, the queue keeps partners apart from each other. If one receiver hangs, only its own jobs are affected. The same principle is behind the fan-out with SNS and SQS: every consumer gets its own lane.
Aren’t EventBridge API destinations enough?
A fair question, because EventBridge can call HTTP endpoints directly through API destinations. According to the documentation, EventBridge retries errors such as 429 and 5xx for up to 24 hours by default, reads the Retry-After header and supports a DLQ. For internal targets or a handful of fixed partners, that is a good choice because you do not need any code of your own.
With real webhooks, however, I run into limits. A request may take five seconds at most, authentication is limited to basic auth, OAuth or an API key, and every new subscriber needs its own rule with a connection. An HMAC signature over the body per subscriber is not part of it. As soon as partners register themselves and are supposed to verify their messages, your own delivery through SQS is more flexible.
The rules: when is a webhook delivered?
Before you implement webhook retries, it pays to agree on clear rules with the receivers. For that, I follow the open specification Standard Webhooks, which several vendors developed together. It defines three headers: webhook-id is a unique ID that stays the same for every attempt. webhook-timestamp contains the time of sending in seconds. webhook-signature carries an HMAC-SHA256 signature over ID, timestamp and body, prefixed with v1,.
For the receiver’s responses, I use these rules:
| Receiver response | What the deliverer does |
|---|---|
| 2xx (200 to 299) | Delivered, the message is deleted |
| 410 Gone | The receiver no longer wants webhooks: disable the subscription, do not retry |
429 or 503 with Retry-After | Try again, at the earliest after the given time |
| 3xx (redirect) | Do not follow, treat as a failure and retry |
| All other 4xx and 5xx | Try again with a growing delay |
| Timeout or network error | Try again with a growing delay |
The other 4xx codes are debatable. A 400 or 404 often points to a permanent problem, and retries only postpone the error. On the other hand, many systems briefly return a 404 or 401 during a deployment. Here I stick to the specification and retry everything except 410, because lost events are usually more expensive for partners than a few unnecessary attempts.
Step by step: building webhook retries with Lambda
Now for the code. The project consists of a few small modules that you can test one by one.
Prerequisites: Node.js 22, an AWS account with configured credentials, the Serverless Framework v4 (npm i -g serverless) and basic TypeScript knowledge. I tested the code with TypeScript 7.0.2, Node.js 22.23 and the AWS SDK for JavaScript 3.1159.0.
Step 1: Create the project
Create an empty directory and install the dependencies. The tests run with the built-in Node.js test runner, so you need neither Jest nor Vitest.
mkdir kartenkiste-webhooks && cd kartenkiste-webhooks npm init -y npm i @aws-sdk/client-sqs @aws-sdk/client-ssm @aws-sdk/client-dynamodb npm i -D typescript @types/node@22 @types/aws-lambda mkdir src test In package.json, set "type": "module" and two scripts. The --experimental-strip-types option lets Node.js run the TypeScript files directly without compiling them first.
{ "name": "webhook-retries", "version": "1.0.0", "private": true, "type": "module", "scripts": { "typecheck": "tsc --noEmit", "test": "node --test --experimental-strip-types test/*.test.ts" }, "dependencies": { "@aws-sdk/client-dynamodb": "^3.1159.0", "@aws-sdk/client-sqs": "^3.1159.0", "@aws-sdk/client-ssm": "^3.1159.0" }, "devDependencies": { "@types/aws-lambda": "^8.10.164", "@types/node": "^22.19.0", "typescript": "^7.0.2" } } { "compilerOptions": { "target": "ES2023", "module": "NodeNext", "moduleResolution": "NodeNext", "strict": true, "noEmit": true, "allowImportingTsExtensions": true, "verbatimModuleSyntax": true, "erasableSyntaxOnly": true, "skipLibCheck": true, "types": ["node"] }, "include": ["src", "test"] } Step 2: Sign webhooks
The signature is the first thing I build, because sender and receiver must understand it the same way. HMAC stands for “hash-based message authentication code”: a shared secret and the content produce a check value that only someone with the same secret can create. The string that gets signed is <id>.<timestamp>.<body>. Because the timestamp is included, an attacker cannot simply resend an intercepted message later.
// src/signature.ts import { createHmac, timingSafeEqual } from 'node:crypto'; // Secrets in the Standard Webhooks format: "whsec_" + base64 function secretKey(secret: string): Buffer { const raw = secret.startsWith('whsec_') ? secret.slice(6) : secret; return Buffer.from(raw, 'base64'); } // Signs "<id>.<timestamp>.<body>" with HMAC-SHA256 and returns "v1,<base64>" export function signWebhook(secret: string, id: string, timestamp: number, body: string): string { const mac = createHmac('sha256', secretKey(secret)) .update(`${id}.${timestamp}.${body}`) .digest('base64'); return `v1,${mac}`; } export interface VerifyInput { secret: string; id: string | undefined; timestamp: string | undefined; signatureHeader: string | undefined; body: string; // the raw body, never re-serialized JSON nowSeconds?: number; toleranceSeconds?: number; } // Receiver side: check the timestamp, then compare every signature in the header export function verifyWebhook(input: VerifyInput): boolean { const { secret, id, timestamp, signatureHeader, body } = input; if (!id || !timestamp || !signatureHeader) return false; const ts = Number(timestamp); if (!Number.isInteger(ts)) return false; const now = input.nowSeconds ?? Math.floor(Date.now() / 1000); if (Math.abs(now - ts) > (input.toleranceSeconds ?? 300)) return false; const expected = Buffer.from(signWebhook(secret, id, ts, body)); // Multiple signatures are separated by spaces (key rotation) return signatureHeader.split(' ').some((candidate) => { const given = Buffer.from(candidate); return given.length === expected.length && timingSafeEqual(given, expected); }); } Two details matter. timingSafeEqual compares in constant time so that nobody can guess the signature character by character from response times. And the header may contain several signatures separated by spaces. That way you can rotate a secret without receivers rejecting messages during the transition.
Step 3: Calculate delays
The heart of webhook retries is the delay. It grows by a factor of three with every attempt: one minute, three, nine, 27 and so on, capped at six hours. To keep all failed messages from coming back at the same time, half of the delay is random. With eight attempts, this results in a window of roughly six to twelve hours. That covers a partner outage overnight without keeping messages around for days.
// src/backoff.ts const BASE_SECONDS = 60; const FACTOR = 3; const MAX_DELAY_SECONDS = 6 * 60 * 60; // 6 hours, SQS allows at most 12 const MAX_RETRY_AFTER_SECONDS = 12 * 60 * 60; // Exponentially growing delay (1, 3, 9, 27 … minutes) with "equal jitter": // half of it is fixed, the other half is random export function retryDelaySeconds(attempt: number, random: () => number = Math.random): number { const exp = Math.min(MAX_DELAY_SECONDS, BASE_SECONDS * FACTOR ** Math.max(0, attempt - 1)); const half = exp / 2; return Math.max(1, Math.round(half + random() * half)); } // Retry-After may contain seconds or an HTTP date export function parseRetryAfter(value: string | null, nowMs: number = Date.now()): number | undefined { if (!value) return undefined; const trimmed = value.trim(); let seconds: number; if (/^\d+$/.test(trimmed)) { seconds = Number(trimmed); } else { const date = Date.parse(trimmed); if (Number.isNaN(date)) return undefined; seconds = Math.ceil((date - nowMs) / 1000); } return Math.min(MAX_RETRY_AFTER_SECONDS, Math.max(1, seconds)); } If the receiver sends a Retry-After header, it takes precedence. It may contain a number of seconds or an HTTP date, so the function handles both forms.
Step 4: Send a webhook
The sendWebhook function builds the body exactly once, signs this string and sends it. After that, it sorts the result into one of three categories: delivered, unsubscribed or try again.
// src/deliver.ts import { parseRetryAfter } from './backoff.ts'; import { signWebhook } from './signature.ts'; export interface DeliveryJob { webhookId: string; // stays the same for every attempt subscriberId: string; eventType: string; url: string; payload: unknown; occurredAt: string; } export type DeliveryResult = | { kind: 'delivered'; status: number } | { kind: 'gone' } | { kind: 'retry'; reason: string; retryAfterSeconds?: number }; export type FetchFn = typeof fetch; export async function sendWebhook( job: DeliveryJob, secret: string, fetchFn: FetchFn = fetch, nowSeconds: number = Math.floor(Date.now() / 1000), timeoutMs = 10_000, ): Promise<DeliveryResult> { // Serialize the body exactly once: this string is signed and sent const body = JSON.stringify({ type: job.eventType, timestamp: job.occurredAt, data: job.payload }); let response: Response; try { response = await fetchFn(job.url, { method: 'POST', redirect: 'manual', // do not follow redirects signal: AbortSignal.timeout(timeoutMs), headers: { 'content-type': 'application/json', 'user-agent': 'Kartenkiste-Webhooks/1.0', 'webhook-id': job.webhookId, 'webhook-timestamp': String(nowSeconds), 'webhook-signature': signWebhook(secret, job.webhookId, nowSeconds, body), }, body, }); } catch (error) { // Timeout, DNS error, refused connection: try again later const name = error instanceof Error ? error.name : 'Error'; return { kind: 'retry', reason: `network:${name}` }; } // Discard the body so the connection is released await response.body?.cancel().catch(() => undefined); if (response.status >= 200 && response.status < 300) { return { kind: 'delivered', status: response.status }; } if (response.status === 410) { return { kind: 'gone' }; } return { kind: 'retry', reason: `http:${response.status}`, retryAfterSeconds: parseRetryAfter(response.headers.get('retry-after')), }; } The ten-second limit comes from AbortSignal.timeout. The specification even recommends 15 to 30 seconds. I deliberately stay below that, because a Lambda function processes up to ten messages per invocation and a slow partner would otherwise hold up the whole batch. With redirect: 'manual', the deliverer does not follow redirects. A partner who has moved should update their URL instead of your messages silently ending up somewhere else.
Step 5: The deliverer with batch failures
Now comes the core of webhook retries. The function receives a batch from SQS and reports back through batchItemFailures which messages failed. Only those stay in the queue, Lambda deletes all others. To make this work, you have to enable the ReportBatchItemFailures option on the event source mapping.
The actual trick is ChangeMessageVisibility. Normally, a failed message becomes visible again after the queue’s visibility timeout expires, so always after the same time. If you set a new visibility timeout for exactly this message before reporting back, you decide yourself when it returns. SQS supplies the number of previous attempts in the ApproximateReceiveCount attribute.
🎬 Webhook retries with growing wait times – this animated diagram only runs on my site: watch it on sascha-ms.de
// src/handler.ts import type { SQSBatchResponse, SQSEvent, SQSRecord } from 'aws-lambda'; import { ChangeMessageVisibilityCommand, SQSClient } from '@aws-sdk/client-sqs'; import { GetParameterCommand, SSMClient } from '@aws-sdk/client-ssm'; import { DynamoDBClient, UpdateItemCommand } from '@aws-sdk/client-dynamodb'; import { retryDelaySeconds } from './backoff.ts'; import { sendWebhook, type DeliveryJob, type DeliveryResult } from './deliver.ts'; export interface DeliveryDeps { maxAttempts: number; getSecret: (subscriberId: string) => Promise<string>; send: (job: DeliveryJob, secret: string) => Promise<DeliveryResult>; delayMessage: (receiptHandle: string, seconds: number) => Promise<void>; disableSubscription: (job: DeliveryJob) => Promise<void>; random?: () => number; log?: (entry: Record<string, unknown>) => void; } export function createDeliveryHandler(deps: DeliveryDeps) { const log = deps.log ?? ((entry) => Source：https://dev.to/sascha-ms/webhook-retries-reliable-delivery-with-sqs-and-lambda-3im5Related posts
Your AI Coding Agent Needs More Than a Good Prompt
How to Migrate WordPress to Next.js Without Losing SEO
A 5.86 MB font sat in my app for three months. No CSS rule ever referenced it.
Türkçe Slug Üretirken I/ı/İ/i Sorununu Çözmek
Pray for Chaos
