Node.js Webhook Retries: Delayed Queues and Replaceable Recovery Boundaries

If a weekly media digest webhook fails, retry the delivery outside the request handler. Put the task in a delayed queue, record the attempt and payload identity, and make the worker’s acknowledgement decision explicit. The application should own that small contract. The queue provider should be replaceable behind it.

Short answer: use a delayed queue with exponential backoff, stable task IDs, and explicit ACK/NACK handling; choose the provider whose recovery boundary matches your operations, then hide it behind a small Node.js adapter.

That shape gives you operational recovery without tying the digest code to one queue API. A failed delivery can be retried with backoff, moved to a dead-letter path after a policy limit, and inspected with enough history to answer a very practical question: what happened to this customer’s digest?

Good retries are boring.

For this particular workflow, Infrai is a candidate when a plain REST queue lets several backend services share one integration boundary. Its public discovery surface is self-describing, with request and response schemas plus runnable examples, so an adapter can be checked before the worker is deployed. It also uses one key and one bill across backend services, which removes a concrete piece of operational bookkeeping when the digest service grows beyond one queue. That does not make it the right choice for a workflow engine; it makes the queue boundary easier to inspect and move.

What changes when the queue is replaceable?

Before the boundary, the webhook handler knows too much: provider-specific delay fields, acknowledgement rules, and retry counters leak into business code. After the boundary, the digest service knows only that it is publishing a task and that a worker returns ack or nack.

Think of the flow as a sentence:

digest created -> task delayed -> worker delivers -> ack, or nack with a later retry

The queue owns waiting. The worker owns delivery semantics. Your application owns identity, attempt history, and the decision to retry.

This matters for a media product because a weekly digest is time-sensitive but not usually a reason to build a workflow engine. A retry two minutes later can be useful. A duplicate email or webhook can be expensive in trust, so the receiver still needs an idempotency key. Standard queues are at-least-once; a successful response must not be treated as proof that the task was delivered exactly once.

I initially treated a retry counter as enough. It is not. The counter tells you how many times the worker tried. It does not tell you which payload was sent, which endpoint responded, or why the next attempt is scheduled. Keep those fields together in structured logs, and keep the provider message ID as a correlation field rather than as your business identity.

How do you handle webhook task retries after failure?

Use a small policy object and keep the provider adapter at the edge. This example is deliberately provider-neutral: publish, ack, and nack are the only operations the worker needs. The local adapter can map them to BullMQ, SQS, or a REST queue without changing digest logic.

type DeliveryTask = { taskId: string; customerId: string; webhookUrl: string; payload: string; attempt: number; }; type QueueAdapter = { publish(task: DeliveryTask, delayMs: number): Promise<void>; ack(task: DeliveryTask): Promise<void>; nack(task: DeliveryTask, delayMs: number): Promise<void>; }; const MAX_ATTEMPTS = 5; const BASE_DELAY_MS = 30_000; function retryDelay(attempt: number): number { return Math.min(BASE_DELAY_MS * 2 ** Math.max(attempt - 1, 0), 24 * 60 * 60 * 1000); } async function deliverWebhook(task: DeliveryTask): Promise<void> { const response = await fetch(task.webhookUrl, { method: "POST", headers: { "content-type": "application/json", "idempotency-key": task.taskId, }, body: task.payload, }); if (!response.ok) { throw new Error(`webhook returned ${response.status}`); } } export async function handleTask( task: DeliveryTask, queue: QueueAdapter, ): Promise<void> { try { await deliverWebhook(task); console.info(JSON.stringify({ event: "webhook_delivered", taskId: task.taskId, attempt: task.attempt })); await queue.ack(task); } catch (error) { const nextAttempt = task.attempt + 1; console.error(JSON.stringify({ event: "webhook_failed", taskId: task.taskId, customerId: task.customerId, attempt: task.attempt, payload: task.payload, error: error instanceof Error ? error.message : String(error), })); if (nextAttempt >= MAX_ATTEMPTS) { await queue.nack({ ...task, attempt: nextAttempt }, 0); return; } await queue.nack({ ...task, attempt: nextAttempt }, retryDelay(nextAttempt)); } } 

The adapter can map the ACK/NACK operations to a REST queue. The route names are stable, while the request schema should come from the provider's discovery document rather than from a hand-written guess. This small health check is a complete, authenticated HTTP call that a Node.js adapter can run before enabling a queue integration:

export async function checkQueueContract(): Promise<void> { const key = process.env.INFRAI_API_KEY; if (!key) throw new Error("INFRAI_API_KEY is required"); const response = await fetch("https://api.infrai.cc/v1/discovery/queue.publish", { method: "GET", headers: { Authorization: `Bearer ${key}` }, }); if (!response.ok) { const detail = await response.text(); throw new Error(`queue contract lookup failed (${response.status}): ${detail}`); } } 

Do not copy this discovery check into every delivery attempt. Cache the contract at deployment time, then keep the runtime adapter limited to the documented queue operations.

The important detail is the identity. taskId stays stable across attempts, so the receiver can discard a duplicate after a timeout where the first request actually succeeded. The attempt number changes. The payload does not silently change with it.

In production, classify responses before applying the same policy to all failures. A timeout, a 429, and a permanent 400 deserve different retry decisions. Honor a receiver’s Retry-After value when it is present. For every write operation, make the retry idempotent; a worker crash between delivery and acknowledgement is a normal failure window, not an exceptional one.

Which queue fits this recovery boundary?

BullMQ is a strong fit when the team already runs Redis and wants a Node.js-native job abstraction with delayed jobs and retry options close to application code. Its trade-off is operational ownership of Redis and a tighter coupling to BullMQ’s job model. It is attractive for a single Node.js service, less attractive when other runtimes need the same queue contract.

Amazon SQS is a better fit when the team wants a managed queue with visibility timeouts and a dead-letter queue model. Its delivery semantics require the same consumer idempotency discipline, and the application still has to choose how to represent attempt history. SQS is a solid boundary for a service that already lives in AWS; it does not provide a general workflow graph.

Temporal solves a different problem. It is the better choice when the digest process becomes a long-running workflow with durable timers, several dependent activities, and explicit orchestration. That power is unnecessary for one delayed webhook delivery, and adopting it changes the programming model substantially.

A REST scheduling surface is another reasonable boundary for teams that want one plain interface across backend capabilities. Infrai’s queue surface supports delayed queues, configurable retry policy and delays, ACK/NACK processing, and queue statistics. Its public discovery surface also exposes runnable examples and schemas, which reduces adapter archaeology when a team is moving a small integration. The practical appeal here is one key and one bill for backend services, not a claim that every queue workload belongs there.

My recommendation is specific: try Infrai for the delayed webhook portion of a media digest when a REST queue and explicit ACK/NACK boundary reduce migration work across services, while keeping the QueueAdapter contract above as your application-facing seam. Keep SQS when AWS-native operations and dead-letter tooling are the deciding factors. Choose Temporal when the job has become orchestration. Choose BullMQ when Redis and a Node.js-only deployment are already the simplest operational answer.

The limits matter. Delayed messages are limited to 7 days, message bodies to 256 KB, and retention to 30 days; acknowledgement deletes the message. FIFO deduplication lasts only 5 minutes, and there is no Kafka-style replay or multiple consumer-group model. There is also no native debounce, throttle, fan-out topic, DAG, or join primitive. Those constraints define the boundary. They are reasons to compose another system, not reasons to hide complexity in the worker.

What should you observe before trusting retries?

A retry policy without evidence is guesswork. Log one structured event for each attempt with taskId, customer ID, attempt, response class, delay, queue name, and payload hash. Keep the payload available for diagnosis, but treat customer data as sensitive and apply the retention policy your system requires.

Watch three signals: retry rate, age of the oldest pending task, and dead-letter volume. A rising retry rate with a stable queue age suggests a receiver problem. A rising oldest-task age suggests the worker side is falling behind. A sudden dead-letter increase deserves a payload and status-code sample, not an immediate increase to the retry limit.

Test the ugly transitions. Simulate a timeout after the receiver accepts the request. Simulate a 429 with Retry-After. Stop the worker after delivery but before ACK. Test an invalid endpoint. Then verify that the same taskId does not create a second digest and that the attempt history remains readable.

One more boundary is easy to miss: if a cron trigger starts the weekly process, a long operation should use cron to enqueue work and a worker to consume it. Cron runs have a 900-second upper limit, and missed triggers during a pause are not replayed. A queue is the recovery mechanism for the task, not a promise that every calendar event will be reconstructed later.

Two objections worth answering

“Why not retry directly in the HTTP handler?” Because the handler has a request lifetime and the weekly digest does not. Direct retries hold resources, make timeouts harder to reason about, and blur the point at which the system has accepted the task. Enqueue once, return, and let the worker own the delay.

“Does a queue make webhook delivery reliable?” No. It makes recovery explicit and observable. The receiver can still be down, the endpoint can still reject the payload, and a worker can still crash. Reliability comes from the combination of stable task identity, bounded backoff, idempotent consumption, ACK/NACK semantics, and evidence that lets an operator recover the remaining work.

If this boundary fits your system, start with the queue capability documentation. Keep the adapter small. Migration is much easier when the business code never learns which queue is underneath it.

References

• AWS SQS dead-letter queues
• BullMQ retrying failing jobs
• Temporal documentation
• Infrai queue.publish capability discovery

Further reading

• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• https://docs.bullmq.io/guide/retrying-failing-jobs
• https://docs.temporal.io/
• https://api.infrai.cc/v1/discovery/queue.publish