Part 7 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The sign-up example is the one from Part 1, Part 2, Part 3, Part 4, Part 5, and Part 6. The numbers in this post come from running the code in it against a real Redis, with a fake email provider. Tested with Node 22, BullMQ 6.5.3, ioredis 6.0.0, and Redis 7.

Part 6 left a loose end. When a job fails on its last attempt, it does not disappear. It stays in the failed set, the worker moves on to the next job, and nobody is told. The first alarm is a customer who never got a welcome email. This post gives failed jobs a place to go, separates the failures that a retry can fix from the ones it cannot, and raises an alert when jobs pile up, before a customer notices.

Where a failed job goes

Imagine a store with 20 new sign-ups. Two people typed their address wrong, with a comma instead of a dot, and the email provider rejects those addresses for good. Three more sign-ups hit a provider outage that lasts longer than the retries do. The queue is the one from Part 6: three attempts per job, exponential backoff with jitter.

Run that, and 15 users get their email. The other five jobs end up in the failed set, and nothing in the code looks at it. It is the first row of the results further down.

A failed job is not lost. BullMQ keeps it in Redis with the error message in failedReason, the stack trace in stacktrace, and the number of tries in attemptsMade, and queue.getFailed() returns them. But nobody reads the set unless they think to look, and nothing removes its jobs unless you ask. Part 6 set removeOnComplete and left failed jobs alone, so they only pile up. A job that always fails stays there after its third attempt, as Part 6 showed.

Two kinds of failure

The five failed jobs are not the same problem. The provider answers the two typos with a 400: this address will never work. It answers the outage with a 503: try again later. Retries help with the second kind and waste time on the first.

The fake provider for this post makes the difference visible:

import { sendEmail } from "./emails.ts"; import type { Signup } from "./signup-queue.ts"; /** The provider says this address can never work (HTTP 400). Trying again cannot help. */ export class RejectedAddressError extends Error {} /** A fake email provider that rejects bad addresses and can be down for some users. */ export class FakeProvider { calls = 0; // every request that reached the provider, whether it worked or not down = new Set<string>(); // users whose requests get a 503 until recover() is called constructor(private run: string) {} async send(signup: Signup): Promise<void> { this.calls++; if (!/^[^@\s]+@[^@\s]+\.[^@\s]+$/.test(signup.email)) throw new RejectedAddressError(`400 invalid address: ${signup.email}`); if (this.down.has(signup.userId)) throw new Error("503 service unavailable"); await sendEmail(this.run, signup.userId, `welcome-email:${signup.userId}`); // the key from Part 5 } recover(): void { this.down.clear(); } }

FakeProvider counts every request that reaches it, so the demo can show how many were wasted. It rejects an address that does not look like an address with RejectedAddressError, and it answers 503 service unavailable for the users in down until recover() is called. It sends the idempotency key from Part 5 with every email, because replaying a job later is exactly the case where a job can run twice.

BullMQ has a way to say "do not retry this". If the handler throws an UnrecoverableError, the documentation describes the job as moved to failed without using the remaining attempts. A test I wrote for this post confirms it: with attempts: 3, a permanent error is tried once and a temporary error is tried three times. The function that turns one into the other is classify, in the next file.

Run the sign-ups again with classify in the handler. The same 15 users get their email and the same five jobs fail, but the provider receives 26 requests, not 30. The four requests saved are the two extra attempts on each of the two bad addresses. The second row below shows it.

Park what gave up

Now the failed jobs need somewhere to go. The pattern has a name: a dead-letter queue, a second queue that no worker reads, where jobs wait for a person or a script. BullMQ 6.5.3 has no such feature. I looked for a dead-letter option or class in the installed package, and there is none, so you build it from the pieces BullMQ gives you:

import { Queue, UnrecoverableError, type Worker } from "bullmq"; import { connection, jobIdFor, type Signup } from "./signup-queue.ts"; import { RejectedAddressError } from "./provider.ts"; /** A sign-up that ran out of attempts, with what we need to understand and replay it. */ export type DeadSignup = Signup & { reason: string; attempts: number; failedAt: number; jobId: string }; /** The queue for jobs that gave up. No worker reads it. */ export const createDeadQueue = (name: string) => new Queue<DeadSignup>(`${name}-dead`, { connection }); /** Adding a sign-up: the job ID from Part 6, and a limit on how many failed jobs Redis keeps. */ export const signupJobOptions = (signup: Signup) => ({ jobId: jobIdFor(signup), removeOnFail: { count: 100 } }); /** A permanent error must not be retried. BullMQ skips the remaining attempts for an UnrecoverableError. */ export function classify(error: unknown): Error { return error instanceof RejectedAddressError ? new UnrecoverableError(error.message) : (error as Error); } /** When a job fails for the last time, copy it to the dead queue. */ export function parkFailedJobs(worker: Worker<Signup>, dead: Queue<DeadSignup>): void { worker.on("failed", async (job, error) => { if (!job) return; const lastAttempt = error instanceof UnrecoverableError || job.attemptsMade >= (job.opts.attempts ?? 1); if (!lastAttempt) return; // BullMQ will try this job again try { const entry: DeadSignup = { ...job.data, reason: error.message, attempts: job.attemptsMade, failedAt: Date.now(), jobId: job.id! }; await dead.add("welcome-email", entry, { jobId: `dead-${job.id}` }); } catch (parkError) { console.error("could not park", job.id, parkError); // the job is still in the failed set } }); } /** Put parked sign-ups back in the main queue. Call it after the cause is fixed. */ export async function replay(dead: Queue<DeadSignup>, queue: Queue<Signup>, only = (entry: DeadSignup) => true): Promise<number> { const parked = (await dead.getJobs(["waiting"])).filter((entry) => only(entry.data)); for (const entry of parked) { const { reason, attempts, failedAt, jobId, ...signup } = entry.data; await (await queue.getJob(jobId))?.remove(); // a failed job keeps its ID, and an ID in use is ignored await queue.add("welcome-email", signup, signupJobOptions(signup)); await entry.remove(); } return parked.length; }

Here is what each part does:

  • createDeadQueue makes a queue next to the main one. It is named signups-dead for a main queue called signups. No worker reads it, so a job added to it just waits.
  • signupJobOptions limits the failed set. removeOnFail: { count: 100 } keeps the last 100 failed jobs in the main queue (see auto-removal in the documentation). The dead queue is now the record of what gave up, so the failed set can be bounded.
  • classify separates the two kinds. A rejected address becomes an UnrecoverableError. Everything else passes through unchanged and is retried.
  • parkFailedJobs copies a job when it fails for the last time. The worker's failed event fires on every failed attempt, not only the last one. The function acts only when the error is an UnrecoverableError or when attemptsMade has reached the job's attempts. The entry keeps the sign-up, the reason, the number of attempts, the time, and the ID of the original job.
  • The ID dead-<job id> is the same every time, so parking the same job twice leaves one entry, the same idea as the job IDs in Part 6.

Run it, and the dead queue holds the five jobs. The third row of the results shows it, with the failed set still at five and the provider still at 26 requests.

Failed set Dead queue
Who writes it BullMQ Our code, on the failed event
How long jobs stay Until removeOnFail removes them Until someone replays or removes them
What it keeps The job, failedReason, stacktrace, attemptsMade The sign-up, the reason, the attempts, the time, the original job ID
If the copy fails Not affected The entry is missing (see below)

Find out before a customer does

A queue that holds failed jobs and tells nobody is only a tidier version of the same problem. The simplest check looks at two numbers:

import type { Queue } from "bullmq"; import type { DeadSignup } from "./dead-letter.ts"; export type Limits = { maxDepth: number; maxAgeMs: number }; /** Look at the dead queue. Every problem found is passed to alert() and returned. */ export async function checkDeadLetters( dead: Queue<DeadSignup>, limits: Limits, alert: (message: string) => void, now = Date.now(), ): Promise<string[]> { const parked = await dead.getJobs(["waiting"]); const problems: string[] = []; if (parked.length > limits.maxDepth) problems.push(`${parked.length} parked jobs (limit ${limits.maxDepth})`); if (parked.length > 0) { const ageMs = now - Math.min(...parked.map((job) => job.data.failedAt)); if (ageMs > limits.maxAgeMs) problems.push(`the oldest parked job is ${Math.round(ageMs / 1000)} s old (limit ${limits.maxAgeMs / 1000} s)`); } for (const problem of problems) alert(problem); return problems; }

checkDeadLetters returns the problems it found and passes each one to alert. In the demo, alert prints a line. In your system, it would send a message to the channel your team reads. Delivery depends on the tools you use, so it is not built here.

The two limits catch different things:

  • Depth catches a burst. In the demo, five parked jobs against a limit of three print ALERT 5 parked jobs (limit 3).
  • Age catches the slow leak. One or two jobs never cross a depth limit, and they can sit for days. A test I wrote for this post leaves one parked job and runs the check as if two minutes had passed. The alert says the oldest job is about 120 seconds old, against a limit of 60.

Something has to run the check on a schedule. Part 9 covers repeating jobs, and a cron entry that calls the function works until then.

Replay after the fix

Once the cause is fixed, the parked jobs go back to the main queue. replay at the end of the same file, it does it in four steps for each entry: find the original job, remove it, add the sign-up again, and remove the dead entry.

The first two steps are there for a reason I found by testing. A failed job keeps its ID, and BullMQ ignores a job added with an ID that is already in use. A test I wrote for this post shows it: add the same sign-up again after it failed, and the queue still has zero waiting jobs and one failed job. So replay removes the failed job first. Part 6 said a removed job's ID can be used again, and this is that rule in use.

replay takes an optional filter, and the demo uses it. The outage is over, so the three jobs that failed with a 503 are worth replaying. The two typos are not: the address is wrong, and sending it again fails again. The demo replays everything except the entries whose reason starts with 400. The fourth row shows the result: 18 users emailed, and two entries left in the dead queue for a person to fix. Those two are what the age alert is for.

Replaying is safe because of Part 5. A test I wrote for this post sends the email, loses the reply on every attempt, parks the job, and replays it. The provider received the same key on every attempt and again on the replay, and sent one email.

The four rows

Here are the runs side by side, with the same 20 sign-ups each time: two bad addresses, and the provider down for three users until it recovers.

Strategy Emailed In failed set In dead queue Provider requests
Part 6 as is 15 5 0 30
Permanent errors not retried 15 5 0 26
Failed jobs parked 15 5 5 26
After replay 18 2 2 29

The last row has 29 requests: the 26 from before plus one for each of the three replayed jobs. The two jobs that stay in the dead queue are the typos, and they are waiting for a person.

What this does not fix

The copy can be lost. parkFailedJobs runs in the worker process. If the process dies between the failure and the copy, nothing is parked. The job is still in the failed set, at least until removeOnFail trims it. I did not test that crash.

Jobs that fail by stalling. Part 6 showed that a job whose worker dies is run again, and that one that stalls too many times is moved to the failed set. I did not test whether parkFailedJobs sees that case.

The alert is a function. Delivering it is up to you, and so is running the check on a schedule.

Redis is still one container. The dead queue lives in the same Redis as the main queue, with the same caveats as Part 6.

Seeing what happened. A parked entry says what failed and how many times. It does not say which worker handled it or what else was going on. Part 8 adds structured logs and correlation IDs for that.

Takeaways

  1. A job that fails for the last time stays in BullMQ's failed set, and nobody is told. BullMQ 6.5.3 has no dead-letter queue, so you build one from the failed event and a second queue.
  2. Retry only what a retry can fix. An UnrecoverableError skips the remaining attempts and saved 4 of 30 requests here.
  3. A dead-letter queue is only useful if something looks at it. Check depth for bursts and age for the slow leak, and replay once the cause is fixed.

Try it

This is a teaching version. A more complete implementation comes later in the series. You need Node 22, pnpm, and Docker. Use the folder from Part 6. If you do not have it, create a folder and run these commands inside it:

pnpm init pnpm add bullmq ioredis pnpm add -D tsx

Open the package.json that pnpm init created and add the line "type": "module", next to "name". Keep docker-compose.yml, signup-queue.ts, and emails.ts from Part 6 in the folder, and save provider.ts, dead-letter.ts, and monitor.ts from above next to them. Then save demo.ts, which runs the same 20 sign-ups three ways:

import { Worker } from "bullmq"; import { connection, createSignupQueue, makeSignups } from "./signup-queue.ts"; import { closeEmails, emailsPerUser, forgetEmails } from "./emails.ts"; import { FakeProvider } from "./provider.ts"; import { classify, createDeadQueue, parkFailedJobs, replay, signupJobOptions } from "./dead-letter.ts"; import { checkDeadLetters } from "./monitor.ts"; const SIGNUPS = 20; const BAD_ADDRESSES = ["user_4", "user_9"]; // typos the provider rejects for good const OUTAGE = ["user_12", "user_13", "user_14"]; // the provider is down for these until it recovers const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); /** 20 sign-ups, two of them with a typo in the address. */ function signups() { return makeSignups(SIGNUPS).map((s) => (BAD_ADDRESSES.includes(s.userId) ? { ...s, email: s.email.replace(".com", ",com") } : s)); } async function runScenario(run: string, mode: "plain" | "unrecoverable" | "parked") { const queue = createSignupQueue(run); const dead = createDeadQueue(run); await queue.obliterate({ force: true }); await dead.obliterate({ force: true }); await forgetEmails(run); const provider = new FakeProvider(run); provider.down = new Set(OUTAGE); const worker = new Worker( run, async (job) => { try { await provider.send(job.data); } catch (error) { throw mode === "plain" ? error : classify(error); } }, { connection }, ); worker.on("error", (error) => console.error(error)); if (mode === "parked") parkFailedJobs(worker, dead); for (const signup of signups()) await queue.add("welcome-email", signup, signupJobOptions(signup)); /** Wait until nothing is waiting, running or delayed, and every failed job is parked (in parked mode). */ const settle = async () => { for (;;) { const { waiting, active, delayed, failed } = await queue.getJobCounts("waiting", "active", "delayed", "failed"); const parked = (await dead.getJobCounts("waiting")).waiting; if (waiting + active + delayed === 0 && (mode !== "parked" || parked === failed)) return; await sleep(100); } }; const row = async (strategy: string) => ({ strategy, emailed: Object.keys(await emailsPerUser(run)).length, "in failed set": (await queue.getJobCounts("failed")).failed, "in dead queue": (await dead.getJobCounts("waiting")).waiting, "provider requests": provider.calls, }); await settle(); const rows Source：https://dev.to/sandrosd/failed-jobs-in-nodejs-dead-letter-handling-with-bullmq-1i1c
My favorites
PreviousHow we rewrote our backend with AI agents working from specsNextSaaS Renewal Reminders: HTTPS Queue Push, Polling, and Failure Recovery

Related posts

Distribution Release: openKylin 3.0 SP1

Distribution Release: openKylin 3.0 SP1

DistroWatch
How Block powers collaborative AI agents with Buzz and Unity Gateway

How Block powers collaborative AI agents with Buzz and Unity Gateway

Databricks Blog
Lakehouse//RT: Scale real-time analytics, not your bill

Lakehouse//RT: Scale real-time analytics, not your bill

Databricks Blog
From AI ambition to business impact

From AI ambition to business impact

Microsoft Blog
Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan

Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan

PyTorch Blog
Integrating SGLang into Spyre in Under a Week with AI

Integrating SGLang into Spyre in Under a Week with AI

PyTorch Blog