Part 6 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior full-stack engineer.

The sign-up example is the one from Part 1, Part 2, Part 3, Part 4, and Part 5. The numbers in this post come from a test that starts and kills real processes against a real Redis, not from a simulation. Tested with Node 22, BullMQ 6.5.3, ioredis 6.0.0, and Redis 7.

Every queue in this series so far has lived in the memory of the process that runs it. That works until the process stops. A deploy, a crash, or an out-of-memory kill empties the queue, and every sign-up still waiting for its welcome email is gone, with no error and no log line. This post moves the queue to Redis with BullMQ, and then kills the worker on purpose to see what survives.

What a restart does to an in-memory queue

The queue from Part 2 keeps its jobs in an array inside the Node.js process. When the process dies, the array dies with it. Nothing was ever written down, so nothing can be recovered.

Imagine a store with 20 new sign-ups waiting in the queue. The worker handles them one at a time, and it dies right after it sends the welcome email for the sixth one. The process restarts with an empty queue.

Six users get their email. The other 14 never will, and there is no failed job to look at, because queue.failed lived in the same memory.

That is the first row of the results further down. The next sections build the other two.

A queue that lives in Redis

Redis is a data store that runs as its own process. BullMQ is a Node.js library that keeps a queue inside it: the jobs, their state, the retry settings, and a lock that says which worker is handling which job. The process that adds jobs and the process that runs them no longer share memory. They share Redis, so when the worker dies, the jobs are still there.

In BullMQ 6, the Redis client is a separate package. The documentation says, "By default, BullMQ creates Redis connections with ioredis," so you install both with pnpm add bullmq ioredis. I tried without ioredis: the first new Queue(...) throws an error that asks for it.

Redis itself runs in Docker. This docker-compose.yml starts it with the two settings that BullMQ's production guide asks for:

services: redis: image: redis:7 command: ["redis-server", "--appendonly", "yes", "--maxmemory-policy", "noeviction"] ports: - "6379:6379" volumes: - redis-data:/data volumes: redis-data:

--appendonly yes makes Redis write its changes to disk. The guide says, "We recommend enabling AOF (Append Only File), which provides a robust and fast solution." --maxmemory-policy noeviction stops Redis from deleting keys when memory runs out, and a deleted key can be a waiting job. The guide calls it "the only setting that guarantees the correct behavior of the queues." The volume keeps the data when the container is replaced.

The queue and the worker

Two files do the work. signup-queue.ts creates the queue and holds the options that every job gets:

import { Queue } from "bullmq"; export type Signup = { userId: string; email: string }; export const connection = { host: "localhost", port: 6379 }; /** The same sign-ups for every run: user_1, user_2, and so on. */ export function makeSignups(count: number): Signup[] { return Array.from({ length: count }, (_, i) => ({ userId: `user_${i + 1}`, email: `user${i + 1}@example.com` })); } /** The jobs live in Redis. These options apply to every job unless a job overrides them. */ export function createSignupQueue(name: string): Queue<Signup> { return new Queue<Signup>(name, { connection, defaultJobOptions: { attempts: 3, // up to 3 tries per job, the same limit as in Part 3 backoff: { type: "exponential", delay: 1000, jitter: 1 }, // up to 1 s, then up to 2 s, chosen at random removeOnComplete: { count: 1000 }, // keep the last 1,000 finished jobs, not all of them }, }); } /** A job ID that names the business action, so the same sign-up added twice is one job. */ export const jobIdFor = (signup: Signup) => `welcome-email-${signup.userId}`;

Four things in it are new:

  • Retries are options now. With attempts: 3, a job that throws is tried up to three times. With exponential backoff, the documentation says BullMQ retries "after 2 ^ (attempts - 1) * delay milliseconds." Here, that means 1 second before the second attempt and 2 seconds before the third.
  • jitter: 1 is the full jitter from Part 3. The same page says that with a value of 1, "random delays will be generated between 0 and max generated value."
  • removeOnComplete: { count: 1000 } limits what Redis keeps. By default, BullMQ keeps finished jobs in Redis. This keeps the last 1,000.
  • jobIdFor gives each sign-up a job ID that names the business action, the same idea as the key in Part 5. The documentation says that if you add a job with an existing ID, "that job will just be ignored." A sign-up that arrives twice, for example because a client timed out and sent the request again, becomes one job. The ID cannot contain a colon or consist only of digits, which is why it is welcome-email-user_1 and not welcome-email:user_1. Tests I wrote for this post confirm all three rules, and a fourth: once a job is removed, its ID can be used again. The protection lasts only as long as the job stays in Redis.

The second file is the worker. It runs as its own process:

import { Worker } from "bullmq"; import { connection, type Signup } from "./signup-queue.ts"; import { closeEmails, sendEmail } from "./emails.ts"; const run = process.env.RUN!; // names the queue and the email records of this run const crashUser = process.env.CRASH_USER; // die while handling this user const useKey = process.env.USE_KEY === "1"; // send an idempotency key to the provider const worker = new Worker<Signup>( run, async (job) => { await sendEmail(run, job.data.userId, useKey ? `welcome-email:${job.data.userId}` : undefined); // The email went out. Now the process dies before it can tell the queue. if (job.data.userId === crashUser) process.kill(process.pid, "SIGKILL"); }, // The defaults for both timers are 30,000 ms. They are shorter here so the demo finishes quickly. { connection, lockDuration: 2000, stalledInterval: 1000 }, ); worker.on("error", (error) => console.error(error)); // The parent process sends a message when the queue is empty: close cleanly. process.on("message", async () => { await worker.close(); await closeEmails(); process.exit(0); });

The worker takes one job at a time, as the queue from Part 2 did by default. The two lines with crashUser are test scaffolding: they kill the process after the email goes out and before the job is reported as done.

The two timers need an explanation. While a worker runs a job, it keeps renewing a lock on it. If the worker dies, the lock expires, and a check that runs every stalledInterval finds the job and puts it back in line. The defaults are 30,000 ms for both (worker options). Here they are 2,000 and 1,000 ms, so the test takes seconds, not minutes.

In real use, keep the defaults unless you have a reason to change them: a worker that blocks the event loop for longer than the lock lasts looks dead, and its job runs again.

Kill the worker

kill-test.ts is in the Try it section below. It runs three cases with 20 sign-ups each, and in every case the worker process dies right after it sends the welcome email for user_6. In the Redis cases, the process that adds the jobs stays alive, and a new worker takes over after the crash and runs until the queue is empty. The fake email provider records its emails in Redis, so the count survives the crash.

Strategy Users emailed Users never emailed Duplicate emails
In-memory queue 6 14 0
BullMQ, no key 20 0 1
BullMQ, key sent to the provider 20 0 0

Here is what each row says:

  • In-memory queue. Six users got an email, and the other 14 sign-ups vanished. The restarted process had nothing to run.
  • BullMQ, no key. Every user got an email, and the sixth got two.
  • BullMQ, key sent to the provider. Every user got exactly one.

Why the sixth user got two emails

The worker sent the email for user_6 and died before it told Redis. From Redis's point of view, the job was still running when its lock expired. The documentation says, "A stalled job is moved back to the waiting status and will be processed again by another worker." So the job ran again, and the second email went out.

This is the lost reply from Parts 4 and 5 in another form: the process that did the work was killed before it could report. The queue cannot tell "the email was never sent" from "the email was sent, and I was not told," so it runs the job again. The store from Part 5 would not have helped either, because it lived in the memory of the process that died.

The fix is the one from Part 5: the key has to reach whoever performs the side effect. The third row sends welcome-email:user_6 with the email. The provider has seen that key, so the second run sends nothing.

A job that stalls more than once does not loop forever: the same page says it is moved to the failed set, and the default allows one recovery per job.

Retries move into the queue

Part 3 wrapped the handler in withRetry. BullMQ does the same work from the outside: when a job throws, the queue holds it for the backoff delay and then tries it again.

Two differences matter. First, the wait no longer blocks the worker. In Part 3, a job waiting for its retry kept its concurrency slot. A test I wrote for this post shows the opposite with BullMQ: with one worker and one slot, user_1 fails its first attempt, user_2 runs while user_1 waits, and then user_1 gets its second attempt.

Second, BullMQ retries the whole job, like the whole-job retry from Part 4. A job with several steps repeats the steps that already worked. You can keep retryEachStep from Part 4 and the keys from Part 5 inside the job handler, and let BullMQ's attempts catch what is left.

What this does not fix

Redis is now a part you run. AOF and noeviction lower the risk of losing jobs, and they do not remove it. While the single Redis container is down, no job can be added or run. Replication is outside this post.

The key store from Part 5. It lives in memory, so a restart forgets it. This post sends the key to the provider instead, and the provider remembers it. A provider that takes no key needs a shared store, and Redis can be that store.

Failed jobs. After the last attempt, a job stays in the failed set, and nobody is told. A test I wrote for this post confirms it: a job that always fails is tried 3 times and then stays there. Part 7 covers where failed jobs go and how to notice them.

Seeing it happen. In the test, the next worker took over without a single log line. Part 8 adds structured logs, so you can tell which job ran twice and why.

Takeaways

  1. A queue in memory loses every waiting job when the process stops, and nothing reports it.
  2. A queue in Redis keeps the jobs, but a job that was running when the worker died runs again. The job has to be safe to run twice, and the key has to reach the provider.
  3. Retries become options on the queue. They retry the whole job, so the lessons of Parts 4 and 5 still apply.

Try it

This is a teaching version of a Redis-backed queue. A more complete implementation comes later in the series. You need Node 22, pnpm, and Docker. Create a folder and run these commands inside it:

pnpm init pnpm add bullmq ioredis pnpm add -D tsx

Open the package.json that pnpm init created and add the line "type": "module", next to "name". The example uses top-level await, which only works in a module.

Keep queue.ts from Part 2 in the folder. Save the docker-compose.yml, signup-queue.ts, and worker.ts files from above, and then save these three. emails.ts is the fake email provider:

import { Redis } from "ioredis"; const redis = new Redis({ host: "localhost", port: 6379 }); /** A fake email provider. It writes every email to Redis, so the record survives a crash of our process. */ export async function sendEmail(run: string, userId: string, key?: string): Promise<void> { // A provider that honors idempotency keys (Part 5) sends nothing for a key it has already seen. if (key && (await redis.set(`${run}:seen:${key}`, "1", "NX")) === null) return; await redis.hincrby(`${run}:emails`, userId, 1); } /** How many emails each user received in this run. */ export async function emailsPerUser(run: string): Promise<Record<string, number>> { const counts = await redis.hgetall(`${run}:emails`); return Object.fromEntries(Object.entries(counts).map(([userId, count]) => [userId, Number(count)])); } export async function forgetEmails(run: string): Promise<void> { await redis.del(`${run}:emails`, ...(await redis.keys(`${run}:seen:*`))); } export const closeEmails = () => redis.quit();

memory-worker.ts is the in-memory case. It receives the sign-ups, handles them with the queue from Part 2, and dies at user_6:

import { SimpleQueue } from "./queue.ts"; import { makeSignups, type Signup } from "./signup-queue.ts"; import { closeEmails, sendEmail } from "./emails.ts"; const run = process.env.RUN!; const crashUser = process.env.CRASH_USER; const count = Number(process.env.SIGNUPS ?? 0); // how many sign-ups this process receives const queue = new SimpleQueue<Signup>(async (job) => { await sendEmail(run, job.data.userId); // The email went out. Now the process dies, and so does every job waiting in memory. if (job.data.userId === crashUser) process.kill(process.pid, "SIGKILL"); }); for (const signup of makeSignups(count)) queue.add(signup); await queue.onIdle(); await closeEmails();

kill-test.ts runs the three cases and prints the table:

import { spawn, type ChildProcess } from "node:child_process"; import { fileURLToPath } from "node:url"; import { createSignupQueue, jobIdFor, makeSignups } from "./signup-queue.ts"; import { closeEmails, emailsPerUser, forgetEmails } from "./emails.ts"; const SIGNUPS = 20; const CRASH_USER = "user_6"; /** Start worker.ts or memory-worker.ts as a separate process. */ function start(file: string, env: Record<string, string>): ChildProcess { return spawn(process.execPath, ["--import", "tsx", fileURLToPath(new URL(file, import.meta.url))], { env: { ...process.env, ...env }, stdio: ["ignore", "inherit", "inherit", "ipc"], }); } const exited = (child: ChildProcess) => new Promise<void>((resolve) => child.once("exit", () => resolve())); /** The queue is in memory: the process dies, and a new process starts with an empty queue. */ async function memoryQueue(run: string) { await exited(start("memory-worker.ts", { RUN: run, CRASH_USER, SIGNUPS: String(SIGNUPS) })); await exited(start("memory-worker.ts", { RUN: run, SIGNUPS: "0" })); // the restart has no jobs to run } /** The queue is in Redis: a worker dies, and a new worker takes over. */ async function redisQueue(run: string, useKey: boolean) { const queue = createSignupQueue(run); await queue.obliterate({ force: true }); for (const signup of makeSignups(SIGNUPS)) await queue.add("welcome-email", signup, { jobId: jobIdFor(signup) }); const env = { RUN: run, USE_KEY: useKey ? "1" : "0" }; await exited(start("worker.ts", { ...env, CRASH_USER })); // dies while handling user_6 const restarted = start("worker.ts", env); for (;;) { const { waiting, active, delayed } = await queue.getJobCounts("waiting", "active", "delayed"); if (waiting + active + delayed === 0) break; await new Promise((resolve) => setTimeout(resolve, 100)); } restarted.send("stop"); await exited(restarted); await queue.obliterate({ force: true }); await queue.close(); } async function row(strategy: string, run: string) { const perUser = Object.values(await emailsPerUser(run)); return { strategy, "users emailed": perUser.length, "users never emailed": SIGNUPS - perUser.length, "duplicate emails": perUser.reduce((sum, count) => sum + count - 1, 0), }; } const runs = ["signups-memory", "signups-redis", "signups-redis-key"]; for (const run of runs) await forgetEmails(run); await memoryQueue(runs[0]); await redisQueue(runs[1], false); await redisQueue(runs[2], true); console.log(`${SIGNUPS} sign-ups. The worker dies right after it sends the email for ${CRASH_USER}.`); Source：https://dev.to/sandrosd/jobs-that-survive-a-crash-nodejs-and-bullmq-o0b
My favorites
PreviousEliminating AI Coding Loops: Deterministic Flight Recording & Polyglot Blast-Radius with genesis-blackbox v0.2.0NextBuilding hldays #9: The architecture, TypeScript on Cloudflare Workers

Related posts

FiveM Script Database Migrations: What to Check Before You Install

FiveM Script Database Migrations: What to Check Before You Install

dev.to
The insurance edge case that breaks event-driven policy pipelines

The insurance edge case that breaks event-driven policy pipelines

dev.to
The fix for an empty state isn't filling it

The fix for an empty state isn't filling it

dev.to UX
Five Things That Make Dependabot Painful for Terraform

Five Things That Make Dependabot Painful for Terraform

dev.to GitHub
Tracking Roblox player counts: what the public data actually gives you

Tracking Roblox player counts: what the public data actually gives you

dev.to Data
Telegram gold signals this week: results to Oct 30

Telegram gold signals this week: results to Oct 30

dev.to Data