he second-most-common reason I reach for Redis is a job queue.
Not a high-throughput event bus — just "send this email", "sync this customer",
"generate this PDF". A few hundred jobs a day on a single VM. And yet the
default advice is to add BullMQ or Sidekiq, which means adding Redis, which
means another process to monitor, another persistence model to reason about,
and another backup job nobody wrote.
There is a middle path that fits the same small deployment as the app: keep the
jobs in the same database as the data, and make the queue semantics explicit
with a single atomic operation.
In this post we build a durable job queue with:
- atomic claim (no two workers run the same job),
- fenced leases (a slow worker cannot finish a job after losing it),
- exponential backoff and a maximum attempt count,
- idempotent enqueue,
- at-least-once delivery, and handlers that can live with that.
The database is NonoDB, a single Rust daemon over LMDB. The same pattern works
against any document store with an atomic find-and-update, but NonoDB's
find_and_modify makes the claim a one-liner.
The job document
Everything is a document in the _jobs collection. There is no hidden queue
state.
{ "_id": "01J8Z6Q3K4", "type": "send_email", "status": "pending", "payload": {"template": "invite", "to": "new@example.com"}, "attempts": 0, "max_attempts": 5, "due_at": 1762800000000, "created_at": 1762799940000, "updated_at": 1762799940000 } Status is one of pending, running, done, failed. due_at is a Unix
timestamp in milliseconds; a job is claimable when status == "pending" and
due_at <= now. A claim adds lease_id, lease_until, worker_id, and
started_at.
The two operations that matter are:
- Claim: select the oldest due job and move it to
runningwith a new lease, in one transaction. - Complete or fail: write the result only if the lease still matches.
The claim
In NonoDB, find_and_modify selects one document, applies the update, and
returns the image in the same writer transaction. There is no read-modify-write
gap between "is it pending?" and "mark it running".
import uuid from datetime import datetime, timedelta, timezone def claim_job(jobs, worker_id: str, lease_seconds: int = 60): now = datetime.now(timezone.utc) lease_until = now + timedelta(seconds=lease_seconds) lease_id = uuid.uuid4().hex job = jobs.find_and_modify( { "status": "pending", "due_at": {"$lte": int(now.timestamp() * 1000)}, }, { "$set": { "status": "running", "lease_id": lease_id, "lease_until": int(lease_until.timestamp() * 1000), "worker_id": worker_id, "started_at": int(now.timestamp() * 1000), "updated_at": int(now.timestamp() * 1000), }, "$inc": {"attempts": 1}, }, sort=[("due_at", 1), ("_id", 1)], new=True, ) return job sort=[("due_at", 1), ("_id", 1)] gives FIFO-ish behavior and a deterministic
tie-break. With the compound index below, the claim is an index range scan, not
a collection scan:
jobs.create_index(fields=["status", "due_at"]) jobs.create_index(fields=["status", "lease_until"]) If claim_job returns None, there is nothing due. Sleep, then poll again.
That is the whole scheduler.
The important detail is that this remains correct with N workers. The writer is
serialized; the first transaction flips the job to running, and the next
worker's filter no longer matches it. No SELECT ... FOR UPDATE SKIP LOCKED, no
Lua script, no distributed lock service.
Complete with a fence, not a hope
A lease without fencing is a bug waiting for a slow handler. Suppose worker A
claims a job with a 60-second lease, then pauses for 90 seconds — GC, a slow
HTTP call, a container freeze. Worker B recovers the expired lease and starts
the same job. Now A wakes up and writes "done".
The job runs twice and the bookkeeping lies about who finished it. That is why
every claim writes a fresh lease_id and every completion filters on it:
def complete_job(jobs, job: dict, result: dict | None = None) -> bool: now = int(datetime.now(timezone.utc).timestamp() * 1000) res = jobs.update_result( {"_id": job["_id"], "status": "running", "lease_id": job["lease_id"]}, { "$set": { "status": "done", "result": result or {}, "finished_at": now, "updated_at": now, }, "$unset": {"lease_id": "", "lease_until": ""}, }, ) return res["matched"] == 1 If worker B recovered the lease, it overwrote lease_id. Worker A's update
matches zero documents and returns False. A can log "lost lease, discard
result" and move on. The unique fence per claim is what makes the lease safe,
not the timestamp.
Use find_and_modify with remove=True instead of an update when the job
document should disappear on success.
Fail with backoff
A failed handler should not spin. Return the job to pending with a growing
due_at, up to max_attempts:
def fail_job(jobs, job: dict, error: str) -> str: now = datetime.now(timezone.utc) attempt = job["attempts"] max_attempts = job.get("max_attempts", 5) if attempt >= max_attempts: jobs.update( {"_id": job["_id"], "lease_id": job["lease_id"]}, { "$set": { "status": "failed", "last_error": error[:2000], "finished_at": int(now.timestamp() * 1000), }, "$unset": {"lease_id": "", "lease_until": ""}, }, ) return "failed" delay = min(300, 2 ** attempt) # 2s, 4s, 8s ... capped at 5 min retry_at = now + timedelta(seconds=delay) jobs.update( {"_id": job["_id"], "lease_id": job["lease_id"]}, { "$set": { "status": "pending", "due_at": int(retry_at.timestamp() * 1000), "last_error": error[:2000], "updated_at": int(now.timestamp() * 1000), }, "$unset": {"lease_id": "", "lease_until": ""}, }, ) return "retrying" Again, the filter includes lease_id. A worker that lost its lease cannot mark
someone else's job as failed.
Recover expired leases
Crashed workers leave jobs stuck in running. A periodic sweep moves them back
to pending:
def recover_expired(jobs) -> int: now = int(datetime.now(timezone.utc).timestamp() * 1000) return jobs.update( {"status": "running", "lease_until": {"$lt": now}}, { "$set": {"status": "pending", "updated_at": now}, "$unset": {"lease_id": "", "lease_until": "", "worker_id": ""}, }, multi=True, ) update(multi=True) is one transaction, so the sweep cannot interleave with a
claim that is already in flight. Run it at worker startup and every 30 seconds
while running.
Does the recovered job double-run? It may, if the original worker is still
alive. That is exactly why handlers must be idempotent. The fence protects the
queue state; it cannot un-send an email. At-least-once is the honest contract.
Enqueue idempotently, and atomically with your data
The producer side is a plain insert, but production code usually needs one of
two guarantees.
Deduplicate by business key. Use a deterministic _id and treat a duplicate
as success:
from nonodb import NonoDBError def enqueue(jobs, job_id: str, job_type: str, payload: dict): try: jobs.insert({ "_id": job_id, "type": job_type, "status": "pending", "payload": payload, "attempts": 0, "max_attempts": 5, "due_at": 0, "created_at": 0, "updated_at": 0, }) except NonoDBError as exc: # A duplicate _id is an already-enqueued job, not an error. if exc.code != "duplicate_key": raise A better job_id than a random UUID is f"{job_type}:{entity_id}:{version}".
Then "enqueue twice because the request retried" becomes a no-op.
Write the job with the business change. If an order insert must always
produce a sync job, a crash between the two writes would silently lose work.
NonoDB's write_batch commits both operations or neither:
db.collection("orders").write_batch([ { "op": "insert", "coll": "orders", "doc": {"_id": "o1", "status": "new", "total": 1200}, }, { "op": "insert", "coll": "_jobs", "doc": { "_id": "sync_order:o1", "type": "sync_order", "status": "pending", "payload": {"order_id": "o1"}, "attempts": 0, "max_attempts": 5, "due_at": 0, "created_at": 0, "updated_at": 0, }, }, ]) That is a transactional outbox without a separate outbox table and without a
CDC pipeline.
Run the worker
The repository ships a small Python helper that implements exactly this
contract. A handler module is just functions registered by name:
# jobs_handlers.py from nonodb_worker import Job def send_email(job: Job): payload = job.payload # send it, with an idempotency key derived from job.id return {"sent": True} def register(runner) -> None: runner.register("send_email", send_email) PYTHONPATH=client:workers/python:examples \ NONODB_SOCK=/run/nonodb/nonodb.sock \ python3 -m nonodb_worker \ --module jobs_handlers \ --collection _jobs \ --recover-expired \ --lease-seconds 60 --once claims and runs at most one job, which is handy for a systemd timer if
the jobs are infrequent. A long-running worker with a one-second poll is the
low-latency option. The queue semantics are the same either way.
What you get, and what you give up
You get:
- one data directory and one daemon,
- a queue you can inspect with
find, the CLI, or the GUI, - atomic claim and fenced completion,
- retries and backoff,
- idempotent enqueue and transactional outbox,
- no Redis persistence tuning.
You give up:
- Throughput. One writer thread serializes claims. That is fine for typical background workloads, not for a firehose.
- Exactly-once. Nobody honest offers exactly-once side effects. Assume a handler can run twice and make it idempotent.
- Priority queues with strict fairness. Add a
priorityfield, sort byprioritythendue_at, and accept that low-priority jobs can starve. For strict fairness you need more state than one collection. - Cron-style recurrence. Store a recurring schedule as a job type that enqueues its next run; keep the schedule document separate from the jobs.
When a dedicated queue is the right call
If you already run Redis for caching, adding BullMQ is cheap. If you need
hundreds of thousands of jobs per second, per-job priorities with fair
scheduling, or cross-region workers, use a system built for that.
But if the real requirement is "a few durable background tasks on the same box
as the app", a collection with two indexes and one atomic operation is a lot
less infrastructure than a message broker. The next article takes the same
daemon and adds an audit trail and one-click undo, again without a second
database.