Dead Letter Queues, Poison Messages & Safe Replay Playbooks

A job that fails once is a retry. A job that fails forever is a poison message. Without a dead letter path, that payload retries until it burns CPU, blocks a fair share of the queue, and pages the wrong people at 2 AM. I have seen a single malformed webhook JSON take down a worker fleet that was otherwise correctly idempotent and fairly sharded. Dead letter queues (DLQs), poison detection and safe replay are how you keep async failures from becoming a second outage.

I wrote this with help from AI tools and checked every example and claim myself. At Loop Origin we build Laravel and Node SaaS systems where queues are part of the product surface, not an afterthought. This is Part 3 of Reliable Queues & Async Work in SaaS. Part 1 covered idempotent jobs and exactly-once effects. Part 2 covered backpressure, fairness and tenant isolation. Here we park the bad apples without losing the evidence.

📚 Series Roadmap — Reliable Queues & Async Work in SaaS

• Part 1: Idempotent Jobs & Exactly-Once Effects: Designing Safe Retries
• Part 2: Queue Backpressure, Fairness & Tenant Isolation in Multi-Tenant SaaS
• Part 3 (This Post): Dead Letter Queues, Poison Messages & Safe Replay Playbooks
• Part 4: Delayed Jobs, Schedulers & Clock Skew Gotchas
• Part 5: Observability for Async Work: Correlation IDs & Job Traces
• Part 6: Choosing Redis, SQS or Database Queues for Laravel and Node SaaS

Earlier Loop Origin series: Multi-Tier Caching & Stampede Protection · LLM Gateway & FinOps

What is a dead letter queue and when should a job go there?

Direct answer: A DLQ holds messages that exceeded retry policy or failed validation so workers stop looping on them. Move a job there after bounded attempts (or immediately on non-retryable errors), preserve payload plus failure metadata, alert humans, and replay only after a fix—with the same idempotency keys from Part 1 so replay cannot double-apply side effects.

Failure modes: retry vs poison vs overload

Fairness from Part 2 matters here: a poison message that retries aggressively is a noisy neighbor with a stack trace.

Define non-retryable vs retryable explicitly

Do not let every Exception look the same.

// Laravel-ish classification final class JobErrorClassifier { public function shouldDeadLetter(Throwable $e): bool { return $e instanceof InvalidPayloadException || $e instanceof SchemaValidationException || $e instanceof UnrecoverableBusinessRuleException; } public function shouldRetry(Throwable $e): bool { return $e instanceof TransientHttpException || $e instanceof DeadlockException || $e instanceof TimeoutException; } } 

// Node / BullMQ sketch function classify(err: unknown): "retry" | "dlq" | "ignore" { if (err instanceof ValidationError) return "dlq"; if (err instanceof TransientNetworkError) return "retry"; if (err instanceof DuplicateIgnoredError) return "ignore"; // idempotent no-op return "retry"; // default bounded } 

Wire classification into your failed handler so poison does not consume the full retry budget before landing in the DLQ.

Laravel: failed jobs table as a practical DLQ

Laravel's failed_jobs table is a built-in dead letter store when you configure retry_after, $tries, and failed() hooks deliberately.

class ProcessWebhook implements ShouldQueue { public int $tries = 5; public array $backoff = [10, 30, 60, 120, 300]; public function handle(WebhookPayload $payload): void { $data = WebhookValidator::parse($payload->raw); // throws InvalidPayloadException // side effects under idempotency key from Part 1 $this->processor->apply($data, $payload->idempotencyKey); } public function failed(Throwable $e): void { Log::error('webhook.dead_lettered', [ 'idempotency_key' => $this->payload->idempotencyKey ?? null, 'tenant_id' => $this->payload->tenantId ?? null, 'error' => $e->getMessage(), ]); // optional: notify Slack, write to quarantine table, metric++ } } 

# Inspect and replay carefully php artisan queue:failed php artisan queue:retry <id> # one php artisan queue:retry all # dangerous—never on an unfixed poison php artisan queue:flush # only after export/backup 

For Horizon, watch the Failed tab the same way you watch throughput. A rising failed count with flat throughput is often poison, not load.

Node: BullMQ failed set + explicit DLQ queue

import { Queue, Worker } from "bullmq"; const main = new Queue("webhooks"); const dlq = new Queue("webhooks-dlq"); const worker = new Worker("webhooks", async (job) => { const parsed = parseWebhook(job.data); // throws ValidationError await applyOnce(parsed, job.data.idempotencyKey); }, { connection: redis }); worker.on("failed", async (job, err) => { if (!job) return; const max = job.opts.attempts ?? 5; if (job.attemptsMade >= max || err instanceof ValidationError) { await dlq.add("dead", { original: job.data, err: String(err), failedAt: new Date().toISOString(), attempts: job.attemptsMade, }); // metric: webhooks_dlq_total{tenant=...} } }); 

SQS users: configure RedrivePolicy with maxReceiveCount and a dedicated DLQ ARN. The product rules stay identical—bounded retries, preserve body, alert, replay with idempotency.

Safe replay playbooks (the part teams skip)

Replaying is a deploy-like action. Treat it as one.

Playbook A — Single poison after a code fix

1. Ship the fix; confirm on a staging replay of the saved payload.
2. Re-validate the payload offline (schema + business rules).
3. Replay one message with the same idempotency key.
4. Confirm side effects once (DB row, Stripe charge id, email message id).
5. Only then batch-retry siblings from the same bug class.

Playbook B — Downstream outage, not poison

1. Do not DLQ every failure if the classifier says transient.
2. Pause or delay consumers (Part 2 backpressure) until the dependency is healthy.
3. Let normal retries drain; use DLQ only for stragglers that still fail after recovery.

Playbook C — Mass invalid import

1. Quarantine the batch (move to DLQ / quarantine table).
2. Sample 20 payloads; find the schema break.
3. Fix producer; backfill a corrected batch under new idempotency keys or same keys if apply is truly idempotent.
4. Never retry all blindly on 50k failed rows.

REPLAY CHECKLIST (print next to the runbook) [ ] Fix merged and deployed [ ] Payload re-validated offline [ ] Idempotency key strategy decided (same vs new) and written down [ ] Metrics/dashboards open for double-apply detection [ ] One-message canary succeeded [ ] Tenant owner notified if user-visible [ ] DLQ entry archived, not silently deleted 

When we implement these controls on client systems—whether Laravel development or server setup and maintenance—we treat DLQ depth as a product SLO, same as API error rate. At Loop Origin that usually means a pager on dlq_depth > N for 15m, not a forgotten failed_jobs table.

What to store with the dead letter

Minimum metadata:

• Original payload (or object-store pointer if large)
• Idempotency key, tenant id, job name, correlation id
• Exception class + message + first stack frames
• Attempt count, first failure time, last failure time
• Classifier decision (poison vs exhausted_retries)

Dropping the payload and keeping only "job failed" makes replay guesswork.

Metrics that catch poison early

• jobs_failed_total by job name, tenant, error class
• dlq_depth and dlq_age_p95
• retry_attempts_p95 (poison often pegs the max)
• Ratio of DLQ vs successful completes per hour

Alert on new error class spikes, not only raw depth—depth alone can be a stalled replay backlog you already know about.

Diagnostic checklist

1. Is there a bounded $tries / attempts on every side-effecting job?
2. Are validation errors routed to DLQ without burning the full retry budget?
3. Do failed handlers log tenant id + idempotency key + error class?
4. Is DLQ depth on a dashboard with a page threshold?
5. Can you export a failed payload without SSH archaeology?
6. Does replay reuse Part 1 idempotency so double-apply is impossible?
7. Is queue:retry all / bulk replay restricted (break-glass only)?
8. Do poison retries respect Part 2 per-tenant concurrency caps?
9. After downstream outages, do you pause intake instead of DLQ-storming?
10. Have you rehearsed a one-message canary replay in staging this quarter?

FAQ

Is Laravel failed_jobs "enough" to call it a DLQ?

Yes for many apps, if you alert on it, preserve payloads, and have a replay playbook. At higher volume you may add a dedicated quarantine table or broker DLQ for retention and multi-service access.

Should every failure go to the DLQ after one try?

No. Transient errors need retries with backoff. DLQ is for poison and for exhausted retry budgets.

How do idempotency keys interact with replay?

Prefer the same key when the operation must happen at most once. Use a new key only when you intentionally want a new business effect (for example a corrected re-import line). Write that choice down before bulk replay.

What about messages that partially applied then threw?

That is why Part 1 exists. Make handlers safe to resume; DLQ replay of a half-applied job without idempotency is how you double-charge.

Do we delete DLQ entries after success?

Archive or mark resolved; keep enough history for audit. Immediate hard delete removes forensic value.

Where do delayed jobs fit?

Part 4. Poison detection still applies to delayed/scheduled work—expiry and clock skew create their own failure classes.

Next in this series, Part 4 covers delayed jobs, schedulers and clock skew gotchas so "run at 09:00" means the same thing on every worker.

Follow & Connect

• Website: Loop Origin
• LinkedIn: Loop Origin
• X: Muhammad Ali Mughal (@AliMughal887)
• GitHub: Loop Origin (@LoopOrigin)