SQS dead-letter queue: alarm, triage and redrive

An SQS dead-letter queue takes two minutes to set up: one entry in the redrive policy and you are done. It gets serious on the day a few hundred orders are sitting in it. Who notices? What is in those messages? And how do you get them back into processing without knocking over the downstream API again?

I care about this topic because in many serverless architectures the DLQ ends up as a ticked box on a checklist. It exists, nobody looks inside, there is no alarm and no plan for the way back. At that point it turns into a place where messages quietly disappear after a few days. In this article you will therefore use the Serverless Framework to build a queue with a DLQ that raises an alarm, can be inspected safely and is redriven in a controlled way by a script.

In short

An SQS dead-letter queue collects messages that were not processed after maxReceiveCount receives. It only becomes a safety net once you add an alarm, a way to inspect messages without deleting them and a throttled redrive with StartMessageMoveTask. The biggest pitfall is retention: for standard queues the time spent in the source queue counts too, so the DLQ has to keep messages longer.

What you will learn in this article

• How to align visibility timeout, maxReceiveCount and retention
• How to set up a queue, a DLQ and two alarms with the Serverless Framework
• How to inspect DLQ messages and group them by cause without deleting them
• How to throttle, monitor and cancel a redrive with StartMessageMoveTask

What is an SQS dead-letter queue?

A dead-letter queue (DLQ) is a perfectly ordinary SQS queue that another queue uses as a holding area for problem cases. The two are connected through the redrive policy of the source queue. It contains two settings: the ARN of the DLQ and the value maxReceiveCount. This value defines how often a message may be received without being deleted. Once that limit is exceeded, SQS moves the message to the DLQ.

The life cycle of a message helps to understand this. A consumer, in our example a Lambda function, picks it up. From that moment on, the message is invisible to everyone else for the duration of the visibility timeout. If the consumer deletes it within that time, it is done. If the consumer fails, the message reappears once the visibility timeout expires, and its receive counter ApproximateReceiveCount goes up by one.

AWS calls the opposite direction a redrive: messages travel from the DLQ back to their source queue or to another queue of your choice. The API for this is StartMessageMoveTask, so you no longer need a home-grown copy script. The animated diagram shows the whole cycle using a fictional ticket shop that accompanies us through the article.

🎬 How a message ends up in the SQS dead-letter queue – this animated diagram only runs on my site: watch it on sascha-ms.de

Configuring an SQS dead-letter queue correctly

Three values decide whether your SQS dead-letter queue works reliably. Each one is quick to explain, but the way they interact is often overlooked.

Why the DLQ has to keep messages longer

The third point is the trickiest. With a standard queue, a message keeps its original timestamp when it moves to the DLQ. An example from the AWS documentation makes this tangible: if a message spends one day in the source queue and then lands in a DLQ with four days of retention, it is gone from the DLQ after three days. With FIFO queues, on the other hand, the clock restarts when the message is moved.

For our ticket shop this means: the source queue keeps messages for four days, the DLQ for the maximum of 14 days. Even a message that used up its full time in the source queue therefore stays in the DLQ for at least ten more days. That covers a long weekend plus a holiday handover.

By the way, the metric ApproximateAgeOfOldestMessage of a standard DLQ shows the age since the move, not since the original send. For a “nobody is taking care of this” alarm that is exactly right, but for the question “when does this message expire?” it is misleading.

Step by step: an SQS dead-letter queue with the Serverless Framework

We rebuild the ticket shop flow: the shop puts orders into a queue, and a Lambda function issues the tickets through an external ticket API. The example is entirely made up and small enough to rebuild in an hour.

Prerequisites

• Node.js 22 or newer and npm
• an AWS account with AWS CLI v2 set up (a profile with permissions for CloudFormation, Lambda, SQS, SNS and CloudWatch)
• Serverless Framework v4 (npm install -g serverless); on first start it asks you to sign in to Serverless Inc.
• Basic knowledge of Lambda and SQS. I explain partial batch responses in detail in SQS partial batch response.

Step 1: Create the project

mkdir sqs-dlq-redrive && cd sqs-dlq-redrive npm init -y npm pkg set type=module npm install @aws-sdk/client-sqs npm install -D typescript tsx vitest @types/aws-lambda @types/node 

The tsconfig.json uses "module": "NodeNext" and "strict": true. You can find it together with all other files in the example repository; the link is at the end of the article.

Step 2: Validate the order

Before we talk about the DLQ, we need a clear distinction. Some errors are temporary: the ticket API responds with 503, or a timeout kicks in. Others are permanent: the message is broken JSON or wants to book 50 seats. A retry only helps with the first kind. That is why validation gets its own error type, which we reuse later for the analysis.

// src/order.ts export interface TicketOrder { orderId: string; eventId: string; seats: number; email: string; } export class InvalidOrderError extends Error { override name = 'InvalidOrderError'; } // Checks the message body. Errors here are permanent: // retrying the same message will never fix them. export function parseOrder(body: string): TicketOrder { let data: unknown; try { data = JSON.parse(body); } catch { throw new InvalidOrderError('Body is not valid JSON'); } if (typeof data !== 'object' || data === null) { throw new InvalidOrderError('Body is not an object'); } const o = data as Record<string, unknown>; if (typeof o.orderId !== 'string' || o.orderId === '') { throw new InvalidOrderError('orderId is missing'); } if (typeof o.eventId !== 'string' || o.eventId === '') { throw new InvalidOrderError('eventId is missing'); } if (!Number.isInteger(o.seats) || (o.seats as number) < 1 || (o.seats as number) > 10) { throw new InvalidOrderError('seats must be between 1 and 10'); } if (typeof o.email !== 'string' || !o.email.includes('@')) { throw new InvalidOrderError('email is invalid'); } return { orderId: o.orderId, eventId: o.eventId, seats: o.seats as number, email: o.email }; } 

Step 3: Handler with partial batch response

The handler processes every message individually and reports only the failed ones. This way a whole batch of ten orders does not end up in the DLQ just because one of them was broken. The log contains metadata such as the message ID and the receive count, but never the full body, because it contains an email address.

// src/handler.ts import type { SQSBatchResponse, SQSEvent } from 'aws-lambda'; import { InvalidOrderError, parseOrder, type TicketOrder } from './order.js'; export type IssueTickets = (order: TicketOrder) => Promise<void>; export function createHandler(issueTickets: IssueTickets) { return async (event: SQSEvent): Promise<SQSBatchResponse> => { const batchItemFailures: SQSBatchResponse['batchItemFailures'] = []; for (const record of event.Records) { try { const order = parseOrder(record.body); await issueTickets(order); } catch (error) { // Only log metadata, never the full body (it contains an email address) console.error(JSON.stringify({ level: 'ERROR', messageId: record.messageId, receiveCount: Number(record.attributes.ApproximateReceiveCount), permanent: error instanceof InvalidOrderError, error: error instanceof Error ? error.message : String(error), })); // Only this message goes back to the queue, the rest of the batch counts as done batchItemFailures.push({ itemIdentifier: record.messageId }); } } return { batchItemFailures }; }; } const issueTicketsViaApi: IssueTickets = async (order) => { const url = process.env.TICKET_API_URL; if (!url) throw new Error('TICKET_API_URL is not set'); const response = await fetch(url, { method: 'POST', headers: { 'content-type': 'application/json', 'idempotency-key': order.orderId }, body: JSON.stringify(order), signal: AbortSignal.timeout(5_000), }); if (!response.ok) { throw new Error(`Ticket API responded with status ${response.status}`); } }; export const handler = createHandler(issueTicketsViaApi); 

The idempotency-key header is not decoration. A redrive replays messages that may already have been half processed. If the ticket API recognises the order number, it does not issue duplicate tickets.

Step 4: Queue, DLQ and alarms in serverless.yml

Now for the core. The function has a 30-second timeout, so the source queue gets a 180-second visibility timeout. The redrive policy points to the DLQ and allows five receives. On top of that come an SNS topic for notifications and two alarms.

# serverless.yml service: ticketshop-orders frameworkVersion: '4' provider: name: aws runtime: nodejs24.x region: eu-central-1 memorySize: 512 timeout: 30 environment: TICKET_API_URL: ${param:ticketApiUrl, 'https://tickets.example.com/api/issue'} functions: issueTickets: handler: src/handler.handler events: - sqs: arn: !GetAtt TicketOrdersQueue.Arn batchSize: 10 functionResponseType: ReportBatchItemFailures resources: Resources: TicketOrdersQueue: Type: AWS::SQS::Queue Properties: QueueName: ${self:service}-${sls:stage}-orders VisibilityTimeout: 180 # 6 x function timeout (30 s)  MessageRetentionPeriod: 345600 # 4 days  RedrivePolicy: deadLetterTargetArn: !GetAtt TicketOrdersDlq.Arn maxReceiveCount: 5 TicketOrdersDlq: Type: AWS::SQS::Queue Properties: QueueName: ${self:service}-${sls:stage}-orders-dlq MessageRetentionPeriod: 1209600 # 14 days, longer than the source queue  AlertTopic: Type: AWS::SNS::Topic AlertSubscription: Type: AWS::SNS::Subscription Properties: TopicArn: !Ref AlertTopic Protocol: email Endpoint: ${param:alertEmail, 'team@example.com'} DlqNotEmptyAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmName: ${self:service}-${sls:stage}-dlq-not-empty AlarmDescription: Messages in the DLQ. Runbook - peek, fix the cause, start a redrive. Namespace: AWS/SQS MetricName: ApproximateNumberOfMessagesVisible Dimensions: - Name: QueueName Value: !GetAtt TicketOrdersDlq.QueueName Statistic: Maximum Period: 60 EvaluationPeriods: 1 Threshold: 0 ComparisonOperator: GreaterThanThreshold TreatMissingData: notBreaching AlarmActions: [!Ref AlertTopic] OKActions: [!Ref AlertTopic] DlqForgottenAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmName: ${self:service}-${sls:stage}-dlq-forgotten AlarmDescription: Oldest DLQ message older than 3 days. Nobody has taken care of it. Namespace: AWS/SQS MetricName: ApproximateAgeOfOldestMessage Dimensions: - Name: QueueName Value: !GetAtt TicketOrdersDlq.QueueName Statistic: Maximum Period: 300 EvaluationPeriods: 1 Threshold: 259200 ComparisonOperator: GreaterThanThreshold TreatMissingData: notBreaching AlarmActions: [!Ref AlertTopic] Outputs: OrdersQueueUrl: Value: !Ref TicketOrdersQueue DlqUrl: Value: !Ref TicketOrdersDlq DlqArn: Value: !GetAtt TicketOrdersDlq.Arn 

Step 5: Deploy and test with a broken message

First the unit tests run, then the deployment. Replace the email address with your own and confirm the SNS subscription email afterwards, otherwise no alarm will arrive.

npx vitest run serverless deploy --stage dev --param="alertEmail=you@example.com" 

The tests should finish with 4 passed. After the deployment, the framework prints the outputs OrdersQueueUrl, DlqUrl and DlqArn. Now send a deliberately broken order to the source queue:

aws sqs send-message \ --queue-url <OrdersQueueUrl> \ --message-body '{"orderId":"o-1042","eventId":"concert-42","seats":50,"email":"kim@example.com"}' 

Expected result: the function fails five times, each attempt one visibility timeout of three minutes apart. After roughly a quarter of an hour the message is in the DLQ, and shortly afterwards the alarm email arrives. That takes a while, and it is meant to: exactly this gap gives a briefly disrupted ticket API the chance to recover.

Alarm: who notices that the SQS dead-letter queue is filling up?

The first alarm fires as soon as a single message is visible in the SQS dead-letter queue. For other metrics a threshold of zero would be overkill. Here, however, every message stands for an order whose tickets were never issued. That is a real business problem. How to build alarms that people take seriously is something I describe in Avoid alert fatigue: alarms that really matter.

The second alarm is my favourite, because it measures an organisational problem. It fires when the oldest message has been sitting in the DLQ for more than three days. That means someone saw the first alarm and clicked it away. At the latest now, the topic belongs in the daily stand-up.

A short runbook keeps the team from improvising when an alarm fires. The following diagram shows the flow that the next two sections fill with tools.

Five steps from the alarm to an empty DLQ. Invalid messages take a separate path.

How to write and maintain such a runbook is covered in Writing a runbook: template and examples.

Inspecting messages in the SQS dead-letter queue

The most common mistake after an alarm from the SQS dead-letter queue is the reflex to “just redrive it”. If the cause still exists, the messages simply run through the function five more times and end up back in the DLQ. So you look inside first.

You can do this in the SQS console via “Send and receive messages” and “Poll for messages”. With more than a handful of messages a script is more practical. It reads messages with ReceiveMessage but deletes nothing. Thanks to a short visibility timeout of 60 seconds, they reappear afterwards. It also requests the system attributes SentTimestamp and DeadLetterQueueSourceArn, so you can see when and from which queue a message came.

// scripts/dlq-peek.ts // Reads messages from a dead-letter queue WITHOUT deleting them // and groups them by cause. // Usage: npx tsx scripts/dlq-peek.ts <dlq-url> [max-messages] import { ReceiveMessageCommand, SQSClient, type Message } from '@aws-sdk/client-sqs'; import { parseOrder } from '../src/order.js'; const [queueUrl, maxArg = '50'] = process.argv.slice(2); if (!queueUrl) { console.error('Usage: npx tsx scripts/dlq-peek.ts <dlq-url> [max-messages]'); process.exit(1); } const maxMessages = Math.min(Number(maxArg), 200); const sqs = new SQSClient({}); // Mask email addresses before anything ends up in the terminal or a ticket const mask = (text: string) => text.replace(/[^\s"@]+@[^\s"@]+/g, '***@***'); function classify(message: Message): string { try { parseOrder(message.Body ?? ''); return 'valid – probably a temporary error (redrive candidate)'; } catch (error) { return `invalid: ${(error as Error).message}`; } } const seen = new Map<string, Message>(); let emptyPolls = 0; while (seen.size < maxMessages && emptyPolls < 2) { const { Messages = [] } = await sqs.send(new ReceiveMessageCommand({ QueueUrl: queueUrl, MaxNumberOfMessages: 10, WaitTimeSeconds: 2, // Short visibility: the messages reappear after 60 seconds VisibilityTimeout: 60, MessageSystemAttributeNames: ['ApproximateReceiveCount', 'SentTimestamp', 'DeadLetterQueueSourceArn'], })); if (Messages.length === 0) emptyPolls++; for (const m of Messages) if (m.MessageId) seen.set(m.MessageId, m); } const groups = new Map<string, Message[]>(); for (const m of seen.values()) { const key = classify(m); groups.set(key, [...(groups.get(key) ?? []), m]); } console.log(`${seen.size} message(s) peeked, nothing deleted.\n`); for (const [cause, messages] of [...groups].sort((a, b) => b[1].length - a[1].length)) { const sample = messages[0]; const sent = new Date(Number(sample.Attributes?.SentTimestamp)).toISOString(); console.log(`${messages.length}× ${cause}`); console.log(` Example ${sample.MessageId} (sent ${sent}, source ${sample.Attributes?.DeadLetterQueueSourceArn ?? 'unknown'})`); console.log(` ${mask(sample.Body ?? '').slice(0, 160)}\n`); } 

The script uses the same parseOrder function as the handler. As a result, it automatically sorts messages into two camps: valid orders that probably failed because of a temporary error, and invalid ones with a specific reason. Email addresses are masked before they reach the terminal. The output looks like this, for example:

npx tsx scripts/dlq-peek.ts <DlqUrl> 7 message(s) peeked, nothing deleted. 5× valid – probably a temporary error (redrive candidate) Example 4f1c2b7e-... (sent 2026-11-10T21:14:03.120Z, source arn:aws:sqs:eu-central-1:123456789012:ticketshop-orders-dev-orders) {"orderId":"o-1043","eventId":"concert-42","seats":2,"email":"***@***"} 2× invalid: seats must be between 1 and 10 Example 9a03d1c4-... (sent 2026-11-10T21:02:44.871Z, source arn:aws:sqs:eu-central-1:123456789012:ticketshop-orders-dev-orders) {"orderId":"o-1042","eventId":"concert-42","seats":50,"email":"***@***"} 

With this pictur...