Lambda Destinations: Catch Failed Async Events

Lambda destinations are one of the most underrated settings in AWS Lambda, in my view. As soon as a service like S3, SNS or EventBridge invokes your function, nobody waits for the response. If something goes wrong, Lambda tries twice more and then quietly throws the event away. No error in the frontend, no ticket, just a line in a log that nobody reads.

That is why I check two things first for every asynchronously invoked function: how often does it retry, and where does an event end up when it fails anyway? In this article you build the answer step by step: a function with limited retries, a failure queue as the on-failure destination, a small script that replays failed events once the fix is in, and tests to go with it.

In short

For asynchronous invocations, Lambda retries failed events twice by default and discards them afterwards. With Lambda destinations, every event that finally fails goes to an SQS queue together with its error message, so you can reprocess it later on purpose. The most common pitfall: destinations do nothing for synchronous invocations or for functions that consume an SQS queue.

What you will learn in this article

• How Lambda buffers and retries asynchronous invocations internally
• When Lambda destinations are a better choice than a dead-letter queue
• How to set up a failure queue with the Serverless Framework and with AWS CDK
• How to read, analyze and replay failed events
• Typical mistakes where destinations simply do nothing

What happens during an asynchronous Lambda invocation?

When you invoke a function synchronously, for example through API Gateway, the caller waits for the result and gets an error back directly. An asynchronous invocation works differently: Lambda accepts the event, responds right away with status code 202 and puts it into an internal queue. From there, Lambda invokes your function as soon as there is enough capacity. The caller never hears about the result.

If the function fails, say because your code throws an exception or hits the timeout, Lambda retries the invocation. According to the docs, it makes two more attempts by default: the second after about one minute, the third after another two minutes. Throttling (not enough free concurrency) and system errors are handled differently. In that case the event stays in the queue, and Lambda keeps trying for up to six hours, with intervals growing from one second to five minutes.

After the last attempt, or once the event is too old, Lambda discards it. Without further configuration, all that remains is the AsyncEventsDropped metric. On top of that, the docs point out that the same event can arrive more than once even without an error, because the internal queue is only eventually consistent. So your function needs to be idempotent, as I showed in the article on idempotency in AWS Lambda.

Which services invoke Lambda asynchronously?

Amazon S3 (for example for new objects), Amazon SNS and Amazon EventBridge are among the services that invoke your function asynchronously. You can trigger this yourself with the Event invocation type, for instance via the AWS CLI or the SDK. The pattern “function A kicks off function B and carries on” works exactly like this in many projects.

🎬 How Lambda processes an asynchronous invocation – this animated diagram only runs on my site: watch it on sascha-ms.de

Lambda destinations or dead-letter queue?

There are two safety nets for discarded events. The older option is a dead-letter queue (DLQ) on the function: an SQS queue or SNS topic that receives the original event content plus a few attributes. The newer option is Lambda destinations. They send a complete record with request, response and error message to a target of your choice, on success, on failure or both.

My recommendation is clear: for new functions I use destinations. The record already contains the error message and the number of attempts, which saves you the trip to the logs while debugging. On the other hand, you do not have to rebuild an existing DLQ right away if it works and someone monitors it. Configuring both at the same time is possible, but it makes troubleshooting more confusing.

Which targets make sense?

For failures, an SQS queue is almost always the best choice. Messages stay for up to 14 days, and you can inspect them calmly and replay them on purpose. EventBridge is a good fit for success notifications that other parts of the system should react to. SNS is tricky, because a message there can be at most 256 KB, while asynchronous events can be up to 1 MB. For large events, the docs therefore point to SQS or S3. You can also look up which service fits which purpose in the AWS service finder.

When Lambda destinations do not apply

The most common misconception: destinations are not a general safety net for every Lambda function. They apply to asynchronous invocations and, for failures only, to event source mappings on streams. An event source mapping is the mechanism Lambda uses to poll a source such as SQS, Kinesis or DynamoDB Streams itself and feed your function with batches.

Destinations apply to asynchronous invocations and, for failures only, to stream sources.

So if your function consumes an SQS queue, a destination on the function does nothing. There, the dead-letter queue belongs on the source queue, combined with a redrive policy and ideally with partial batch responses. The article on SQS partial batch response shows how. For synchronous invocations through API Gateway or function URLs, the caller gets the error back directly and has to decide what happens next.

Step by step: a failure queue with the Serverless Framework

The example is a fictional bakery that takes preorders online. After each order, a function is invoked asynchronously to email a confirmation to the customer. If the mail service is down or the address is invalid, the order should not get lost but land in a failure queue.

Prerequisites

• Node.js 22 or newer and npm
• an AWS account with credentials configured for the AWS CLI
• Serverless Framework v4 (npm i -g serverless, using it requires signing in)
• basic TypeScript knowledge

Step 1: Create the project

Create a folder and install the dependencies. Serverless Framework v4 builds TypeScript handlers itself with esbuild, so you do not need an extra plugin.

mkdir bakery-preorders && cd bakery-preorders npm init -y npm i @aws-sdk/client-lambda @aws-sdk/client-sqs npm i -D typescript @types/node @types/aws-lambda vitest tsx 

The tsconfig.json is deliberately simple. allowImportingTsExtensions allows imports ending in .ts, which tsx and Vitest run directly.

{ "compilerOptions": { "target": "ES2023", "module": "NodeNext", "moduleResolution": "NodeNext", "strict": true, "skipLibCheck": true, "types": ["node"], "noEmit": true, "allowImportingTsExtensions": true }, "include": ["src", "scripts", "test"] } 

Step 2: Write the handler

The handler receives the preorder as its event and sends the email. Sending mail is a placeholder here. Addresses ending in .invalid deliberately trigger an error, so you can see the destination in action right away.

// src/confirm-preorder.ts export interface Preorder { orderId: string; customerEmail: string; items: string[]; pickup: string; // ISO date, e.g. "2026-10-31" } // Placeholder for a real mail service. Addresses ending in ".invalid" // simulate a permanent error so you can see the destination at work. async function sendMail(to: string, subject: string): Promise<void> { if (to.endsWith(".invalid")) { throw new Error(`Mail service rejected address: ${to}`); } console.log(JSON.stringify({ msg: "Mail sent", to, subject })); } export async function handler(event: Preorder): Promise<{ orderId: string; status: string }> { if (!event?.orderId || !event.customerEmail) { // An error here ends up in the on-failure destination after all attempts throw new Error("Invalid preorder: orderId or customerEmail missing"); } await sendMail( event.customerEmail, `Your preorder ${event.orderId} is ready for pickup on ${event.pickup}`, ); return { orderId: event.orderId, status: "confirmed" }; } 

What matters is that the function really throws an error when something goes wrong. If you catch every exception and return a result anyway, Lambda treats the invocation as successful, and no destination fires.

Step 3: Retries and destination in serverless.yml

Now for the core of it. With maximumRetryAttempts you define how often Lambda retries after a function error (0 to 2). The maximumEventAge value sets, in seconds, how old an event may get (60 seconds to 6 hours). Under destinations.onFailure you attach the failure queue.

# serverless.yml service: bakery-preorders provider: name: aws runtime: nodejs22.x region: eu-central-1 architecture: arm64 memorySize: 256 timeout: 10 functions: confirmPreorder: handler: src/confirm-preorder.handler # Asynchronous invocations: at most 1 retry, events at most 1 hour old maximumRetryAttempts: 1 maximumEventAge: 3600 destinations: onFailure: type: sqs arn: Fn::GetAtt: [FailedConfirmations, Arn] resources: Resources: FailedConfirmations: Type: AWS::SQS::Queue Properties: QueueName: preorder-confirmation-failed MessageRetentionPeriod: 1209600 # 14 days, the SQS maximum SqsManagedSseEnabled: true FailureAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmDescription: Failed confirmations are waiting to be handled Namespace: AWS/SQS MetricName: ApproximateNumberOfMessagesVisible Dimensions: - Name: QueueName Value: !GetAtt FailedConfirmations.QueueName Statistic: Maximum Period: 300 EvaluationPeriods: 1 Threshold: 0 ComparisonOperator: GreaterThanThreshold TreatMissingData: notBreaching Outputs: FailureQueueUrl: Value: !Ref FailedConfirmations 

Because the queue ARN is resolved through Fn::GetAtt, the Serverless Framework needs the type: sqs setting. That is how it can grant the function the matching sqs:SendMessage permission. The alarm fires as soon as a single message sits in the failure queue. Connect it to an SNS topic or your chat channel, otherwise it alarms into the void.

Step 4: Deploy and trigger an error

serverless deploy aws lambda invoke \ --function-name bakery-preorders-dev-confirmPreorder \ --invocation-type Event \ --cli-binary-format raw-in-base64-out \ --payload '{"orderId":"B-1042","customerEmail":"customer@example.invalid","items":["Rye bread"],"pickup":"2026-10-31"}' \ response.json 

The CLI responds right away with "StatusCode": 202. That only means Lambda has accepted the event. After the first failure, Lambda waits about a minute and tries a second time. After that, the record is in the queue. serverless info --verbose shows you the queue URL under outputs.

aws sqs receive-message \ --queue-url <FailureQueueUrl> \ --query "Messages[0].Body" --output text 

Step 5: Understand the record

The message contains the so-called invocation record. For our example it looks roughly like this:

{ "version": "1.0", "timestamp": "2026-10-27T17:05:12.000Z", "requestContext": { "requestId": "4f1c2a9e-0000-4000-8000-000000000001", "functionArn": "arn:aws:lambda:eu-central-1:111122223333:function:bakery-preorders-dev-confirmPreorder:$LATEST", "condition": "RetriesExhausted", "approximateInvokeCount": 2 }, "requestPayload": { "orderId": "B-1042", "customerEmail": "customer@example.invalid", "items": ["Rye bread"], "pickup": "2026-10-31" }, "responseContext": { "statusCode": 200, "executedVersion": "$LATEST", "functionError": "Unhandled" }, "responsePayload": { "errorType": "Error", "errorMessage": "Mail service rejected address: customer@example.invalid" } } 

You will need three fields almost every time. requestPayload holds the original event you restart with later. condition tells you why the event was given up, here because all attempts were used. And responsePayload contains your function’s error message. So you do not have to pick this structure apart by hand everywhere, here is a small module with types and a validation function:

// src/invocation-record.ts // Shape of the record Lambda sends to a destination export interface InvocationRecord<TRequest = unknown> { version: string; timestamp: string; requestContext: { requestId: string; functionArn: string; condition: string; // e.g. "RetriesExhausted" or "Success" approximateInvokeCount: number; }; requestPayload: TRequest; responseContext?: { statusCode: number; executedVersion: string; functionError?: string; }; responsePayload?: unknown; } function isObject(value: unknown): value is Record<string, unknown> { return typeof value === "object" && value !== null; } // Validates an SQS message body and returns the typed record export function parseInvocationRecord(body: string): InvocationRecord { let data: unknown; try { data = JSON.parse(body); } catch { throw new Error("Message is not valid JSON"); } if (!isObject(data) || !isObject(data.requestContext) || !("requestPayload" in data)) { throw new Error("Message is not a Lambda invocation record"); } const ctx = data.requestContext; if (typeof ctx.functionArn !== "string" || typeof ctx.condition !== "string") { throw new Error("requestContext is incomplete"); } return data as unknown as InvocationRecord; } // Short, readable failure description for logs and tickets export function describeFailure(record: InvocationRecord): string { const payload = record.responsePayload; const message = isObject(payload) && typeof payload.errorMessage === "string" ? payload.errorMessage : "no error message"; return `${record.requestContext.condition} after ${record.requestContext.approximateInvokeCount} attempt(s): ${message}`; } 

Step 6: Replay failed events

A failure queue is only as good as the way back. Once the mail service is reachable again or the bug is fixed, you want to reprocess the collected events. The following script reads the queue, prints every error message and invokes the function asynchronously again with the original event. Only then does it delete the message.

// scripts/replay-failures.ts // Reads failed events from the queue and invokes the function again asynchronously. // Usage: npx tsx scripts/replay-failures.ts <QueueUrl> [--dry-run] import { InvokeCommand, LambdaClient } from "@aws-sdk/client-lambda"; import { DeleteMessageCommand, ReceiveMessageCommand, SQSClient } from "@aws-sdk/client-sqs"; import { describeFailure, parseInvocationRecord } from "../src/invocation-record.ts"; const [queueUrl, flag] = process.argv.slice(2); if (!queueUrl) { console.error("Please provide the queue URL"); process.exit(1); } const dryRun = flag === "--dry-run"; const sqs = new SQSClient({}); const lambda = new LambdaClient({}); let replayed = 0; while (true) { const { Messages = [] } = await sqs.send( new ReceiveMessageCommand({ QueueUrl: queueUrl, MaxNumberOfMessages: 10, WaitTimeSeconds: 2 }), ); if (Messages.length === 0) break; for (const message of Messages) { const record = parseInvocationRecord(message.Body ?? ""); console.log(`${record.requestContext.requestId}: ${describeFailure(record)}`); if (dryRun) continue; await lambda.send( new InvokeCommand({ FunctionName: record.requestContext.functionArn.replace(/:\$LATEST$/, ""), InvocationType: "Event", Payload: new TextEncoder().encode(JSON.stringify(record.requestPayload)), }), ); await sqs.send(new DeleteMessageCommand({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle })); replayed++; } if (dryRun) break; } console.log(dryRun ? "Dry run finished, nothing changed" : `${replayed} event(s) sent again`); 

# First look at what is in there npx tsx scripts/replay-failures.ts <FailureQueueUrl> --dry-run # Then actually send them again npx tsx scripts/replay-failures.ts <FailureQueueUrl> 

During the dry run, messages are read but not deleted. They stay invisible for the queue’s visibility timeout (30 seconds by default) and then show up again. A replayed invocation that fails again simply lands in the queue again. So you lose nothing, but you should not replay blindly in a loop while the cause is still there.

Step 7: Test

The tests check reading the record, the failure description and the handler’s behavior. They run without AWS access.

// test/invocation-record.test.ts import { describe, expect, it } from "vitest"; import { describeFailure, parseInvocationRecord } from "../src/invocation-record.ts"; import { handler } from "../src/confirm-preorder.ts"; const failureBody = JSON.stringify({ version: "1.0", timestamp: "2026-10-27T17:05:12.000Z", requestContext: { requestId: "4f1c2a9e-0000-4000-8000-000000000001", functionArn: "arn:aws:lambda:eu-central-1:111122223333:function:bakery-preorders-dev-confirmPreorder:$LATEST", condition: "RetriesExhausted", approximateInvokeCount: 2, }, requestPayload: { orderId: "B-1042", customerEmail: "customer@example.invalid", items: ["Rye bread"], pickup: "2026-10-31" }, responseContext: { statusCode: 200, executedVersion: "$LATEST", functionError: "Unhandled" }, responsePayload: { errorType: "Error", errorMessage: "Mail service rejected address: customer@example.invalid" }, }); describe("parseInvocationRecord", () => { it("reads a failure record", () => { const record = parseInvocationRecord(failureBody); expect(<...