Fan-out with SNS and SQS: One Event, Many Consumers

Fan-out with SNS and SQS is the pattern I reuse most often in serverless projects: something happens once, and several parts of the system react to it without knowing about each other. Sounds trivial. Even so, I keep seeing Lambda functions that call three other services in a row after saving data and trip over the second call.

In this part of the series, I build the pattern for the fictional bookshop from the first two parts. An order comes in, and the warehouse, accounting and express shipping each get their own copy. This time everything is built with the Serverless Framework, including filters, dead-letter queues and tests. At the end, I explain when I prefer EventBridge over SNS and when combining the two makes sense.

In short

With fan-out, a service publishes an event once to an SNS topic, and every consumer reads it from its own SQS queue. No consumer fails because another one is stuck, and each gets its own retries and its own dead-letter queue. The most common pitfall is a missing queue policy: the subscription exists, but SNS is not allowed to send to the queue.

What you will learn in this article

• How fan-out with SNS and SQS works and why every queue has exactly one consumer
• How to filter on message attributes and on the message body with filter policies
• A complete, tested project with serverless.yml and TypeScript
• When EventBridge is the better choice and when to combine both
• The typical mistakes that silently swallow a delivery

What does fan-out mean?

Fan-out means spreading one thing out into many. A sender sends a message exactly once, and a piece of infrastructure hands copies of it to every interested consumer. The sender does not know the consumers. All it knows is that there is a topic it reports to.

On AWS, this role usually goes to Amazon SNS (Simple Notification Service). An SNS topic is a distributor: you publish to it, and every subscription receives a copy. Subscribers can be Lambda functions, HTTP endpoints, email addresses or SQS queues. A queue holds messages until someone picks them up and processes them.

For the bookshop, it looks like this: the function that accepts orders reports “order placed” on the topic. The warehouse reserves the books, accounting writes the invoice, and express shipping books a courier, but only for express orders.

The architecture at a glance: one topic, three queues, three workers, each queue with its own dead-letter queue.

Why fan-out with SNS and SQS instead of direct calls?

The obvious solution would be a function that calls three services one after another once the order is saved. That works fine until the invoice service is slow for a minute. Then order intake waits, the customer waits, and if the call times out, you do not know whether the warehouse has already reserved the books. On top of that, you have to touch the sender’s code for every new consumer.

Fan-out turns three dependencies into one: the sender only needs the topic. New consumers attach with a subscription, and nothing changes on the sender’s side. In addition, every consumer works at its own pace.

Why every consumer needs its own queue

A common misunderstanding: “I will just attach three Lambda functions to the same SQS queue.” That is not fan-out, it is competition. With a queue, only one consumer ever receives a given message. The three functions would split the orders between them instead of each getting all of them.

That is why every consumer gets its own queue in a fan-out setup. This gives you three more things I would not want to miss. If accounting goes down, its messages pile up in its queue while the warehouse keeps working. Each queue gets its own dead-letter queue (DLQ), a parking spot for messages that still fail after several attempts. And you can control batch size and parallelism per consumer.

Why not subscribe the Lambda functions to SNS directly? You can, and for non-critical cases that is perfectly fine. With a queue in between, though, you get a buffer: load peaks are smoothed out, you can reprocess messages calmly, and partial failures in a batch can be reported cleanly.

🎬 Animation: fan-out with a filter – this animated diagram only runs on my site: watch it on sascha-ms.de

SNS or EventBridge: which one fits when?

In the second part of the series, I distributed domain events via an EventBridge bus, and in the article on SQS and EventBridge with Lambda, a rule routes events into a queue. So both services can distribute. The difference lies in the details:

My rule of thumb: if there is a clearly defined event with a fixed group of consumers and a lot of load, I use SNS with SQS. It is simple, fast and cheap. If domain events flow across several teams or accounts and I want to add new rules later or replay old events, I use EventBridge. And sometimes I combine both: EventBridge as the central bus, a rule forwards to an SNS topic, and SNS fans out to many queues behind it.

If you are unsure which service fits your case, my AWS service finder helps you with a few questions.

Fan-out with SNS and SQS step by step

Time to build. I use the Serverless Framework because the previous part was built with the AWS CDK. If you want to compare the two tools, you can find my take in the article Serverless Framework vs CDK vs SST.

Prerequisites

• Node.js 22 or newer and npm
• An AWS account with credentials in your terminal (for example via aws configure or AWS SSO)
• Serverless Framework v4 (npm install -g serverless); since v4 you need a free Serverless Inc. account on first start, and small organizations below the revenue threshold pay nothing
• Basic knowledge of Lambda and SQS; if you are missing it, part 1 of the series helps

Step 1: Create the project

Create a folder and install the dependencies. Since v4, the framework builds TypeScript itself with esbuild, so you do not need a plugin for that.

mkdir bookshop-fanout && cd bookshop-fanout npm init -y npm install @aws-sdk/client-sns npm install -D typescript vitest @types/aws-lambda @types/node serverless mkdir src test 

Then replace the package.json and create the tsconfig.json:

// package.json { "name": "bookshop-fanout", "version": "1.0.0", "private": true, "scripts": { "typecheck": "tsc --noEmit", "test": "vitest run", "deploy": "serverless deploy --stage dev", "remove": "serverless remove --stage dev" }, "dependencies": { "@aws-sdk/client-sns": "^3.1156.0" }, "devDependencies": { "@types/aws-lambda": "^8.10.164", "@types/node": "^22.20.0", "serverless": "^4.45.0", "typescript": "^7.0.2", "vitest": "^5.0.5" } } 

// tsconfig.json { "compilerOptions": { "target": "ES2023", "module": "ESNext", "moduleResolution": "Bundler", "strict": true, "noEmit": true, "skipLibCheck": true, "types": ["node"] }, "include": ["src", "test"] } 

Step 2: Define the event format

Before anything gets sent, I define what the message looks like. An event always gets a type, a unique ID and a timestamp. The ID matters later because SNS and SQS only deliver “at least once”. A worker can therefore see the same message twice and has to recognize that.

// src/order-events.ts export type ShippingMethod = "standard" | "express"; export interface OrderItem { isbn: string; quantity: number; } export interface OrderPlaced { eventType: "OrderPlaced"; eventId: string; orderNo: string; shipping: ShippingMethod; items: OrderItem[]; totalCents: number; placedAt: string; } // Validates the request body and returns an error message or the data export function validateOrder( body: unknown, ): { ok: true; data: Omit<OrderPlaced, "eventType" | "eventId" | "placedAt"> } | { ok: false; error: string } { if (typeof body !== "object" || body === null) { return { ok: false, error: "Body must be a JSON object" }; } const b = body as Record<string, unknown>; if (typeof b.orderNo !== "string" || b.orderNo.length === 0) { return { ok: false, error: "orderNo is missing" }; } if (b.shipping !== "standard" && b.shipping !== "express") { return { ok: false, error: "shipping must be standard or express" }; } if (!Array.isArray(b.items) || b.items.length === 0) { return { ok: false, error: "items must not be empty" }; } for (const p of b.items as Array<Record<string, unknown>>) { if (typeof p?.isbn !== "string" || !Number.isInteger(p?.quantity) || (p.quantity as number) < 1) { return { ok: false, error: "every item needs an isbn and quantity >= 1" }; } } if (!Number.isInteger(b.totalCents) || (b.totalCents as number) < 0) { return { ok: false, error: "totalCents must be an integer >= 0" }; } return { ok: true, data: { orderNo: b.orderNo, shipping: b.shipping, items: b.items as OrderItem[], totalCents: b.totalCents as number, }, }; } // Reads an event from an SQS record (raw message delivery is on) export function parseEvent(body: string): OrderPlaced { const event = JSON.parse(body) as OrderPlaced; if (event.eventType !== "OrderPlaced" || typeof event.eventId !== "string") { throw new Error("Unknown event format"); } return event; } 

Step 3: Accept and publish the order

The intake function validates the body, builds the event and publishes it with a single PublishCommand. I also set the type as the message attribute eventType. Afterwards, it responds with 202 because the actual work is still to come.

// src/place-order.ts import { randomUUID } from "node:crypto"; import { PublishCommand, SNSClient } from "@aws-sdk/client-sns"; import type { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from "aws-lambda"; import { validateOrder, type OrderPlaced } from "./order-events.js"; const sns = new SNSClient({}); export async function handler(event: APIGatewayProxyEventV2): Promise<APIGatewayProxyResultV2> { let body: unknown; try { body = JSON.parse(event.body ?? ""); } catch { return respond(400, { error: "Body is not valid JSON" }); } const check = validateOrder(body); if (!check.ok) { return respond(400, { error: check.error }); } const message: OrderPlaced = { eventType: "OrderPlaced", eventId: randomUUID(), placedAt: new Date().toISOString(), ...check.data, }; // One publish, any number of consumers: SNS delivers to every subscription await sns.send( new PublishCommand({ TopicArn: process.env.TOPIC_ARN, Message: JSON.stringify(message), MessageAttributes: { eventType: { DataType: "String", StringValue: message.eventType }, }, }), ); return respond(202, { orderNo: message.orderNo, eventId: message.eventId }); } function respond(statusCode: number, data: unknown): APIGatewayProxyResultV2 { return { statusCode, headers: { "content-type": "application/json" }, body: JSON.stringify(data), }; } 

A note on persistence: in this example, the function publishes directly without storing the order in a database first. If you need to do both, you have exactly the dual-write problem from the second part. In that case, put the outbox pattern in front, and SNS becomes the target of the pipe.

Step 4: Write workers with partial failures

All three workers share a small helper. It processes every record on its own and reports only the failed messages back. SQS then redelivers just those instead of repeating the whole batch. I explain the details in the article on SQS partial batch response.

// src/batch.ts import type { SQSBatchResponse, SQSEvent } from "aws-lambda"; import { parseEvent, type OrderPlaced } from "./order-events.js"; // Processes each record on its own and reports only the failed ones export async function processBatch( event: SQSEvent, process: (order: OrderPlaced) => Promise<void>, ): Promise<SQSBatchResponse> { const batchItemFailures: SQSBatchResponse["batchItemFailures"] = []; for (const record of event.Records) { try { await process(parseEvent(record.body)); } catch (error) { console.error(JSON.stringify({ messageId: record.messageId, error: String(error) })); batchItemFailures.push({ itemIdentifier: record.messageId }); } } return { batchItemFailures }; } 

// src/workers.ts import type { SQSEvent } from "aws-lambda"; import { processBatch } from "./batch.js"; // Warehouse: reserves the books of every order export const reserveStock = (event: SQSEvent) => processBatch(event, async (b) => { const pieces = b.items.reduce((sum, p) => sum + p.quantity, 0); console.log(JSON.stringify({ action: "reserve", orderNo: b.orderNo, eventId: b.eventId, pieces })); }); // Accounting: creates the invoice export const createInvoice = (event: SQSEvent) => processBatch(event, async (b) => { const amount = (b.totalCents / 100).toFixed(2); console.log(JSON.stringify({ action: "invoice", orderNo: b.orderNo, eventId: b.eventId, amount })); }); // Express shipping: thanks to the filter, only express orders arrive here export const scheduleExpress = (event: SQSEvent) => processBatch(event, async (b) => { if (b.shipping !== "express") { throw new Error(`Wrong shipping method ${b.shipping} – check the filter`); } console.log(JSON.stringify({ action: "express", orderNo: b.orderNo, eventId: b.eventId })); }); 

The workers only log here to keep the example small. In a real system, you would use the eventId to catch duplicate deliveries. I show how to do that with Powertools and DynamoDB in Idempotency in AWS Lambda.

Step 5: Topic, queues and subscriptions in serverless.yml

Now for the core. The serverless.yml describes the four functions, and the resources section holds everything else CloudFormation should create: the topic, three queues with one DLQ each, a queue for undeliverable messages, the queue policy and the three subscriptions.

# serverless.yml service: bookshop-fanout frameworkVersion: "4" provider: name: aws runtime: nodejs22.x region: eu-central-1 architecture: arm64 memorySize: 256 logRetentionInDays: 14 environment: TOPIC_ARN: !Ref OrdersTopic iam: role: statements: - Effect: Allow Action: sns:Publish Resource: !Ref OrdersTopic functions: placeOrder: handler: src/place-order.handler events: - httpApi: method: POST path: /orders reserveStock: handler: src/workers.reserveStock events: - sqs: arn: !GetAtt WarehouseQueue.Arn batchSize: 10 functionResponseType: ReportBatchItemFailures createInvoice: handler: src/workers.createInvoice events: - sqs: arn: !GetAtt InvoiceQueue.Arn batchSize: 10 func...