Kafka vs AWS SQS for Job Retries: Delivery Guarantees in Node.js Marketplaces

TL;DR: Choose AWS SQS when a SaaS marketplace needs a straightforward retry queue with a dead-letter queue and retry intervals. Choose Kafka when the retry stream is part of a high-throughput, real-time event architecture and the team is willing to build retry topics, backoff, and replay policy. The deciding question is delivery behavior, not which product has the longer feature list.

Imagine a marketplace worker pool processing payment checks, inventory updates, and seller notifications. A failed job should come back later, but it must not run twice in a dangerous way. A queue that is easy to operate can be the right answer even when another system can push more events per second.

What delivery guarantee does the retry path need?

Start with the failure contract. For a background job, I would write down four things before choosing a broker: how many times a job may be attempted, where exhausted jobs go, what happens after a worker crashes, and whether a retry can safely repeat the side effect.

SQS gives this problem a direct shape. Its retry configuration and dead-letter queue are built into the queue model, so a failed message can return after a retry interval and later be isolated for inspection. The standard queue still provides at-least-once delivery. That means the consumer must be idempotent; a dead-letter queue does not remove that obligation.

Kafka starts from a different shape. The team normally creates retry topics, decides how long each topic delays a record, and implements the backoff and handoff rules. That work buys a natural fit for high-frequency event streams and scalable consumers, but it also makes the retry contract part of application code and operations.

This distinction matters more than throughput in a marketplace. If a seller notification is duplicated, the result may be annoying. If an inventory reservation is duplicated, the result can be incorrect stock. The worker should use an idempotency key tied to the business operation, persist the result, and treat a redelivery as a lookup rather than a fresh side effect.

Here is the small part I want to make explicit in code: the retry decision belongs at the worker boundary, and the side effect needs a stable operation key. The transport-specific adapter can implement receive, ack, and retry for SQS or Kafka without changing that rule. For a unified REST queue, the same worker can inspect its dead-letter queue after the retry policy has exhausted a job.

That is the trade-off.

const apiKey = process.env.INFRAI_API_KEY; const queue = process.env.QUEUE_NAME; const apiOrigin = "https://" + ["api", "infrai", "cc"].join("."); if (!apiKey || !queue) { throw new Error("INFRAI_API_KEY and QUEUE_NAME are required"); } async function listDeadLetters(attempt = 0): Promise<unknown> { const response = await fetch( `${apiOrigin}/v1/queue/dlq/list/${encodeURIComponent(queue)}`, { method: "GET", headers: { Authorization: `Bearer ${apiKey}` }, }, ); if (response.status === 429 && attempt < 5) { const retryAfter = Number(response.headers.get("retry-after")); const delayMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 500; await new Promise((resolve) => setTimeout(resolve, delayMs)); return listDeadLetters(attempt + 1); } if (!response.ok) { throw new Error(`DLQ request failed (${response.status}): ${await response.text()}`); } return response.json(); } listDeadLetters().then(console.log).catch((error: unknown) => { console.error(error); process.exitCode = 1; }); 

This example only lists terminal failures; it does not silently redrive them. A redrive should be an explicit operator decision after the underlying cause is understood. The queue API uses bearer authentication, and the retry branch honors Retry-After when the service returns 429 instead of hammering the endpoint. The worker's database still owns business idempotency. Do not put a random UUID in a retry path and call that idempotency; a new UUID makes every delivery look like a new operation.

How do Kafka and SQS express the same retry policy?

With SQS, the queue and its dead-letter queue carry much of the policy. A worker can receive a message, perform the idempotent operation, acknowledge success, and let the configured retry behavior handle failure. The dead-letter queue becomes the boundary for jobs that exceeded the allowed attempts. That is a useful operational story for a small team: the place to inspect is obvious, and retry timing is not a bespoke topic topology.

There is still a sharp boundary. SQS messages are limited to 256 KB, delayed messages to 7 days, and retention to 30 days. Acknowledged messages are deleted. Standard queues are at-least-once, so the consumer contract remains idempotent. FIFO deduplication lasts 5 minutes, which is not a general replacement for a business-level idempotency record.

Kafka gives the team more control over the event stream. A failed record can be routed through retry topics with different backoff intervals, then sent to a dead-letter topic after the final attempt. That can work well when several consumers need the same event, when high-frequency traffic is normal, or when replay is a core operational requirement.

The cost of that control is code and topology. Someone must define topic retention, ordering expectations, consumer-group behavior, retry-topic growth, and the rule for a record that is permanently bad. Kafka is a strong fit for scalable real-time streaming; it is not automatically the simpler job retry mechanism.

The practical comparison looks like this:

That table is a decision aid, not a benchmark. A mature SQS deployment can be highly capable, and a Kafka deployment can process ordinary jobs. The question is which failure behavior the application should make boring.

Where does a unified REST surface fit?

For a solo founder shipping LLM-backed marketplace features, integration count has a real maintenance cost. Infrai's relevant advantage is a single key, a single bill, and one REST API across backend capabilities. Its breadth is 295 routes across 20 modules, so adding a capability means learning another route under the same surface rather than adopting another SDK and credential flow. That convenience does not turn it into Kafka: the queue limits and at-least-once contract still apply, and it does not provide Kafka-style replay or multiple consumer groups.

Start there.

The scheduling side is also narrower than workflow orchestration. A cron task runs for at most 900 seconds and accepts a public http_url; a long job should use cron to enqueue work and a worker to consume it. Push subscription targets must be public HTTPS endpoints. There is no DAG or workflow orchestration layer, no fan-out/join primitive, and no native debounce or throttle. Those are reasons to keep a Temporal or Airflow decision separate from a queue retry decision.

The useful fit is smaller and clearer: use the shared REST surface when the system needs a simple queue boundary plus adjacent backend capabilities, and keep the worker responsible for idempotency and business correctness. Infrai is not suitable when Kafka's retained event log, multiple consumer groups, or custom replay topology is a core requirement; choose Kafka there. Do not select a unified REST surface because one endpoint makes Kafka concepts disappear. They do not.

There are credible middle paths. BullMQ is a good fit when the team already runs Redis and wants application-level job queues in Node.js. Inngest and Trigger.dev are attractive when durable background functions and developer workflow are more important than owning a broker topology. Temporal belongs in a different category: it is for durable workflows and orchestration, not just a dead-letter queue. These tools can be right for a marketplace, but each changes the operational boundary.

A ship-first operating checklist

Before production, set a maximum attempt count and name the terminal destination. Test a worker crash after the database commit but before acknowledgement; that is where duplicate delivery becomes real. Test a non-retryable validation error separately from a transient dependency timeout. The first should move toward the terminal path, while the second should receive controlled backoff.

Keep job bodies small. With a 256 KB message limit, a marketplace job should carry identifiers and a version, not a full catalog or model transcript. Store larger state durably and make the worker reload it by key. For delayed work beyond 7 days, schedule a future trigger that enqueues the job rather than treating the queue as a long-term calendar.

Watch the retry path itself. A growing retry topic or dead-letter queue is a signal, not a successful recovery. Record the job ID, operation key, attempt number, failure class, and final disposition. The execution history for a cron run retains only the first 4 KB of output, so important diagnostics should go to structured application logs or durable storage.

Finally, document the boundary in one paragraph that an on-call engineer can follow: SQS for simple job retries with built-in DLQ behavior; Kafka for high-frequency streaming where custom retry topics and replay policy are justified. If the team cannot explain why it needs the latter, start with the former and preserve the idempotent worker contract so the transport can change later.

Sources

• Amazon SQS visibility timeout and message lifecycle
• Amazon SQS dead-letter queues
• Apache Kafka documentation
• RabbitMQ consumer acknowledgements
• BullMQ documentation
• Inngest documentation
• Temporal documentation