Every broker delivers when nothing breaks. So compare them on the crash.
SQS = a take-a-number ticket. SNS = a group text. Kafka = a tape recorder. Redis Pub/Sub = live radio.
The grid
Kafka here means ordinary consumer groups. Share groups get their own section below.
| Case | SQS | SNS | Kafka | Redis Pub/Sub |
|---|---|---|---|---|
| Who gets it | Competing workers; standard can duplicate | Every matching subscription | Each subscribed group; one owner per partition | Matching subscribers connected now |
| A worker dies | Undeleted message becomes visible after timeout | Retryable delivery failures get retries | Resumes from committed offsets | No replay |
| It keeps failing | DLQ with a redrive policy | Subscription DLQ for failed delivery, if configured | Your code decides | Nothing to retry |
| It arrives twice | Standard can; FIFO deduplicates sends within its window and scope | Standard can; FIFO deduplicates publishes within its window and scope | Uncommitted work can repeat | No redelivery; overlapping subscriptions can duplicate |
| Nobody listening | Retained up to 14 days, default 4 | Standard keeps nothing for later subscribers | Replay while records remain retained | Gone |
Sources: SQS visibility, SQS quotas, SQS DLQ, SNS filtering, SNS retries, SNS DLQ, Kafka design, Redis Pub/Sub.
SNS retries cover delivery, not your work. Once SNS receives an HTTP endpoint's success response, a later worker crash doesn't trigger a retry. When delivery retries run out, SNS discards the message unless a dead-letter queue is attached.
SNS FIFO can archive and replay when configured. Kafka replay also has limits: deletion and compaction can remove history.
Crash on both sides of the side effect
Say the handler charges a card, without idempotency or a durable handoff. Crash it before the charge, then after the charge succeeds.
For after-work acknowledgement, the crash happens before deletion or offset commit. For before-work acknowledgement, the commit or HTTP success has already been accepted. Assume a healthy broker and retained messages still eligible for recovery.
| Setup | Crash before the charge | Crash after the charge succeeds |
|---|---|---|
| SQS, delete after work | Eligible for retry after timeout | Eligible for retry; charge may repeat |
| Kafka consumer group, commit after work | Re-read on recovery | Re-read on recovery; charge may repeat |
| Kafka consumer group, commit before work | Skipped on recovery; charge missed | Skipped on recovery; charge already made |
| SNS to HTTP, success received before work | No retry for this crash; charge missed | No retry for this crash; charge already made |
| Redis Pub/Sub, one subscription | No replay; charge missed | No replay; charge already made |
These are failure traces inferred from SQS, Kafka, SNS and Redis docs, not live broker tests. They show recovery paths, not guaranteed charge counts. Expiration, dead-letter routing or further failures can prevent completion.
Acknowledge before the work, and a crash can lose it. Acknowledge after, and a crash can repeat it. Kill your worker at both points to check your handler.
For the charge, reuse a payment-provider idempotency key on retries, within its retention rules (Stripe's example). Writing a local "done" flag after charging leaves the same crash window. Kafka transactions can coordinate Kafka outputs and offsets; an external payment still needs cooperation from the payment system.
SQS FIFO, long-running jobs and the 12-hour limit
An SQS FIFO queue can still run your job twice. Its deduplication window of 5 minutes covers repeated sends with the same deduplication ID, within the configured queue or message-group scope. It doesn't prevent redelivery after a visibility timeout. SNS FIFO has its own five-minute window and topic or message-group scope.
That matters for long AI agent runs. The visibility timeout has a ceiling of 12 hours from SQS receiving the ReceiveMessage request. Extending it doesn't reset that. Once it expires, another worker can receive the undeleted message while the first may still be going. FIFO also blocks further receives for that group while messages are in flight.
For long jobs:
- Call
ChangeMessageVisibilityas a heartbeat while the work runs. - Split the work before the ceiling. AWS suggests smaller steps or Step Functions. Keep progress outside the queue.
- Put an idempotency key on each side effect.
Kafka's answer changed
| Setup | A record that keeps failing |
|---|---|
| Ordinary consumer group | Your code decides. |
| Share group, 4.2 | group.share.delivery.count.limit caps attempts: default 5, range 2 to 10. That isn't a DLQ. |
| Share group, 4.4 | Configurable DLQs. Enable share.version=2 and set the group's errors.deadletterqueue.topic.name (Apache). |
That topic name must use the configured prefix, dlq. by default. Auto-creation is off by default: create the topic and set its errors.deadletterqueue.group.enable=true, or enable cluster auto-creation. The DLQ stores metadata by default. Set the group's errors.deadletterqueue.copy.record.enable=true to include the original record (Apache).
Different clocks
| Clock | Value |
|---|---|
| SQS retention | 4 days by default, 14 at most |
| SQS visibility timeout | 30 seconds by default, 12 hours per receive at most |
| SQS FIFO deduplication | 5 minutes |
| SNS FIFO deduplication | 5 minutes |
| SNS retries to SQS or Lambda | 100,015 attempts over 23 days for retryable delivery errors |
| Kafka share-group delivery limit | 5 attempts by default |
How to pick
- Jobs that must survive a crash go on SQS.
- Long AI agent runs too, with a heartbeat and checkpoints.
- One event for many services goes to SNS, often fanned out into SQS queues.
- History you want to replay lives in Kafka.
- For order per customer, use SQS FIFO message groups, or a Kafka key that consistently maps to one partition. The handler must preserve that order. Moving a failed FIFO message to a DLQ can break it.
- Live updates you can afford to miss, like typing indicators, fit Redis Pub/Sub. If you need them kept, use Redis Streams, with durability determined by persistence and replication settings.
Every broker works on a good day. Pick the one whose bad day you can live with.
The maintained article is on argbe.tech.
