Short answer: For Apache Kafka retry patterns, use Kafka when replayable history matters; choose AWS SQS when isolated job retries matter, and put the shipment guarantee in a per-subscriber record either way.
The least complex design that meets this requirement is a durable job record, an explicit retry state, and an idempotent subscriber handler. Let the event stream carry the shipment update, but give the retry decision to a worker that can record why delivery is being attempted again.
That distinction matters in an edtech platform. A shipment update may fan out to a learner, a school administrator, a warehouse integration, and an email service. A missed update is visible. A duplicate update can be worse: it can create two notifications, advance a downstream state twice, or send an operator chasing the wrong parcel.
The alert I want is not merely “consumer lag is high.” The useful page says that a shipment update has remained in a retryable state past its delivery objective, or that the same subscriber has crossed a duplicate-delivery threshold. That takes us back to the first signal: the worker must persist an attempt outcome and the event identifier before it acknowledges successful work.
The alert needs a clock.
Should Apache Kafka or AWS SQS handle retry patterns?
Kafka and SQS answer different questions even when both are used to move work. Kafka is an append-only event log with consumer offsets; SQS is a queue where a consumer receives a message and deletes it after successful processing. RabbitMQ exposes acknowledgements and negative acknowledgements as part of consumer delivery. Those are useful mechanisms, but none of them turns an external side effect into an exactly-once operation.
For shipment fan-out, start with the subscriber contract rather than the transport name. There are three practical choices:
| Requirement | Retry boundary | Operational consequence |
|---|---|---|
| Rebuild state from an ordered event history | Consumer position in a log | Retrying can mean replaying an event; handlers must tolerate reprocessing |
| Isolate each delivery attempt | Queue message visibility or acknowledgement | A failed delivery becomes available again, with a separate policy for poison messages |
| Coordinate several subscribers | Per-subscriber delivery record | One subscriber can retry without holding the others hostage |
The third row is usually the important one for fan-out. A single shipment event should not have one global “done” bit. Store a delivery record keyed by (shipment_id, subscriber_id) or by a stable event ID plus subscriber ID. Then a warehouse timeout does not force the learner notification to repeat, and a learner notification retry does not replay the warehouse call.
This is a decision about ownership of state. The broker can redeliver. The application decides whether the side effect is new, already complete, or safe to try again.
I have been paged for missed work and duplicate deliveries; the useful distinction is which state was lost, not which broker name appears in the dashboard. That is why the per-subscriber record matters more than a transport-level “success” counter.
The page fires first: trace the alert back to the missing signal
Picture the on-call view. A school has received a tracking change, but one subscriber has not. The page says the retry queue is growing. It does not say whether the backlog is caused by a slow endpoint, a poison payload, an authentication failure, or an acknowledgement that was never recorded.
Work backward from that page. For every delivery attempt, emit at least these fields:
-
event_idandsubscriber_id, which define the unit being retried -
attempt,first_attempt_at, andlast_attempt_at - a bounded error class such as
timeout,rate_limited,auth, orinvalid_payload - the next eligible attempt time
- the final state:
pending,delivered, ordead_lettered
The alert should measure age and outcome, not just volume. A queue with 10,000 fresh messages may be healthy; one message older than the shipment-update objective may require action. I would page on oldest pending age and on a sustained rise in terminal failures, with a separate low-noise signal for duplicate suppression.
Thresholds need a false-positive budget. If the page fires on one transient timeout, the team will tune it away. If it waits for a large backlog, the classroom support team discovers the problem first. A sensible starting point is a delivery-age threshold tied to the published objective, then a review of real alert outcomes after a week of normal traffic. The number is a policy, not a property of Kafka or SQS.
What does a retry actually repeat?
A retry repeats an attempt, not necessarily the business action. That is the distinction that keeps a network timeout from becoming two shipment emails.
That is the trap.
The handler should first consult durable state, perform the external action with an idempotency key when the destination supports one, and record completion in a transaction that makes the result visible to the retry worker. The exact transaction boundary depends on the destination. A database write and an HTTP call cannot share a local transaction by wishful thinking.
Here is the shape of a small Go worker. The interfaces are deliberately generic; the important part is the state transition and the stable key, not a vendor client.
type Delivery struct { EventID string SubscriberID string Attempt int } type Store interface { Claim(ctx context.Context, eventID, subscriberID string) (bool, error) MarkDelivered(ctx context.Context, eventID, subscriberID string) error ScheduleRetry(ctx context.Context, eventID, subscriberID string, attempt int, next time.Time, reason string) error } type Sender interface { Send(ctx context.Context, subscriberID, idempotencyKey string) error } func deliver(ctx context.Context, store Store, sender Sender, d Delivery) error { claimed, err := store.Claim(ctx, d.EventID, d.SubscriberID) if err != nil { return err } if !claimed { return nil // Another worker completed or owns the attempt. } key := d.EventID + ":" + d.SubscriberID if err := sender.Send(ctx, d.SubscriberID, key); err != nil { next := time.Now().Add(backoff(d.Attempt)) return store.ScheduleRetry(ctx, d.EventID, d.SubscriberID, d.Attempt+1, next, classify(err)) } return store.MarkDelivered(ctx, d.EventID, d.SubscriberID) } There is a hard edge here. If Send succeeds and the process dies before MarkDelivered, the next worker may send again. The idempotency key makes that repeat safe only if the destination honors it. If it does not, the application needs an outbox, a destination-side deduplication record, or an explicit acceptance of at-least-once side effects. Claiming a row is not a magic exactly-once guarantee.
Kafka, SQS, and RabbitMQ through an SRE lens
The comparison is clearer when reduced to failure behavior. Kafka consumer offsets are useful for replay and ordered history, but an offset commit does not commit an external email or warehouse update. SQS visibility and deletion give a queue-oriented retry boundary, but a message can be delivered more than once and the handler still needs idempotency. RabbitMQ acknowledgements let the consumer tell the broker when work was accepted; acknowledgement state still does not cover a separate side effect.
These products also differ in the operational shape they encourage. Kafka is a good fit when the event history itself is valuable and multiple consumers need independent positions. SQS is a good fit when each delivery is an isolated unit of work and the application wants queue-level redelivery semantics. RabbitMQ is a good fit when explicit acknowledgement and routing patterns are central to the design. Those are boundaries, not rankings.
The trade-off is real. Kafka is a poor fit for a small, independent retry queue if the team does not need replay, while SQS is a poor fit when consumers must rebuild a long event history from shared offsets. RabbitMQ's acknowledgement model is not a substitute for recording a remote side effect. Pick the limitation you can operate deliberately.
For this edtech case, I would make the subscriber delivery table the source of truth and treat any of the three transports as a delivery mechanism. That keeps the decision reversible. It also makes a test meaningful: inject a timeout after the destination accepts the request, then verify that the second attempt uses the same key and cannot advance the subscriber twice.
Do not test only the happy path. Test a worker crash before acknowledgement, a worker crash after the remote response, malformed shipment data, a permanently failing subscriber, and a retry storm after a regional dependency recovers. Keep retryable and terminal errors separate. Retrying an invalid payload is noise; dead-lettering a temporary timeout is data loss.
The runbook is part of the retry design
When the page fires, the first question should be “which subscriber is old?” The next should be “why is it old?” The dashboard needs a view by event age, subscriber, error class, and attempt count. An operator should be able to pause one subscriber or error class without stopping every shipment update, then resume from a known state.
Backoff needs a cap and jitter. Without jitter, a shared dependency can receive a synchronized wave of retries. Without a maximum age or attempt policy, a poison message becomes an immortal page. Dead-lettered records need enough context to repair or replay them safely: the original event ID, subscriber, last error class, and payload version.
The deployment check is small but important. Roll out a new handler with a compatibility test for old shipment payloads, preserve the idempotency-key format, and watch duplicate outcomes as well as successful deliveries. A schema change that alters the key is a delivery behavior change, even if the compiler is satisfied.
The conclusion is straightforward: choose the transport based on replay, routing, and isolation needs, then put delivery guarantees in application state. Kafka, SQS, and RabbitMQ can all participate in a reliable fan-out. None can tell you that a remote side effect happened exactly once. The durable per-subscriber record, stable idempotency key, age-based alert, and tested recovery path are what make the promise operational.
