A shipment update sent to thousands of game players is not one job. It is a set of independently failing deliveries, and that distinction changes the retry design.

Short answer: persist one delivery intent per subscriber and channel, schedule retryable failures as delayed queue messages with capped exponential backoff and full jitter, and make the delivery effect idempotent. Acknowledge only after the next state is durable; route exhausted or permanent failures to a dead-letter state.

This is an exactly-once mindset applied to an at-least-once transport. A queue can redeliver. The application must make redelivery harmless.

What does a shipment fan-out need to preserve?

Consider shipment SHIP-4821 for a newly purchased game item. The update goes to a player inbox, mobile push, a guild feed, and an analytics consumer. Each subscriber has a separate latency budget and failure mode. If all recipients share one message, one slow push provider controls the fate of the whole batch, while reconciliation cannot explain which recipient was actually delivered.

The durable unit should be a delivery intent identified by (shipment_id, subscriber_id, channel). Store its status, attempt count, next due time, last failure classification, and an idempotency key in a database. Keep the shipment event immutable. The delivery row describes what the system intended and what happened; the queue schedules work but should not become the system of record.

The invariants are more important than the delay curve:

  • A subscriber receives at most one business effect for one shipment and channel, even after redelivery.
  • Every retry has a durable due time and a finite attempt budget.
  • A transient failure can be retried without holding unrelated subscribers.
  • A permanent failure remains inspectable instead of consuming infinite capacity.
  • Acknowledgement follows durable business state.

The uncomfortable boundary is the external call followed by the local commit. If a push provider accepts the request and the worker exits before recording success, the next attempt must reuse the same provider-facing idempotency key or query the provider. If the database commits but acknowledgement is lost, redelivery is expected; a unique delivery key turns the second application into a no-op. Those are ordinary states in an at-least-once design, not exotic edge cases.

How should failed jobs use exponential backoff in a delayed queue?

Treat backoff as load shaping. For attempt n, calculate upper = min(cap, base * 2^n) and choose a random delay from zero through upper. Full jitter keeps a shared outage from waking every worker at the same instant. The cap prevents an old shipment from acquiring an unbounded tail, while the attempt budget defines when the system stops trying.

Classification comes before scheduling. A timeout, temporary network failure, or rate limit is usually retryable. An invalid player identifier, revoked consent, or permanently rejected payload should go straight to a dead-letter state. Record the classification with the retry decision; a count without a reason is poor evidence during reconciliation.

The reader may implement the worker in Node.js, but the critical transition below is Go so the state changes are explicit and dependency-free. Scheduler stands for the queue adapter. Its contract is deliberately generic, which keeps the retry policy independent of a commercial client library.

package main import ( "context" "fmt" "math/rand" "time" ) type Delivery struct { ShipmentID string Subscriber string Channel string Attempt int MaxAttempts int } type Scheduler interface { Publish(ctx context.Context, queue string, body Delivery, delay time.Duration, key string) error } func fullJitter(attempt, base, capSeconds int, r *rand.Rand) time.Duration { upper := base for i := 0; i < attempt && upper < capSeconds; i++ { upper *= 2 } if upper > capSeconds { upper = capSeconds } return time.Duration(r.Intn(upper+1)) * time.Second } func scheduleRetry(ctx context.Context, q Scheduler, d Delivery, r *rand.Rand) error { d.Attempt++ key := fmt.Sprintf("%s:%s:%s", d.ShipmentID, d.Subscriber, d.Channel) if d.Attempt >= d.MaxAttempts { return q.Publish(ctx, "shipment-dead-letter", d, 0, key) } wait := fullJitter(d.Attempt, 5, 3600, r) return q.Publish(ctx, "shipment-delivery", d, wait, key) } 

The key is stable for the logical delivery, not freshly generated for each attempt. The queue adapter may receive the same publish request again after a worker restart; its idempotency behavior, or an outbox publisher in front of it, must prevent duplicate scheduling from becoming duplicate business effects.

Do not sleep inside the worker for the whole delay. A short local retry can be reasonable within a request deadline, but a delayed queue message makes pending work visible and lets the worker process other deliveries. That difference matters during a deploy, when in-memory timers disappear or compete with shutdown.

Which retry boundary is easiest to reconcile?

There are several valid boundaries, and they should not be confused:

Boundary Useful for Main limitation
In-worker timer A short operation inside one request budget Uses a worker slot and is fragile across restart
Delayed queue message An independent, reconstructible delivery attempt Requires duplicate handling and attempt metadata
Database due-time scan Long schedules driven by business state Requires polling, leases, and lock discipline
Workflow history Multi-step work with timers and compensation More operational state than one delivery needs

The practical choice is determined by the failure you need to explain. If the question is “why has this player not received the shipment?”, a delivery table with an audit trail is more useful than a queue dashboard. If the question is “which independent attempt should run next?”, a delayed message is a good scheduling boundary. If several steps must agree before the process advances, a workflow record is a better model.

A transactional outbox is useful when writing the delivery row and publishing its first message must be coordinated. Commit the business change and an outbox row together; a publisher can retry the outbox operation using the same logical key. This still does not make the external provider call magically atomic with the database. It makes the recovery evidence durable.

How do Node.js workers prove that retries are safe?

Observability should show more than queue depth. Track ready messages, delayed messages, oldest delivery age, retry counts by failure class, dead-letter growth, and shipment-to-acknowledgement latency. Segment those measures by channel so a mobile provider's rate limit does not hide a healthy in-game inbox consumer.

Each state change should record the shipment ID, subscriber ID, channel, attempt, idempotency key, classification, chosen delay, worker version, and timestamps. Logs are not enough: retention, sampling, and concurrent retries can erase the causal sequence needed for an audit trail.

Test the gaps deliberately: provider acceptance followed by a worker crash, duplicate delivery after a committed transaction, a rate limit with and without a retry hint, malformed payloads, clock skew near a due time, and dead-letter replay. Property tests can assert that the delay never exceeds its cap and that one delivery key cannot produce two successful business transitions.

Three words: make it replayable.

I am not sure one backoff curve can suit a provider with second-scale recovery and another with hour-scale recovery; your mileage will vary with provider behavior and the business deadline. The invariant is stable even when the curve changes: no unbounded retries, no early acknowledgement, and no delivery effect without an auditable idempotency decision.

When are delayed messages the wrong choice?

Delayed delivery fits when each attempt is independent, the payload can be rebuilt, and the deadline fits the queue's retention and delay limits. It is not suitable when the operation needs a long queryable schedule, human approval, compensation across several systems, or coordinated progress among many steps. Persist the workflow state and choose a scheduler or workflow engine whose history matches that business process.

There is a second trade-off. A high-value shipment may justify an outbox, reconciliation job, and manual review even when the queue supports delayed delivery. A low-value cosmetic notification may accept a shorter retention period and a smaller retry budget. The right abstraction follows the consequence of duplication and loss, not the convenience of the queue API.

References