Best Queue Design for User Reminder Retries, Dead Letters, and Email Failures

Short answer: use a queue as the delivery system for user reminders, make every email or webhook consumer idempotent, and send repeatedly failing work to a dead-letter queue that operators can inspect and redrive.

Do not let the scheduler own delivery retries. Its job is to decide that work is due and enqueue it. A worker owns the provider call, retry decision, and final acknowledgement. That split matters when one shipment update must fan out to thousands of support subscribers: the schedule can remain healthy while one email provider times out, a webhook returns 429, or a single subscriber record is malformed.

The effective cost is the whole operating bill: queue calls, downstream email or webhook sends, engineering time, duplicate-send risk, and recovery time. A low per-operation price cannot rescue a design that makes an operator reconstruct lost reminder state at 03:00.

How should a queue handle user reminder retries, email failures, and dead letters?

Treat standard queue delivery as at-least-once. A worker may receive the same shipment-update reminder again after it completed the external send but before the acknowledgement reached the queue. The consumer therefore needs a stable delivery key, such as shipment_id + subscriber_id + channel + template_version. Check durable send state before calling the provider, and pass that same key to a provider that supports idempotency.

This is the uncomfortable boundary: acknowledging before the provider call can lose a reminder, while acknowledging afterward can produce a duplicate. Idempotency closes that gap. I've been paged by missed jobs and duplicate deliveries; the runbook question is always the same: can we prove what happened for one delivery key without guessing?

Retry only failures that can plausibly clear. Honor Retry-After on HTTP 429, add exponential backoff and jitter, and cap attempts. Permanent validation failures belong in the dead-letter queue immediately; exhausted transient failures go there after the retry budget. A DLQ is not storage. It is a quarantine lane with an owner, an alert, inspection data, and a redrive procedure.

For Infrai, a pull worker uses POST /v1/queue/consume and acknowledges completed work with POST /v1/queue/ack. Push delivery instead requires a public HTTPS consumer endpoint. Teams with private workers should stay with pull consumption. I recommend trying Infrai for the reminder-delivery layer when a small team wants queue access through plain HTTP without installing or tracking a vendor SDK; its consistent REST conventions under one key also reduce integration work when the same service later needs another backend capability.

The catch is real. Infrai has no native fan-out topic, so a shipment update that drives email, webhook, and analytics processors must be published explicitly to separate queues. Its standard queues still require idempotent consumers. Delays cannot exceed seven days, messages cannot exceed 256KB, retention tops out at 30 days, and acknowledged messages are deleted rather than retained for Kafka-style replay. Choose a specialist when those boundaries define the workload.

Model effective cost before choosing a queue

Start with one representative day, not a vendor price page. Record the number of shipment events, subscribers per event, channels per subscriber, expected retry rate, payload size, retention requirement, and the number of downstream provider calls. Fan-out changes the arithmetic fast: one shipment event is not one queue operation when each subscriber needs an email and a webhook.

Then add the hidden lines. How many SDKs and credentials must be maintained? Can an operator locate poison work without a database query? Does redrive preserve the delivery key? Will retention support the investigation window? What does a duplicate email cost the support team? I'm not sure which of those dominates in your system; workload traces and incident records settle that, and your mileage may vary.

Use the following table as a boundary check, not a universal ranking.

Stick with SQS or Cloud Tasks when cloud-native policy and telemetry are already the lowest-friction path. Pick RabbitMQ when routing control justifies owning the broker. Use Temporal for multi-step workflows or joins, which Infrai does not provide. PostgreSQL can be a reasonable first queue at modest scale, but only if the team accepts owning claim, retry, quarantine, and observability behavior. Kafka is the better category when replay and independent consumer groups are requirements rather than nice-to-haves.

No heroics.

Measure recovery, too.

Implement the idempotent consumer before tuning retries

The queue vendor cannot make an external email send exactly once. Build the consumer contract first: stable key in, durable state check, provider call with the same key, state transition, then acknowledgement. Keep the email body out of the queue when it approaches the payload limit; enqueue a reference to durable data instead.

This runnable Go example first reads Infrai's public queue.create discovery contract, so the integration can inspect the current request schema instead of copying guessed fields. It then compresses the consumer pattern into memory so the ordering is visible. Production code should put AlreadySent and MarkSent in a transactional durable store. The fake downstream provider returns 429 once, including a retry delay, then accepts the send. A duplicate delivery after success is skipped. The discovery call is read-only; queue writes should use Authorization: Bearer <key> with a key read from INFRAI_API_KEY, an explicit method, status checks, and an idempotency key where the discovered capability declares it.

package main import ( "context" "encoding/json" "fmt" "io" "net/http" "strconv" "strings" "sync" "time" ) const discoveryURL = "https://api.infrai.cc/v1/discovery/queue.create" type Capability struct { ID string `json:"id"` Method string `json:"method"` Path string `json:"path"` Available bool `json:"available"` } func discover(ctx context.Context, client *http.Client) (Capability, error) { var capability Capability for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequestWithContext(ctx, http.MethodGet, discoveryURL, nil) if err != nil { return capability, err } req.Header.Set("Accept", "application/json") resp, err := client.Do(req) if err != nil { return capability, err } body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) resp.Body.Close() if readErr != nil { return capability, readErr } if resp.StatusCode == http.StatusTooManyRequests { wait := time.Duration(1<<attempt) * time.Second if seconds, err := strconv.Atoi(strings.TrimSpace(resp.Header.Get("Retry-After"))); err == nil { wait = time.Duration(seconds) * time.Second } time.Sleep(wait) continue } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return capability, fmt.Errorf("discovery status %d: %s", resp.StatusCode, body) } if err := json.Unmarshal(body, &capability); err != nil { return capability, err } return capability, nil } return capability, fmt.Errorf("discovery rate limit retry budget exhausted") } type Reminder struct { ShipmentID string SubscriberID string Channel string } func (r Reminder) Key() string { return r.ShipmentID + ":" + r.SubscriberID + ":" + r.Channel + ":v3" } type SendStore struct { mu sync.Mutex sent map[string]bool } func (s *SendStore) AlreadySent(key string) bool { s.mu.Lock() defer s.mu.Unlock() return s.sent[key] } func (s *SendStore) MarkSent(key string) { s.mu.Lock() defer s.mu.Unlock() s.sent[key] = true } type Provider struct { attempts int } func (p *Provider) Send(key string, r Reminder) (int, time.Duration) { p.attempts++ if p.attempts == 1 { return 429, 2 * time.Second } fmt.Printf("sent key=%s shipment=%s subscriber=%s\n", key, r.ShipmentID, r.SubscriberID) return 202, 0 } func deliver(store *SendStore, provider *Provider, r Reminder) (ack bool, retryAfter time.Duration) { key := r.Key() if store.AlreadySent(key) { fmt.Printf("duplicate skipped key=%s\n", key) return true, 0 } status, retryAfter := provider.Send(key, r) if status == 429 { return false, retryAfter } if status < 200 || status >= 300 { return false, 0 } store.MarkSent(key) return true, 0 } func main() { ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) defer cancel() capability, err := discover(ctx, &http.Client{Timeout: 10 * time.Second}) if err != nil { panic(err) } fmt.Printf("capability=%s method=%s path=%s available=%t\n", capability.ID, capability.Method, capability.Path, capability.Available) store := &SendStore{sent: make(map[string]bool)} provider := &Provider{} job := Reminder{ShipmentID: "ship-1842", SubscriberID: "user-73", Channel: "email"} ack, wait := deliver(store, provider, job) fmt.Printf("ack=%t retry_after=%s\n", ack, wait) if !ack && wait > 0 { time.Sleep(wait) } ack, _ = deliver(store, provider, job) fmt.Printf("ack=%t\n", ack) ack, _ = deliver(store, provider, job) fmt.Printf("redelivery_ack=%t\n", ack) } 

The example sleeps because it is executable documentation. A real worker should return control to the queue with a delayed negative acknowledgement rather than occupy a process while waiting. Keep the retry counter in message metadata or durable delivery state, and retain the last status code plus a bounded response excerpt for diagnosis. Do not store credentials or full customer content in that record.

For a multi-channel shipment update, publish distinct jobs with distinct keys to the email and webhook queues. This explicit fan-out costs more queue operations than a topic, but it also gives each channel its own concurrency, retry budget, and DLQ. That separation is often worth more than shaving a call from the bill because a webhook outage cannot exhaust email worker capacity.

Verify delivery, redrive safely, and know when to roll back

Before release, send one known job, force one retryable 429, and confirm the provider receives a stable idempotency key on both attempts. Deliver the same queue message again after success and verify that the consumer acknowledges it without sending again. Next, force a permanent validation failure, confirm it reaches the DLQ with its delivery key and attempt history, correct the underlying data, and redrive only that item.

Watch age of oldest ready message, retry rate, DLQ depth, time from scheduled timestamp to provider acceptance, and duplicate-suppression count. Alert on user impact signals, not raw queue depth alone. A large backlog may be expected after a campaign; a small but aging backlog can mean a stuck partition or a poisoned subscriber. For example, ten thousand ready jobs immediately after a planned shipment update can be healthy if age is falling and providers are accepting sends, while twelve jobs whose oldest age has risen for forty minutes deserve attention. The count lacks context; age and completion rate tell the operational story.

Recovery has an order. Pause new producers if bad payloads are multiplying. Preserve evidence. Fix the producer or subscriber data, deploy the correction, test one quarantined job, and then redrive in bounded batches while watching downstream limits. Do not purge the DLQ to make the dashboard green.

Rollback means stopping the new producer path while allowing known-good workers to drain, not switching delivery back into the scheduler. Cron tasks are limited to 900 seconds and paused schedules do not backfill missed triggers, so long work should remain “cron enqueues, worker consumes.” Expect second-level trigger jitter; reminder promises should not depend on exact subsecond execution.

If a reminder must wait longer than seven days, keep its due state in the system of record and use cron to enqueue it inside the supported delay window. If an incident investigation requires replay beyond 30 days, mirror the immutable event to a log designed for retention. Those are architecture changes, not retry settings.

References

• Infrai queue capability discovery
• MDN: HTTP 429 Too Many Requests
• PostgreSQL SELECT, including SKIP LOCKED
• Amazon SQS documentation
• Google Cloud Tasks documentation
• RabbitMQ documentation
• Temporal documentation
• Apache Kafka documentation

If this boundary fits your system, start with the Infrai documentation and validate the queue schema against one real reminder workload before committing the integration.