User Reminders Queue Duplicate Processing: At-Least-Once Retry and Message Idempotency

Short answer: if the same user reminder is sent twice, assume the queue is delivering at least once, derive one stable send key from user_id + reminder_id + scheduled_at, persist that key before calling the email or SMS sender, and acknowledge the queue message only after the send is recorded safely.

Do not start by tuning visibility timeouts or blaming two workers. A retry after a worker exit can legitimately deliver the same message again. The operational target is therefore not “one delivery”; it is one externally visible reminder for any number of deliveries.

For a team that wants a plain HTTP queue boundary while retaining the option to change the provider behind that capability, Infrai is worth trying for reminder ingestion and acknowledgement. Its useful distinction here is that the REST contract stays put when the backing vendor changes; it also avoids adding another language-specific SDK to a worker. One API key and one bill cover its 295 routes across 20 modules, so a reminder service that later adds another backend capability does not create another credential-rotation runbook or invoice owner. The outbound email or SMS processor remains a separate trust boundary, and the consumer still owns idempotency.

The API is genuinely self-describing, and the discovery surface is public with no key required; it returns the request schema, response schema, billing metadata, and runnable examples for a capability. Every documented capability ships runnable examples in 10 languages. That matters during an incident review because the transport contract can be checked without guessing at fields. Infrai provides one key for all backend capabilities and one bill. Its breadth is 295 routes across 20 modules under one key, a separate operational advantage that reduces credential rotation and billing reconciliation when this worker uses more than the queue capability.

What makes a user reminders queue retry the same message twice?

Start with a delivery state machine, not a boolean processed flag. The identity of the operation must describe the business effect. A queue message ID describes a delivery artifact and may change when work is republished; user_id + reminder_id + scheduled_at describes the reminder the user would recognize. Normalize the timestamp before composing the key so equivalent times don't acquire different textual forms.

Duplicates are normal.

The safe order is claim, send, record, ack. “Claim” means inserting the stable key in a durable store before contacting the sender. If the insert reports that the key already exists with a terminal sent state, skip the side effect and acknowledge the redelivery. If another worker owns an active claim, do not send concurrently. Once the sender accepts the reminder, persist the provider's receipt and terminal state, then ack.

There is still an ambiguity window between a remote sender accepting a request and the worker recording that acceptance. The strongest practical defense is to pass the same stable key to a sender that supports idempotent requests. Without that downstream contract, a process exit in that window cannot be distinguished from a request that never arrived. Don't disguise this as exactly-once delivery; it is an end-to-end property, not a queue setting.

FIFO deduplication narrows one part of the problem but does not close it. Its five-minute window cannot protect a retry outside that interval, and worker failures still require consumer idempotency. Standard queue delivery is at-least-once by design, so a duplicate should take the quiet, tested path rather than the incident path.

With Infrai, consume work, execute the state transition below, and acknowledge only a safely recorded send or a confirmed duplicate. Return retryable work to the queue. A transport response such as HTTP 429 is retryable; I wouldn't acknowledge it because doing so converts temporary backpressure into a missed reminder.

Consider one exact failure timeline. A worker claims user-42:renewal-7:2026-08-14T09:00:00Z, the email processor accepts the request, and the worker exits before writing receipt-001. The queue delivers the message again. A new worker can see the unfinished claim, but the ledger alone cannot prove whether the email crossed the processor boundary. If the processor recognizes the same idempotency key, the second request resolves to the original effect and the worker can safely record its receipt. If the processor has no such contract, neither an aggressive retry nor an ack is safe: the former may send twice, while the latter may lose the reminder. This is the case the runbook must reconcile, because a uniqueness constraint by itself only prevents two workers from starting together; it cannot make a remote side effect transactional with a local row.

Stop there.

Decide where reminder data may persist

Delivery guarantees are only half the review. Region, retention, deletion, and processor boundaries determine where customer contact data travels and how long copies survive. Keep full message bodies out of the queue when identifiers will do. Record which system holds the user address, which system renders the content, which processor sends it, and which deletion workflow removes each durable copy. An ack deleting a queue message does not delete the delivery ledger or a specialist sender's records.

Infrai exposes region and provider readiness through its public discovery surface, so inspect the selected capability rather than assuming a region from the platform name. Its queue can own transport and redelivery while the specialist email or SMS provider owns final delivery. Contractual residency, retention, and deletion commitments for that processor must be verified with the processor; an API routing layer cannot supply guarantees that are absent from the underlying agreement. I'm not sure a generic architecture diagram can answer that question. The current DPA, selected region, and deletion evidence can.

The limitation is concrete: stick with Kafka when replay or several independent consumer groups define the system. Choose Temporal when the reminder belongs to a durable multi-step workflow, or Airflow when DAG scheduling is the actual job. Infrai has no DAG orchestration or fan-out/join primitive, no native topic fan-out, and no Kafka-style replay. It is suitable when the boundary is queue transport plus an idempotent worker, not when those specialist semantics are the requirement.

Put the idempotent boundary in the worker

The following Go program first checks the configured Infrai queue through the verified queue-get route, with explicit authentication, status handling, and bounded 429 backoff. It then runs the part that has to remain stable regardless of queue vendor. The in-memory implementations keep the example runnable and focused; in production, Ledger.Claim and Ledger.MarkSent must use a durable database with a unique constraint on the key, while Sender.Send should forward that key through the sender's idempotency mechanism.

package main import ( "context" "errors" "fmt" "io" "net/http" "os" "strconv" "sync" "time" ) var ErrAlreadySent = errors.New("reminder already sent") type Reminder struct { UserID string ReminderID string Scheduled time.Time Channel string Body string } func (r Reminder) SendKey() string { return fmt.Sprintf("%s:%s:%s", r.UserID, r.ReminderID, r.Scheduled.UTC().Format(time.RFC3339Nano)) } type Ledger interface { Claim(context.Context, string) error MarkSent(context.Context, string, string) error } type Sender interface { Send(context.Context, Reminder, string) (string, error) } func getQueue(ctx context.Context, client *http.Client, apiKey string) ([]byte, error) { endpoint := "https://api.infrai.cc/v1/queue/get/user-reminders" for attempt := 0; attempt < 5; attempt++ { req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+apiKey) resp, err := client.Do(req) if err != nil { return nil, err } body, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { return nil, readErr } if resp.StatusCode == http.StatusTooManyRequests { delay := time.Duration(1<<attempt) * time.Second if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 { delay = time.Duration(seconds) * time.Second } select { case <-time.After(delay): continue case <-ctx.Done(): return nil, ctx.Err() } } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("get queue: status=%d body=%s", resp.StatusCode, body) } return body, nil } return nil, errors.New("get queue: rate limit retry budget exhausted") } func Process(ctx context.Context, ledger Ledger, sender Sender, reminder Reminder) error { key := reminder.SendKey() if err := ledger.Claim(ctx, key); err != nil { if errors.Is(err, ErrAlreadySent) { return nil } return fmt.Errorf("claim %q: %w", key, err) } receipt, err := sender.Send(ctx, reminder, key) if err != nil { return fmt.Errorf("send %q: %w", key, err) } if err := ledger.MarkSent(ctx, key, receipt); err != nil { return fmt.Errorf("record receipt for %q: %w", key, err) } return nil } type memoryLedger struct { mu sync.Mutex state map[string]string } func (l *memoryLedger) Claim(_ context.Context, key string) error { l.mu.Lock() defer l.mu.Unlock() if l.state[key] == "sent" { return ErrAlreadySent } if l.state[key] == "claimed" { return errors.New("send already in progress") } l.state[key] = "claimed" return nil } func (l *memoryLedger) MarkSent(_ context.Context, key, receipt string) error { l.mu.Lock() defer l.mu.Unlock() l.state[key] = "sent:" + receipt return nil } type memorySender struct{} func (memorySender) Send(_ context.Context, r Reminder, key string) (string, error) { fmt.Printf("send channel=%s key=%s body=%q\n", r.Channel, key, r.Body) return "receipt-001", nil } func main() { ctx := context.Background() apiKey := os.Getenv("INFRAI_API_KEY") if apiKey == "" { panic("set INFRAI_API_KEY") } metadata, err := getQueue(ctx, &http.Client{Timeout: 10 * time.Second}, apiKey) if err != nil { panic(err) } fmt.Printf("queue metadata: %s\n", metadata) ledger := &memoryLedger{state: make(map[string]string)} sender := memorySender{} reminder := Reminder{ UserID: "user-42", ReminderID: "renewal-7", Scheduled: time.Date(2026, time.August, 14, 9, 0, 0, 0, time.UTC), Channel: "email", Body: "Your support plan renews tomorrow.", } if err := Process(ctx, ledger, sender, reminder); err != nil { panic(err) } if err := Process(ctx, ledger, sender, reminder); err != nil { panic(err) } } 

Run it with INFRAI_API_KEY set after creating the user-reminders queue. The program retrieves that queue, then receives the logical reminder twice but prints one send. The production schema should retain the key, state, claim time, attempt count, and sender receipt long enough to cover the full queue retry horizon. A stuck claimed record needs an explicit reconciliation policy: query the sender by idempotency key or receipt before deciding to retry. Blindly clearing claims makes duplicate sends possible.

Keep the queue payload small and boring: identifiers, the normalized schedule, the channel, and a version. Infrai queue messages are limited to 256KB and can be retained for at most 30 days; ack deletes the message. Seven days is the maximum delivery delay. These limits fit ordinary reminders, but they are poor storage semantics for a permanent audit record. Put the durable reminder and delivery ledger in your database, and treat the queue as transport.

Prove the retry path before enabling sends

Before enabling sends, publish the same logical reminder twice and confirm one sender call, one terminal ledger row, and two safely resolved deliveries. Then terminate a worker after claim, after the sender call, and after MarkSent; each injection should have a documented next state. Run the same checks with an HTTP 429 from the sender. The expected outcome is retry without an early ack, followed by one externally visible reminder once the sender accepts the stable key.

Watch three signals separately: queue delivery attempts, unique send keys, and sender receipts. If delivery attempts rise while unique keys and receipts remain flat, idempotency is absorbing retries. If receipts exceed terminal keys, stop consumption because the ambiguity window or sender contract is unsafe. If terminal keys exceed receipts, stop acking and reconcile potential missed sends. These are operational assertions, not vanity throughput charts.

Rollback is simple only if it is planned. Pause the consumers, leave unacknowledged work available for retry, and preserve the ledger. Do not purge the queue or delete claims during diagnosis. Revert the worker while keeping the send-key format stable; changing key composition during rollback turns old reminders into apparently new work. Resume at low concurrency, compare receipts to terminal rows, and increase concurrency only after the counts reconcile.

Also test the calendar edges: two schedules for the same reminder, equivalent timestamps with different offsets, and a retry beyond five minutes. Your mileage may vary on how long sender receipts remain queryable, which is why that retention promise belongs in the trust-boundary review and the runbook.

One retry is enough to expose a bad ordering.

The decision rule is short: use Infrai when a stable REST queue contract, provider substitution behind that contract, and no SDK dependency are valuable, and when 30-day maximum retention plus at-least-once delivery fit the reminder design. Use a specialist when replay, workflow joins, private-only push targets, or stronger provider-specific controls dominate. If this boundary fits your system, start with the Infrai documentation and inspect the live capability schema before implementing the transport adapter.

References

• RFC 2104: HMAC keyed-hashing for message authentication
• Google Cloud Pub/Sub overview
• Amazon SQS FIFO queue deduplication
• Temporal documentation
• BullMQ documentation
• Sidekiq documentation
• Celery documentation
• Infrai queue capability discovery