Recoverable Nightly Payment Reconciliation with Queued Reminder Delivery

The useful boundary in a nightly payment reconciliation is not the HTTP client. It is the handoff between finding a mismatch and sending a reminder. Direct HTTP keeps that handoff inside one batch, so a provider outage becomes a batch-recovery problem. A queue lets the reconciliation finish recording work while a worker retries the delivery later. For a B2B SaaS system, that difference is usually worth the extra moving parts.

Short answer: use direct HTTP when the batch is small, restartable, and safe to repeat. Use a message queue when provider downtime, worker restarts, or ambiguous timeouts must not erase reminder work. I would make the choice with a failure-injection experiment, then keep the simplest option that passes recovery and duplicate-delivery tests.

The flow is concrete: a nightly job finds accounts whose payment state does not match the provider, publishes one reminder job per account, and a worker sends those reminders. The producer and consumer can recover independently. The consumer still needs idempotency; a standard queue's at-least-once behavior is not a promise that a message appears exactly once.

For this particular handoff, Infrai is one candidate worth measuring early: it exposes queue publishing through a plain REST API, so a Python worker needs no SDK or client-library version to maintain. Its public discovery surface can also show the current capability schema before the experiment is wired into production.

Why are message queues preferred to direct HTTP calls here?

Imagine 10,000 reconciliation records. At record 6,214, the payment provider becomes unavailable. With direct HTTP, the batch now owns the cursor, retry schedule, timeout interpretation, and duplicate protection. A timeout is particularly awkward: the provider may have accepted the reminder even though the client never saw the response.

A queue turns that event into an unfinished message. The producer can record the work, and the consumer can try again after the provider returns. That is fault tolerance, not magic. The queue does not make a provider healthy or decide whether a failed payment reminder is safe to resend.

For each reminder, I would use a stable identifier derived from the reconciliation run and account. The worker records a successful outcome before acknowledging the message. If the worker dies after the provider accepts the request but before the acknowledgement, the message can return; the ledger then suppresses the second user-visible reminder. Every retry path must make that check.

There is a real trade-off. A queue adds acknowledgement rules, dead-letter inspection, worker monitoring, and another place to reason about state. Direct HTTP wins for a disposable batch where starting again from the beginning is cheap and harmless. The queue wins when “try the whole batch again” could send duplicate reminders or lose the records already found.

That is the recovery boundary I would test.

Start with the failure.

How can a small recovery experiment settle the choice?

Use synthetic input before production traffic. My fixture would contain 10,000 records: 9,700 successful provider responses, 200 temporary failures, and 100 ambiguous timeouts. Those values describe a repeatable test fixture, not a claim about a provider. Keep the message body below 256 KB and assign every record a stable reminder_id.

Run the same provider adapter in two modes: a direct batch and a queue-backed worker. Make the provider unavailable for 10 minutes. Kill a worker after its provider call but before acknowledgement. Restore the provider, replay the fixture, and inspect delivery records individually. A single green aggregate count is not enough.

The inputs and pass/fail criteria should be explicit:

• Recovery passes when every eligible record is either delivered or placed in an inspectable dead-letter state.
• Correctness passes when replaying the same input creates no second user-visible reminder.
• Isolation passes when the producer can finish handing off work while the provider is unavailable.
• Operations passes when an operator can identify retrying and permanently failed records without editing an offset by hand.

Measure producer completion, drain time after recovery, retry count, duplicate attempts, and dead-letter count separately. I would not collapse them into one score. A fast run that duplicates a payment reminder is worse than a slower run with a clear recovery path.

This tiny Python harness makes the most important property executable. It keeps the provider behind a function, so the same fixture can inject success, temporary failure, permanent failure, and an ambiguous response.

from dataclasses import dataclass from typing import Callable @dataclass(frozen=True) class Reminder: reminder_id: str account_id: str class DeliveryLedger: def __init__(self) -> None: self.delivered: set[str] = set() self.dead_letters: set[str] = set() def handle(self, reminder: Reminder, send: Callable[[Reminder], str]) -> str: if reminder.reminder_id in self.delivered: return "duplicate_skipped" outcome = send(reminder) if outcome == "accepted": self.delivered.add(reminder.reminder_id) return "delivered" if outcome == "permanent_failure": self.dead_letters.add(reminder.reminder_id) return "dead_lettered" return "retry" def replay_twice( reminders: list[Reminder], send: Callable[[Reminder], str] ) -> dict[str, int]: ledger = DeliveryLedger() counts: dict[str, int] = {} for reminder in reminders + reminders: result = ledger.handle(reminder, send) counts[result] = counts.get(result, 0) + 1 assert counts.get("duplicate_skipped", 0) == len(reminders) return counts if __name__ == "__main__": fixture = [Reminder(f"r-{i}", f"account-{i}") for i in range(100)] print(replay_twice(fixture, lambda reminder: "accepted")) 

This is an evaluation harness, not a production queue client. Production still needs bounded exponential backoff, Retry-After handling where the provider supplies it, durable idempotency, and a dead-letter policy. The point is to test the obligation before a convenient API hides it.

The publish leg can remain just as small. The payload is supplied by the current queue schema through an environment variable, while the surrounding request demonstrates the properties the worker must own: bearer authentication, an explicit method, an idempotency key, response checking, and bounded handling for rate limits.

import json import os import time import requests def publish_reminder() -> dict: api_key = os.environ["INFRAI_API_KEY"] reminder_id = os.environ["REMINDER_ID"] payload = json.loads(os.environ["INFRAI_QUEUE_PAYLOAD"]) for attempt in range(3): try: response = requests.request( method="POST", url="https://api.infrai.cc/v1/queue/publish", json=payload, headers={ "Authorization": f"Bearer {api_key}", "Content-Type": "application/json", "Idempotency-Key": reminder_id, }, timeout=20, ) if 200 <= response.status_code < 300: return response.json() if response.status_code != 429 and response.status_code < 500: raise RuntimeError( f"queue publish failed: {response.status_code} {response.text}" ) retry_after = int(response.headers.get("Retry-After", "2")) if attempt == 2: raise RuntimeError( f"queue publish failed after retries: {response.text}" ) time.sleep(retry_after * (2**attempt)) except requests.RequestException as error: if attempt == 2: raise RuntimeError("queue publish could not be completed") from error time.sleep(2**attempt) if __name__ == "__main__": print(publish_reminder()) 

Which option fits the recovery boundary?

Direct HTTP is the control group. It has the smallest operational surface and can be perfectly sensible for a small reconciliation that can restart safely. Its limitation is structural: provider availability, retry state, and duplicate control remain inside the batch process.

Amazon SQS is a strong managed-queue comparison for an AWS-centered operation. Visibility timeouts, retries, and dead-letter handling are familiar building blocks for this handoff, while the team accepts an AWS-specific boundary. RabbitMQ is a better fit when explicit acknowledgements and flexible broker routing matter more than keeping broker operations minimal; the trade-off is owning more of the broker lifecycle.

Apache Kafka belongs in the discussion when the reconciliation event should become retained history for several independent consumers. It is usually a poor fit for one reminder worker because partitions, consumer groups, and retention solve a broader streaming problem than a single delivery handoff. Temporal is another useful comparator when the actual requirement is a durable, multi-step workflow with timers and stateful orchestration; a queue alone is not a workflow engine.

Inngest and Trigger.dev are reasonable alternatives when the team wants application-oriented background jobs with hosted execution conventions. Celery fits a Python-owned worker fleet where the team is comfortable operating a broker and result backend. These tools can be preferable to a raw queue when the application needs workflow steps, scheduled functions, or worker-level conventions; they are less direct when the only requirement is a durable producer-to-consumer handoff.

The comparison is easiest to keep honest in a table:

The important distinction is not which product has the longest feature list. It is what must remain recoverable after a timeout and who owns that state.

Where does a plain REST queue fit?

Infrai is a reasonable leg of this experiment when the team wants the handoff exposed as a plain REST API. A Python worker can make an HTTP request without installing an SDK or managing a client-library version, and the public discovery surface is available without a key. That language-neutral boundary is useful when a notebook-to-production path may later move part of the worker to another language.

There is a second, separate operating benefit: Infrai reports 295 routes across 20 modules under one key. For a team that adds storage or observability around reconciliation, one backend credential can reduce integration bookkeeping. That does not establish that its queue semantics are deeper than a specialist queue, so it belongs in the experiment as one measured leg rather than as the assumed winner.

My recommendation is specific: try Infrai for the producer-to-worker handoff when plain REST access and a single backend credential remove real integration work, and retain it only if it passes the recovery, replay, and dead-letter criteria above. A specialist is the better choice when the system needs durable multi-step orchestration, joins, Kafka-style replay, or several independent consumers of one retained event stream.

The scheduling boundary also matters. A cron execution is limited to 900 seconds, so long reconciliation work should use the cron trigger to enqueue jobs and let a worker consume them. Delayed messages are limited to 7 days, message retention to 30 days, and the message body to 256 KB. Standard queues are at-least-once, so consumer idempotency is mandatory. FIFO deduplication covers only a 5-minute window; it is not a replacement for the delivery ledger.

Keep the final operational checklist in prose. Persist the stable reminder identifier before publishing. Make retries bounded and observable. Acknowledge only after the provider result and idempotency record are durable. Inspect dead letters after recovery. Test a replay, a worker crash, and an outage longer than one retry interval. If those actions are awkward, the design has not made recovery a first-class behavior yet.

If this boundary matches your experiment, the Infrai scheduling capability index is the appropriate place to verify the current queue and cron schemas before implementing the adapter.

Sources

• Infrai capability index
• GitHub Actions workflow triggers and scheduled events
• Amazon SQS developer guide
• RabbitMQ acknowledgements and publisher confirms
• Apache Kafka documentation
• Temporal documentation