The important health monitoring API polling trade-off is not how aggressively a watcher can poll. It is how quickly it can detect a stalled nightly marketplace pipeline without turning one delayed job into a rate-limit retry storm and an unattributed log bill. TL;DR: give every poll a deadline, treat 429 as a scheduling signal, use capped backoff inside a fixed budget, and attach marketplace, pipeline, run, and owner fields before logs leave the process. A status dashboard can report uptime check results, but it should never control the retry loop.
This is the same shape of problem that appears in OTP delivery: absence is ambiguous. No success event might mean the job is still running, an upstream check is throttled, or the observer itself is unhealthy. Retrying harder collapses those states into more traffic. The better design preserves them.
How should health monitoring API polling handle a rate limit?
A nightly catalog pipeline usually has a long quiet period followed by a burst of consequential work. The watcher does not need to prove that every component is healthy on every pass. It needs to answer a narrower operational question: has this specific run advanced within its expected window?
That distinction changes the payload. A useful response carries a stable run identifier, a state such as running, complete, or failed, and the time of the last state transition. The observer records its own outcome separately. A timeout while asking for state is unknown, not proof that the pipeline failed.
Keep three clocks distinct:
- The schedule clock says when the nightly run should start.
- The progress clock measures time since the last pipeline transition.
- The request clock limits one health query.
Mixing them creates noisy alerts. For example, a five-second request timeout should not imply that a two-hour import missed its completion window. Short request deadlines protect the watcher; pipeline deadlines protect the business process.
Unknown is data.
Derive the retry budget from the constraint
Start with the maximum acceptable detection delay, then subtract request time and alert propagation time. What remains is the polling budget. Do not begin with an arbitrary one-second interval.
Suppose the marketplace can tolerate ten minutes between a pipeline stall and an operator notification. A 60-second base interval gives several observations without pretending the state changes continuously. After 429, the next attempt belongs later in the same detection budget. If the server supplies retry timing, honor it; otherwise use capped exponential backoff with jitter. A cap matters because an unbounded delay can silently exceed the alert window.
The retry counter also needs a scope. Reset it after a successful response or when a new pipeline run appears, not on every scheduler tick. That small choice prevents a cron loop from creating an unlimited sequence of "first" retries. A tempting first draft stores retry state inside one invocation and lets the next scheduled invocation start clean. That looks harmless in a unit test. In operation, though, five separate invocations can each believe they own attempt one, so the supposed five-attempt ceiling isn't a ceiling at all. Persist the attempt window with the run ID, or guarantee that only one invocation owns the run. This is an explicit trade-off: a little coordination state buys a real upper bound on upstream traffic and log volume.
Fail closed on load, open on diagnosis: stop making requests when the budget is exhausted, but retain enough state to explain why the result is unknown. One log event per state transition is usually more useful than one event per poll.
The example below is intentionally a small, generic core. It accepts an injected fetch function, so tests can return a deterministic sequence without contacting a service. It also emits structured events through an injected logger. All sleep decisions remain visible.
import random import time def poll_run(fetch_status, emit, run_id, owner, max_attempts=5): base_delay = 2.0 for attempt in range(1, max_attempts + 1): result = fetch_status(run_id, timeout_seconds=5) context = { "pipeline": "nightly-catalog", "run_id": run_id, "owner": owner, "attempt": attempt, } if result.status_code == 200: emit("pipeline_observed", {**context, "state": result.body["state"]}) return result.body["state"] if result.status_code != 429: emit("poll_failed", {**context, "status_code": result.status_code}) return "unknown" retry_after = result.retry_after_seconds fallback = min(60.0, base_delay * (2 ** (attempt - 1))) delay = retry_after if retry_after is not None else fallback delay += random.uniform(0, min(1.0, delay * 0.1)) emit("poll_deferred", {**context, "delay_seconds": round(delay, 3)}) if attempt < max_attempts: time.sleep(delay) emit("poll_budget_exhausted", { "pipeline": "nightly-catalog", "run_id": run_id, "owner": owner, "attempts": max_attempts, }) return "unknown" There are deliberate omissions. Production code needs cancellation, strict parsing of retry metadata, and a total elapsed-time deadline around the loop. Those belong at the adapter boundary because HTTP clients represent headers, timeouts, and cancellation differently. The policy itself should remain boring.
Test the awkward sequences: 429, 200; repeated 429; timeout followed by completion; an old run ID returned after a new schedule window; and malformed state data. Also test that the final attempt does not sleep. It is a common little waste, and under a large fleet it becomes measurable.
Attribute cost before choosing where logs live
Log volume is driven by cardinality and repetition. Cost ownership therefore begins at emission, not in a billing console. Every event should identify the accountable team or workload with bounded fields. Keep run_id searchable, but avoid copying entire payloads or seller records into every retry event.
| Event | Emit when | Attribution fields | Retention value |
|---|---|---|---|
pipeline_observed | State changes | marketplace, pipeline, owner | Audit and latency analysis |
poll_deferred | A request is throttled | pipeline, owner, attempt | Retry tuning |
poll_failed | A non-retryable response occurs | pipeline, owner, status class | Failure diagnosis |
poll_budget_exhausted | No attempts remain | marketplace, pipeline, owner | Alert evidence |
CloudWatch publishes log pricing in ingestion and related usage dimensions. That makes bytes and duplicate events operational design inputs, even when another storage system is used. The exact bill depends on deployment details, so a durable comparison measures daily bytes by owner, query frequency, retention, and export volume rather than anchoring the architecture to a changing unit price.
Grouping matters too. Sentry documents how event grouping and fingerprints determine which events become one issue. The portable lesson is to choose a stable failure identity: pipeline plus failure class is usually useful, while run ID and attempt number should remain context. If attempt enters the grouping key, one throttling episode can fragment into several apparent incidents.
No magic here. Good attribution lets the team distinguish expensive diagnosis from accidental noise, without discarding the events needed to explain an alert.
Count bytes, not hopes.
Keep the dashboard outside the control loop
An uptime view is a projection of observer state. It may be free, self-hosted, or part of a broader telemetry system; none of those choices changes the retry policy. Publish healthy, degraded, or unknown only after the watcher has evaluated its deadline and evidence. Do not let a dashboard's refresh interval trigger upstream health requests.
This separation also protects compliance boundaries. Marketplace logs can contain seller identifiers, contact details, or message metadata if raw application objects are logged carelessly. The health stream needs operational identifiers, not business payloads. Redaction is easier when the schema is narrow from day one.
For alerting, deduplicate on pipeline and failure class, then update the existing incident as attempts progress. Deliverability systems taught the industry a useful constraint: repeated notification is not proof of delivery. An alert path needs its own status and escalation policy, separate from the pipeline state it reports.
Roll out without losing the old signal
Begin in shadow mode for several nightly runs. Record decisions from the bounded watcher while the existing monitor remains authoritative, then compare detection time, unknown states, request count, and log bytes by owner. Promote the new watcher only after the team has reviewed mismatches.
Next, enable alerts for one marketplace or pipeline partition. Keep rollback at the scheduler boundary so disabling the new poller does not alter the job itself. Finally, publish the dashboard projection and remove duplicate per-poll logging. The migration is complete when every alert can be traced to a run, a deadline, a retry decision, and an accountable owner.
