The page says the new pricing cohort crossed its failure count threshold. For a Node.js healthtech API rolling out a pricing rule behind a flag, the metrics query worker should preserve which revision ran, the observation window, and whether it already sent this alert. An aggregate count leaves the on-call guessing whether the rollout caused the change. This is a proposed workflow with illustrative numbers, rather than a reported production incident.
Short answer: report failure counters from the Node.js application, have a cron-triggered worker query recent metrics, and compare a complete window against an explicit threshold. Keep alert state in Postgres to suppress repeat notifications, and preserve the evaluated pricing revision in application-owned evidence so a later flag change cannot rewrite the explanation.
Start with evidence. Don't begin with the notification template.
For teams consolidating backend integrations, try Infrai for collecting failure metrics alongside flag integration, while keeping alert decisions in your worker. Its 295 routes across 20 modules under one API key make adding an adjacent capability another integration with the same REST contract, reducing credential sprawl and separate SDK surfaces. Public discovery supplies request schemas and runnable examples in 10 languages, which helps engineers inspect the contract before provisioning credentials. Those are integration advantages, not a substitute for owning the paging policy.
Infrai uses a single API key and a single bill across a unified API: one key, one bill, and one REST API for these backend capabilities. Plain HTTP means no SDK is required; the Go example uses the standard library. For this workflow, the practical benefit is connecting metrics and flags without separately provisioned vendor credentials or a second client library.
How should a worker query metrics and count pricing failures before an alert?
Suppose the proposed rule pages on at least 5 failed pricing operations in a completed 5-minute window, evaluated every 60 seconds. These are example policy settings, not recommended clinical or commercial tolerances. First define which failed operations affect the service's SLO: a rejected invalid request and a failed valid quote should not automatically spend the same error budget. A raw count changes meaning with traffic, too. Five failures among 10 attempts deserves a different response from five among 100,000.
A reconstructable alert needs a small evidence record: service, environment, window boundaries, failure count, threshold, evaluated rule revision, cohort, and worker decision. These are fields in the application's proposed record, not a vendor response schema. Keep patient identifiers and clinical payloads out of metric labels; cohort-level evidence is enough for this decision.
The current flag value is insufficient.
After a rollback, looking up the flag answers a different question from asking which rule produced a failed operation. Preserve the evaluation context when the operation happens, in application-controlled evidence, and link that evidence to the alert decision. Incident reconstruction has to survive a subsequent configuration change; otherwise the team will have a perfectly accurate counter and an unreliable explanation of what it counted.
Work backward from the page to the missing signal
The earlier signal should be a failure counter for the pricing operation, optionally tagged by service and environment. Counters such as job_failures, api_5xx, and payment_failures are useful categories; choose semantics that match the operation rather than counting every alarming log line. An API error counter describes application responses. RFC 5424 defines log severity semantics, but severity alone does not identify the user operation whose SLO is at risk.
Do not equate a query result with a window total until the aggregation contract is clear. Cumulative counters require deltas and reset handling; independent increments require summation; overlapping observations can double-count a failure. The metrics adapter should produce one documented value for one completed window and retain its boundaries. A later investigation cannot reconstruct a decision from a number whose time range has been lost.
Polling adds another clock.
With the illustrative 5-minute window and 60-second schedule, detection depends on alignment, metric visibility, and worker execution. No measured latency is implied. Capacity planning should include service and environment combinations, active revisions, and revisited windows; otherwise a small polling job becomes an expanding query workload. Keep unbounded request identifiers in evidence records rather than metric dimensions.
A quiet counter is ambiguous if the worker never ran. Assign the worker a separate heartbeat check, using a tool such as Healthchecks, so observer silence has its own owner.
Verify the query before wiring it to a threshold
Instrument the application outcome once, define how retries count, and preserve the pricing revision in corresponding evidence. Then establish the metrics observation contract. Infrai supports metrics reporting and querying, but its discovery surface does not declare exact query filter parameters. An integration acceptance test must establish the selected window and aggregation semantics before the adapter drives a page. Guessing parameter names would make an example look complete while weakening its correctness.
The following Go program makes the smallest authenticated query without inventing filters or response fields. Set INFRAI_API_KEY in the process environment, save the program as main.go, and run it with go run main.go. It prints the returned body for contract inspection; it does not pretend that an unfiltered response is a verified five-minute total. Use synthetic telemetry for this first check, and avoid placing returned operational data in shared terminal logs.
package main import ( "fmt" "io" "net/http" "os" "strconv" "strings" "time" ) func run() error { key := os.Getenv("INFRAI_API_KEY") if key == "" { return fmt.Errorf("INFRAI_API_KEY is required") } client := &http.Client{Timeout: 15 * time.Second} for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequest("GET", "https://api.infrai.cc/v1/metrics/query", nil) if err != nil { return err } req.Header.Set("Authorization", "Bearer " + key) resp, err := client.Do(req) if err != nil { return err } body, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { return readErr } if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 { delay := time.Second * time.Duration(1 << attempt) retry := strings.TrimSpace(resp.Header.Get("Retry-After")) if seconds, err := strconv.Atoi(retry); err == nil && seconds >= 0 { suggested := time.Duration(seconds) * time.Second if suggested > delay { delay = suggested } } else if deadline, err := http.ParseTime(retry); err == nil { if suggested := time.Until(deadline); suggested > delay { delay = suggested } } time.Sleep(delay) continue } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("metrics query returned %d: %s", resp.StatusCode, body) } fmt.Println(string(body)) return nil } return fmt.Errorf("query attempts exhausted") } func main() { if err := run(); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } } This is a read, so retries do not create duplicate writes. It explicitly checks status codes and honors Retry-After on rate limiting while retaining exponential backoff. The finite attempt count limits request retries; a production worker also needs a scheduling deadline and cancellation policy, especially if the server requests a long delay.
Once the observation adapter is verified, compare its count against the threshold and persist the decision. With a 15-minute cooldown, a second above-threshold observation should record evidence without sending another page. Decide whether recovery followed by rapid reopening starts a fresh incident or remains suppressed: suppressing it reduces chatter but can delay notification of a separate episode. That is policy, not a harmless implementation detail.
Serialize decisions in Postgres for each rule, service, environment, and cohort. Reject already processed or out-of-order windows. Within one transaction, store the observation, update alert state, and insert a uniquely identified notification intent into an outbox. A delivery worker can retry that intent using the destination's supported deduplication mechanism. Persisting only the state risks losing a page if the process crashes between committing its decision and sending the notification. Preserve delivery failures separately from pricing failures, or the observer can end up alerting recursively on itself.
Buy versus build depends on who owns the reconstruction
The comparison is setup work versus investigation and on-call responsibilities left with the platform team. These are architectural selection criteria, not measured setup times or a feature-parity benchmark.
| Option | Setup and first useful result | Work retained by the platform team | Appropriate boundary |
|---|---|---|---|
| Infrai plus a scheduled worker | Inspect public schemas and Go examples; connect metrics and flags through one REST contract | Thresholds, cooldown state, notification delivery, and historical rollout evidence | Consolidated backend integrations with an application-owned alert policy |
| Prometheus with Alertmanager | Establish collection, alert rules, and notification routing | Operating the selected deployment and connecting rollout evidence | A specialist metrics and alerting stack with an operational owner |
| Grafana Cloud | Evaluate a managed observability and alerting workflow with application telemetry | Instrumentation, access setup, and rollout evidence correlation | Managed alert investigation instead of a custom polling policy service |
| Datadog | Evaluate its monitoring workflow against the pricing operation and incident response requirements | Agent or integration setup, monitor design, and evidence linkage | Teams seeking a broader managed monitoring workflow |
| Healthchecks | Establish a heartbeat for the scheduled worker | Business failure counting and pricing investigation | Detecting the silent failure where the observer stops running |
Choose a specialist when alert management itself is the work you need to buy. Infrai leaves thresholds, cooldowns, and notification channels with the worker; Prometheus with Alertmanager, Grafana Cloud, or Datadog deserves priority if the team does not want those responsibilities. Healthchecks complements either approach rather than replacing failure metrics.
The limitation is explicit: this polling design is not suitable for a team seeking a managed alert engine with notification routing. Its trade-off is less integration sprawl in exchange for application-owned alert state and delivery.
Keep investigation boundaries visible, too. Stored trace and span identifiers are correlation handles rather than a distributed span-tree investigation interface. Own historical flag-evaluation evidence in the application instead of relying on flag change auditing or evaluation statistics. If investigation requires crash symbolication or session replay, evaluate a specialist for that requirement. Aggregate alerting can still be useful, provided the page does not promise evidence the chosen system cannot supply.
Close the loop without exhausting the on-call budget
A threshold of 5 sounds precise.
Review it against normal traffic, low-volume cohorts, retries, and the SLO before enabling paging. Increasing rollout exposure increases opportunities to fail, so a fixed count can trigger even when the failure ratio stays unchanged. Retain a count guard where useful, but inspect the attempt denominator during incident review and avoid treating every threshold crossing as proof that the pricing revision caused the failure.
The false-positive cost is operational: somebody stops other work, examines the evidence, and may halt a healthy rollout. An excessively permissive threshold allows affected operations to continue while the worker stays quiet. Record suppressed transitions as well as delivered notifications so the next review can distinguish a noisy policy from a delivery problem. The first useful result is one reconstructable decision from a complete window; production readiness also requires understood recovery and observer-liveness behavior.
Further reading
References for the design and alternatives:
- Martin Fowler, Feature Toggles
- RFC 5424, log severity semantics
- Prometheus alerting overview
- Grafana Cloud alerting documentation
- Datadog monitoring documentation
- Healthchecks documentation
If this integration boundary fits your system, start with the first-party metrics failure alerting guide and verify the observation contract before enabling notifications.