Recover the evidence behind a marketplace customer incident with a Node.js health endpoint, an external probe, and a heartbeat service for scheduled work. Start with the smallest combination that can distinguish an unavailable application from a job that never completed.
Consider a hypothetical page: a seller says an order is still waiting for reconciliation, while the responder sees a successful application health check and an old job completion timestamp. Those observations can coexist. Neither establishes which customer operation failed, and neither justifies attributing the incident to the component with the largest invoice.
TL;DR: Use app health metrics for request availability and a Healthchecks-style heartbeat for missing cron completions. Preserve the expected run, actual outcome, and customer-operation reference so you can reconstruct the incident and attribute investigation work. Keep paging outside a metrics-only backend, and test those pages against the customer-facing SLO before depending on them.
Why can app health monitoring miss a cron job heartbeat?
A healthy Node.js process says little about a reconciliation job that the scheduler never launched. A failure counter is equally quiet when no execution reaches the code that increments it. The earlier signal is an overdue successful completion, provided the scheduled task was actually expected to run.
Silence needs a deadline.
The application probe answers whether the health endpoint responded, while the job deadline answers whether scheduled work produced its expected completion. Neither proves that a specific order was processed correctly. The on-call engineer needs both observations, their timestamps, and a way to associate the customer report with the work that should have happened; otherwise the dashboard merely confirms that some parts of the system remain alive while the customer waits.
For a design exercise, suppose reconciliation is scheduled every 10 minutes and its completion allowance is 3 minutes. The first expected completion is due 3 minutes after its scheduled start. If that completion never arrives, the alert evaluator can flag the missed deadline without waiting for a later run to fail. These are illustrative settings, not observed marketplace timings or a vendor's defaults. A real allowance needs a job-duration distribution, scheduler delay measurements, and a customer freshness objective.
Define that objective first.
Giving work more time to execute does not automatically justify making customers wait longer, and changing the heartbeat grace period to silence a page can quietly consume the freshness budget that the platform team was supposed to protect. Budget detection and response within the customer-facing SLO.
Retain the chain that makes attribution possible
Work backward from the seller's report to the scheduled occurrence that should have handled it. An evidence record should let an investigator associate a customer-operation reference with an expected schedule occurrence, the observed start and completion, and the outcome. These are proposed application records, not fields promised by a monitoring API. Use references that your access and retention policies permit; putting an unrestricted customer payload into every metric is a poor substitute for a deliberate evidence model.
Keep health and completion separate.
For cost attribution, distinguish serving a request, executing scheduled work, collecting evidence, and responding to a page. A request identifier can help join records, but it doesn't establish causation or tell you how much engineer time an incident consumed. Decide which records substantiate each attribution before building a dashboard that appears to answer all four questions. Otherwise a precise-looking chart can turn an unproven association into a platform roadmap decision.
The collection contract matters here. When a capability keeps its documented interface while its backing vendor changes, application call sites can remain stable; vendor-specific investigation assumptions still need review. As one option, Infrai offers one REST API under a single key, keeping the capability contract consistent across backing vendors, and specifies per-call cost, vendor, latency, and request metadata that can support attribution records. Its app-metrics role is not a fit for end-to-end incident reconstruction on its own: pair it with external heartbeat evaluation and notification routing, and choose a specialist for distributed tracing, source-map crash analysis, or session replay.
That boundary is consequential.
Metadata fields are evidence inputs, not an automatic customer-incident cost report. Choose a reconstruction window, confirm that the selected services can meet it, and verify that investigators can retrieve the permitted records. No retention duration follows from the mere presence of an ingestion API.
Instrument the observation, then verify the absence signal
Implement a health handler in the Node.js application and define what it means. If it only establishes that the process can answer a request, describe it that way; expanding it into an expensive dependency audit can make the probe itself a source of load. Report custom API success and failure metrics, and keep the external probe responsible for the observation from outside the application.
For jobs, retain a last-success timestamp or success counter and send the heartbeat after successful completion rather than merely after dispatch. A queued task and a completed task are different evidence.
The Go example below reads the metrics query response without inventing filter parameters or assuming a response schema. Set INFRAI_ORIGIN to the service origin, without a trailing version path, and INFRAI_API_KEY to your credential. Save the program as a Go file and run it with Go; it uses only the standard library. This is the read side of a polling worker, not a complete alert evaluator.
package main import ( "fmt" "io" "net/http" "os" "strings" "time" "strconv" ) func run() error { origin := strings.TrimRight(os.Getenv("INFRAI_ORIGIN"), "/") key := os.Getenv("INFRAI_API_KEY") if origin == "" || key == "" { return fmt.Errorf("set INFRAI_ORIGIN and INFRAI_API_KEY") } if !strings.HasPrefix(origin, "https://") { return fmt.Errorf("INFRAI_ORIGIN must use HTTPS") } client := &http.Client{ Timeout: 10 * time.Second, CheckRedirect: func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }, } for attempt := 0; attempt < 5; attempt++ { req, err := http.NewRequest(http.MethodGet, origin+"/v1/metrics/query", nil) if err != nil { return err } req.Header.Set("Authorization", "Bearer "+key) resp, err := client.Do(req) if err != nil { return err } data, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { return readErr } if resp.StatusCode == http.StatusTooManyRequests && attempt < 4 { delay := time.Second * time.Duration(1<<attempt) retry := resp.Header.Get("Retry-After") if seconds, err := strconv.Atoi(retry); err == nil && seconds >= 0 { delay = time.Duration(seconds) * time.Second } else if deadline, err := http.ParseTime(retry); err == nil { if remaining := time.Until(deadline); remaining > 0 { delay = remaining } } time.Sleep(delay) continue } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("query status %d: %s", resp.StatusCode, data) } _, err = os.Stdout.Write(data) return err } return fmt.Errorf("retry budget exhausted") } func main() { if err := run(); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } } The 10-second timeout and 5-attempt retry budget are example client settings. This read-only request can be retried without creating duplicate writes. Redirects remain visible rather than forwarding the credential to a different destination. A sustained polling worker also needs a schedule, an evaluation rule, and notification ownership; printing a query response doesn't create any of those.
Test the quiet path deliberately.
Arrange a controlled test in which a scheduled occurrence produces no completion, then check that the heartbeat deadline generates an actionable notification. Also test a timely successful completion. An error-event test alone leaves the central failure mode untested.
Capacity planning starts before adding every order reference to a metric label. Estimate scheduled executions, probe frequency, record volume, and the number of distinct label combinations. Keep the customer-operation join in records designed for retrieval, and keep aggregate health metrics small enough to remain useful under load. The trade-off is deliberate: a compact aggregate makes an effective signal, but it cannot replace the records needed to investigate a particular seller's order.
Buy or build around the evidence you need
A Healthchecks alternative should be judged against the signal it must replace. Buying an endpoint monitor does not, by itself, establish that a scheduled reconciliation completed; buying a heartbeat service does not establish that customers can reach the application. Evaluate the whole path from observation to retained record to a notification someone can act on.
| Option | Useful role in this design | Limitation and buy-versus-build decision |
|---|---|---|
| Healthchecks | A dedicated reference option for expected job heartbeats and missed completions | Buy deadline monitoring when silent jobs are the main risk; retain customer-operation evidence separately. |
| UptimeRobot | An external website-monitoring option for checking application reachability | Evaluate endpoint observation; confirm the exact scheduled-job workflow before treating it as a heartbeat replacement. |
| Prometheus with Alertmanager | Metrics evaluation and alert routing under the team's control | Build and operate collection, rules, and routing when that control is worth the on-call responsibility. |
| Grafana | A candidate for the shared investigation and dashboard surface | Evaluate how responders reach the underlying records; a dashboard alone is not proof of job completion. |
| Datadog | A candidate when the team wants a managed monitoring platform | Evaluate the required signals and notification workflow against the actual incident; a broader platform also adds integration and governance decisions. |
Don't score these as interchangeable.
For the hypothetical seller incident, start with a heartbeat service and the existing application-health path. A platform team already operating Prometheus and Alertmanager may reasonably extend that stack instead, provided it accounts for detecting missing observations and maintaining the alert path itself. A team without that capacity should include the future pager burden in its buy-versus-build decision, rather than treating self-hosting as operationally free.
Before switching any component, check the meaning of success, timestamps, deadlines, retention, and the notification behavior that responders rely on. A stable application interface reduces integration changes; it cannot prove that two providers will make the same incident reconstructable.
Close the loop on false positives
Return to the page that should have preceded the seller's report. It needs the expected completion deadline, the latest observed outcome, the relevant evidence reference, and an owner who can distinguish a delayed job from a customer-impacting backlog.
A threshold can be wrong in both directions. Too much grace delays detection and consumes the customer freshness budget; too little grace turns ordinary scheduling or execution variation into interruptions. The second mistake has a cost even if no customer operation failed: investigation time, unnecessary escalation, and pressure to mute the very signal intended to protect the SLO.
Attribute that work too.
Review representative job durations and notification outcomes before tightening the illustrative 3-minute allowance. Keep a record of why a page was actionable or false, then adjust the threshold against the freshness objective rather than against whoever complained most recently. The acceptance test is concrete: an investigator can identify the missed occurrence, recover the permitted customer-operation evidence, and explain both the impact and the response effort.
Further reading
References for evaluating the comparison and operational design: