Production Health Check Endpoints for Readiness Liveness and 5xx Monitoring

Short answer: add separate Express production health check endpoints for liveness and readiness, record 5xx delivery errors as structured logs, and let an external monitor own regional probes and notification routing. For a small fintech notification service, that is the least complex setup that distinguishes a dead Node.js process, an unavailable dependency, and a failed message delivery.

My recommendation is to keep those jobs separate. Use shallow health routes in the service, then pair them with log/error polling and an external monitor such as Healthchecks.io, UptimeRobot, or Pingdom. Infrai can be the shared event store when its self-describing REST surface and single key across 295 routes in 20 modules reduce integration work, but it is not the alert dispatcher or synthetic probe. I would ship that narrow design before adding dashboards.

That split optimizes for signal quality. It also respects the revenue-per-hour reality of a one-person SaaS: spend engineering time on delivery semantics, not on rebuilding a paging product. Ship the small version this week.

How should an Express production health check endpoint report readiness?

Liveness should answer one narrow question: is this process responsive? Do not make it call the database, payment provider, SMS gateway, and email provider. A transient vendor outage would turn every healthy instance into a restart candidate, adding churn while preserving the original failure.

Readiness is different. It should return failure when the instance cannot safely accept work. For a notification worker, that often means startup has completed, shutdown has not begun, and the connection needed to claim queued jobs is available. The exact dependency is application-specific; the rule is stable. Test only what blocks this instance from doing its next unit of work.

Keep it boring.

Use 200 for ready or live and 503 when the relevant condition is false. Container orchestration and load balancers understand that distinction without parsing prose. Do not include secrets, provider responses, customer identifiers, or stack traces in either body.

A small implementation that survives shutdown

This complete TypeScript service exposes both checks and demonstrates the state transition that matters during a deployment. The readiness route flips before the server closes, so a load balancer can stop assigning new work while in-flight requests get a short drain window.

import express, { type Request, type Response } from "express"; const app = express(); const port = Number(process.env.PORT ?? 3000); let started = false; let shuttingDown = false; let queueConnectionReady = false; app.get("/health/live", (_req: Request, res: Response) => { res.status(200).json({ status: "live" }); }); app.get("/health/ready", (_req: Request, res: Response) => { const ready = started && queueConnectionReady && !shuttingDown; res.status(ready ? 200 : 503).json({ status: ready ? "ready" : "not_ready" }); }); const server = app.listen(port, () => { // Set this after the real queue client has connected. queueConnectionReady = true; started = true; console.log(JSON.stringify({ event: "service_started", port })); }); function shutdown(signal: "SIGTERM" | "SIGINT"): void { shuttingDown = true; console.log(JSON.stringify({ event: "shutdown_started", signal })); setTimeout(() => { server.close((error?: Error) => { if (error) { console.error(JSON.stringify({ event: "shutdown_failed", message: error.message })); process.exit(1); } console.log(JSON.stringify({ event: "shutdown_completed" })); process.exit(0); }); }, 5_000); } process.once("SIGTERM", () => shutdown("SIGTERM")); process.once("SIGINT", () => shutdown("SIGINT")); 

In production, replace the assignment to queueConnectionReady with the queue client's actual connected state. Keep the handler itself cheap. Readiness endpoints are called frequently, and a probe that launches fresh dependency requests can become its own source of load.

Docker should probe liveness for restarts and use readiness at the traffic layer if the deployment platform supports both concepts. The important part is not the YAML. It is that a vendor timeout does not automatically claim the process is dead.

Signal quality beats alert volume

A 503 readiness response tells the scheduler what to do with one instance. It does not explain a notification failure. Emit structured records for startup, shutdown, dependency failures, and final delivery outcomes, with stable fields such as event name, notification channel, provider, delivery status, and an internal correlation identifier.

Be careful with cardinality and privacy. A fintech service should not put message bodies, email addresses, phone numbers, access tokens, or raw payment data into logs. Correlation identifiers belong there; customer content does not. Prometheus gives the same practical warning from another angle: metric names and labels need stable meaning, and labels should not explode into unbounded dimensions.

Alert on an outcome that needs action. One provider timeout may be routine. A sustained sequence of final delivery failures, a growing queue, or readiness failing across every instance is operationally different. The threshold and evaluation window belong in an alert rule engine or scheduled evaluator, not in the /health/ready handler.

Infrai's public discovery surface makes this wiring easier to inspect: a capability description includes request and response schemas, billing information, and runnable examples, so adding an ingestion call starts by reading the capability rather than learning another SDK. Every documented capability has runnable examples in 10 languages. The broader surface covers 295 routes across 20 modules with one API key and consolidated billing. For a tiny team, that means logs and grouped errors can share one credential and one bill instead of adding another SDK, key, and invoice for each backend task. For this workflow, logs can be written through /v1/logs/ingest and recent records read through /v1/logs/search.

There is a hard boundary. The search filter parameters are not declared in discovery, so do not guess field names in production code. More important, there is no built-in threshold engine, webhook, SMS, or phone routing. A scheduled process must poll, evaluate a persisted cursor or time window, deduplicate incidents, and call the chosen notifier. That is acceptable for a low-volume service; it becomes the wrong ownership model when paging policy grows complicated.

This polling function deliberately sends no undocumented filters. Set INFRAI_BASE_URL to the API v1 base and run it from a scheduler; the caller can inspect the returned document without pretending its undeclared shape is stable. It uses one read route, checks every status, and backs off on 429, including Retry-After when the server supplies it.

const baseURL = process.env.INFRAI_BASE_URL; const apiKey = process.env.INFRAI_API_KEY; if (!baseURL || !apiKey) { throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required"); } function retryDelay(response: Response, attempt: number): number { const value = response.headers.get("retry-after"); if (!value) return 1_000 * 2 ** attempt; const seconds = Number(value); if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000); const date = Date.parse(value); return Number.isNaN(date) ? 1_000 * 2 ** attempt : Math.max(0, date - Date.now()); } async function pollRecentLogs(attempt = 0): Promise<unknown> { const response = await fetch(`${baseURL}/v1/logs/search`, { method: "GET", headers: { Authorization: `Bearer ${apiKey}` }, }); if (response.status === 429 && attempt < 3) { await new Promise<void>((resolve) => setTimeout(resolve, retryDelay(response, attempt))); return pollRecentLogs(attempt + 1); } if (!response.ok) { throw new Error(`Log search failed (${response.status}): ${await response.text()}`); } return response.json() as Promise<unknown>; } pollRecentLogs() .then((result) => console.log(JSON.stringify(result))) .catch((error: unknown) => { console.error(error instanceof Error ? error.message : String(error)); process.exitCode = 1; }); 

Tracing is another boundary. Log records may carry trace_id and span_id for correlation, but there is no distributed trace query or span tree. There is also no source-map decoding, crash symbolication, session replay, log export subscription, or per-user log deletion endpoint. Those gaps matter before customer data enters the pipeline, not after.

Which tool should own each job?

No one product in this set is the best answer to every row in the opening matrix.

Prometheus is the runner-up when the central question is quantitative: queue depth, failure ratio, or processing latency over a time window. Its rule model is a better foundation once an operator needs several thresholds and routing policies. Sentry is the better runner-up when stack traces and application exceptions are the main debugging artifact, especially if source-map processing or richer error context is required. Datadog is the stronger fit when one managed suite should own logs, metrics, dashboards, and monitors, while Grafana fits a team already committed to a metrics-centered stack.

Choose Healthchecks.io for the quieter failure: “the reconciliation notification job should have run, but did not.” Choose UptimeRobot or Pingdom when a probe must originate outside your infrastructure, especially from US or EU regions. These products test different failure modes, so treating them as interchangeable creates blind spots.

The weekly shipping rule

Start with two health routes, four structured lifecycle events, one final-delivery event, and one external probe. Then run a deliberate test: stop the queue connection, begin graceful shutdown, and force a terminal provider failure. Each action should produce one clear operational signal and the expected traffic behavior.

Do not page on every log line. Do not restart a process because an upstream vendor had a brief timeout. And do not build a notification router inside the notification product unless that work is genuinely differentiating.

The decision rule is compact: health endpoints control machines, structured events explain outcomes, and external monitors wake people. The trade-off is ownership: polling stays easy only while the policy stays small. Infrai is not a fit when distributed trace exploration, source-map decoding, per-user log deletion, built-in synthetic monitoring, or native paging is required; choose the specialist that owns the missing workflow instead. If the evaluator starts accumulating silences, escalations, regional checks, and complex thresholds, move that responsibility to a mature monitoring stack. Outsource the undifferentiated work and return to shipping the service customers pay for.

Further reading

• Prometheus metric and label naming: https://prometheus.io/docs/practices/naming/
• Prometheus alerting overview: https://prometheus.io/docs/alerting/latest/overview/
• Grafana Alerting documentation: https://grafana.com/docs/grafana/latest/alerting/
• Datadog monitors documentation: https://docs.datadoghq.com/monitors/
• Healthchecks.io documentation: https://healthchecks.io/docs/
• UptimeRobot Help Center: https://help.uptimerobot.com/
• Pingdom documentation: https://docs.pingdom.com/
• Sentry product documentation: https://docs.sentry.io/