A Node.js cron health check should detect a missed job by looking for durable run evidence, not by trusting process uptime. For a property-management SaaS, use two independent paths: the scheduled worker writes that evidence, while a heartbeat watchdog checks deadlines and raises one alert when the record is absent. A timeout should mark the cohort experiment run as uncertain, not trigger an invisible retry. This costs more implementation time than a bare heartbeat, but it preserves the facts needed to explain which tenant cohort saw which treatment.
TL;DR: track scheduled, started, completed, and failed timestamps under one run ID; define the deadline from the schedule plus tolerated delay; make retries keep that run ID; and place the watchdog outside the worker process. A completion heartbeat is the runner-up. It works when the sole question is whether a task ran, but it is weak when the real question is what happened to an experiment population.
| Choice | Missed start | Hung run | Duplicate retry | Incident reconstruction |
|---|---|---|---|---|
| Durable run ledger plus external watchdog | Yes | Yes | Yes | Strong |
| Completion heartbeat | Yes | Yes | No | Limited |
| Process uptime check | No | No | No | Weak |
The recommendation is the first row. The revenue-per-hour test is blunt: one compact ledger earns its keep if it prevents even one long session reconstructing cohort membership from application logs. Build the smallest version, ship it this week, and outsource notification delivery to any generic channel adapter.
How should a Node.js cron health check detect a missed job?
The useful unit is a logical run, not a process and not an HTTP request. Give each expected execution a deterministic ID derived from the job name and scheduled instant. For a job that compares rent-reminder wording across tenant cohorts, the record should carry the experiment revision, cohort snapshot ID, expected start, actual start, terminal state, attempt count, and a compact result summary. Do not put tenant records in the monitor. Keep a reference to the immutable input snapshot instead.
This distinction matters after a timeout. A network response can disappear after the database commit succeeded. Retrying with a new identity can then double-apply the experiment action while producing two unrelated log trails. Reusing the logical run ID makes the operation idempotent and keeps every attempt attached to one piece of evidence.
The first criterion is reconstruction quality: can an operator determine the intended cohort, the experiment revision, every attempt, and the final outcome without guessing? A green process check cannot answer that. Neither can a last-seen timestamp once another run overwrites it.
Use a narrow state model. More states feel precise but create transitions nobody tests.
Keep it small.
type RunState = "scheduled" | "running" | "succeeded" | "failed"; type ExperimentRun = { runId: string; job: "tenant-message-cohort-comparison"; scheduledFor: string; deadlineAt: string; cohortSnapshotId: string; experimentRevision: string; state: RunState; attempt: number; startedAt?: string; finishedAt?: string; outcome?: { treated: number; heldOut: number }; errorCode?: string; }; Keep error text in restricted logs if it might contain tenant data. The ledger needs a stable error code, not a stack dump.
How late is actually missed?
The second criterion is deadline correctness. A cron expression says when work becomes eligible; it does not establish when the result is late enough to page someone. Define an explicit grace period from observed scheduling delay and normal runtime, then test it with a clock you control.
Suppose the comparison runs at 02:00 UTC, normally starts within 3 minutes, and has a 20-minute execution budget. A first pass can set deadlineAt to 02:25. Those figures are an example policy, not a benchmark. Production values need actual runtime distributions and the business deadline for the tenant message.
Retries do not extend the logical deadline automatically. If attempt one consumes 19 minutes, a fresh 20-minute timeout on attempt two may turn a late result into an apparently healthy result. Decide this on purpose: either all attempts share the original deadline, or a documented business rule moves it. For cohort experiments, preserving the original deadline is easier to explain.
Silence is ambiguous. The scheduler may not have started the job, the worker may have stopped midway, or the completion write may have failed. The watchdog should report the observed state and age rather than claiming a cause.
That ambiguity is the main limitation of heartbeat monitoring: absence tells you that expected evidence didn't arrive, but it doesn't identify the failed component. The run ledger also adds a write and a schema migration to every scheduled workflow, so it isn't suitable for disposable tasks where nobody will reconstruct inputs or side effects. In that case, a completion heartbeat is the better trade-off. For critical cohort changes, the extra record is justified because the evidence survives worker restarts and keeps retry attempts together.
Implement the ledger and watchdog
Put the ledger behind a tiny interface so the worker and monitor do not depend on a notification service or analytical database. The persistent implementation needs an atomic insert-if-absent for ensureScheduled, an ownership or lease rule for attempts, and conditional terminal updates. The example stays in memory to make the decision logic runnable.
type Clock = { now(): Date }; type RunStore = { ensureScheduled(run: ExperimentRun): Promise<void>; get(runId: string): Promise<ExperimentRun | undefined>; save(run: ExperimentRun): Promise<void>; overdue(now: Date): Promise<ExperimentRun[]>; }; const logicalRunId = (job: string, scheduledFor: Date): string => `${job}:${scheduledFor.toISOString()}`; async function prepareRun( store: RunStore, scheduledFor: Date, cohortSnapshotId: string, experimentRevision: string ): Promise<string> { const job = "tenant-message-cohort-comparison" as const; const runId = logicalRunId(job, scheduledFor); const deadlineAt = new Date(scheduledFor.getTime() + 25 * 60_000); await store.ensureScheduled({ runId, job, scheduledFor: scheduledFor.toISOString(), deadlineAt: deadlineAt.toISOString(), cohortSnapshotId, experimentRevision, state: "scheduled", attempt: 0 }); return runId; } Before doing cohort work, load that record, increment the attempt under a lease, and set startedAt. On success, write the two aggregate counts and finishedAt in the same logical completion path. On a caught failure, store a bounded code and terminal timestamp. Set a per-attempt execution timeout, but do not confuse cancellation with proof that downstream work stopped.
The watchdog is deliberately boring. Good.
One query. One decision.
type AlertSink = { missed(run: ExperimentRun, observedAt: string): Promise<void>; }; async function inspectDeadlines( store: RunStore, alerts: AlertSink, clock: Clock ): Promise<void> { const now = clock.now(); const lateRuns = await store.overdue(now); for (const run of lateRuns) { if (run.state === "succeeded" || run.state === "failed") continue; await alerts.missed(run, now.toISOString()); } } In the persistent version, add an alert claim keyed by runId and policy version. Without it, every watchdog poll repeats the notification. Run the watchdog in a separate failure domain from the scheduled worker. Separate processes on the same machine are not independent when that machine stops.
Deployment deserves the same restraint. Create the ledger schema first, deploy workers that dual-write normal logs and run records, observe for several schedules, then enable alerts. Keep analytical copies downstream; the watchdog query belongs on the operational store because ingestion lag in an analytical pipeline can manufacture false misses. Analytical storage remains useful for comparing duration and failure rates across cohorts once the immediate decision is over.
Test clocks, duplicates, and partial failure
A happy-path test proves little. Freeze time one millisecond before and at the deadline. Start the same logical run twice. Complete work after the caller times out. Make alert delivery fail, then retry the watchdog. Finally, delay the analytical export while leaving the operational ledger current.
import assert from "node:assert/strict"; const deadline = new Date("2026-06-01T02:25:00.000Z"); const before = new Date(deadline.getTime() - 1); assert.equal(before >= deadline, false); assert.equal(new Date(deadline) >= deadline, true); assert.equal( logicalRunId("cohort", new Date("2026-06-01T02:00:00.000Z")), logicalRunId("cohort", new Date("2026-06-01T02:00:00.000Z")) ); Also test the calendar assumptions around the scheduler. Store scheduled instants in UTC, and keep the property locale as separate business data. If a local-time schedule is required, daylight-saving transitions can create a skipped or repeated wall-clock time; the scheduler's documented behavior must become a test fixture.
For the dashboard, show pending and running records past their deadlines, failed terminal records, retry counts, and duration distributions by experiment revision. Web performance metrics such as LCP, CLS, and INP describe user experience; they do not prove a background cohort job ran. Keep those signals separate so an attractive page-health chart cannot mask missing experiment evidence.
When is a heartbeat the better trade?
Choose a completion heartbeat when the job has no meaningful input snapshot, duplicate execution is harmless, and the only operational question is "did it finish near the expected time?" A cache refresh or disposable report may fit. The implementation is smaller, which matters to a solo operator shipping weekly.
A heartbeat is also reasonable as a temporary migration signal while the ledger runs in shadow mode. Send the logical run ID as metadata if the receiver supports opaque fields, but treat the durable ledger as the source of incident evidence. Do not send tenant identifiers.
Process uptime checks solve a different problem. They establish reachability at a moment in time, not scheduled execution. Retry libraries solve another slice: they repeat failed calls, but they cannot detect a scheduler that never invoked the worker. These tools can coexist because each covers a different failure boundary.
The final decision rule is compact: if an incident review must explain cohort membership or side effects, use a run ledger plus an independent deadline watchdog. If completion time alone is enough and duplicates have no consequence, use a heartbeat. Either way, test silence, share one deadline across retries, and keep the monitoring path free of tenant data.
