Cron Jobs That Never Ran: Heartbeats & Dead-Man Switches for SaaS

The nightly invoice job has a try/catch, an error alert, and a Slack webhook for failures. It has been silent for eleven days. Nobody noticed, because silence is exactly what a healthy job looks like. The job did not fail. It never started. The server was rebuilt, the crontab entry was not, and every alert you wrote only fires when code runs.

I wrote this with help from AI tools and checked every config, command and code sample myself. I am a Principal Software Engineer in Karachi with 9+ years across PHP/Laravel, Vue, Node and .NET microservices, and at Loop Origin we build and maintain SaaS backends where scheduled work quietly carries billing, reports and cleanup. This is Part 1 of a new series, Silent Failures in Production SaaS: problems that do not throw errors, so error monitoring never sees them.

📚 Series Roadmap — Silent Failures in Production SaaS

• Part 1 (This Post): Cron Jobs That Never Ran: Heartbeats & Dead-Man Switches for SaaS
• Part 2: Webhooks That Stopped Arriving: Gap Detection & Reconciliation
• Part 3: Email That Never Landed: SPF/DKIM/DMARC Drift & Bounce Pipelines
• Part 4: Config & Feature Flags That Quietly Diverged Across Environments
• Part 5: Backups That Never Restored: Automated Restore Drills
• Part 6: Synthetic Checks & SLOs for the Paths Nobody Clicks

Related earlier posts: Delayed Jobs, Schedulers & Clock Skew Gotchas · Observability for Async Work: Correlation IDs & Job Traces · Production Observability on a Budget

How do you detect a cron job that never ran?

Direct answer: Invert the alert. Instead of the job reporting failures, have it report success to a monitor on every run, and alert when an expected report does not arrive within a grace window. This dead-man switch catches deleted crontabs, stopped schedulers, crashed workers and hung jobs, none of which raise exceptions or reach your error tracker.

Why error alerts cannot see this class of failure

Error monitoring answers "did code that ran throw?" Silent scheduler failures happen before or around your code:

• The crontab line for schedule:run was lost during a server rebuild or image change.
• The scheduler container was scaled to zero, or a deploy left it in a crash loop.
• withoutOverlapping() left a stale lock after a killed process, so every later run skipped.
• The job started, then hung forever on a network call with no timeout.
• A timezone or DST change shifted a run outside the window you expected.
• Someone commented the job out "temporarily" in a hotfix.

In every case the error rate stays at zero. That is why I treat "absence of a signal" as its own monitoring category.

The heartbeat pattern

A heartbeat monitor stores the time of the last successful check-in for each job, plus its expected schedule and a grace period. A separate checker, running somewhere other than the server it watches, alerts when now > last_success + interval + grace. Hosted services such as Healthchecks.io, Cronitor and Better Stack implement this, and many uptime tools now offer it, but the logic is simple enough to own.

Laravel: built-in pings on the scheduler

Laravel's scheduler can call a URL around each task. pingBefore and thenPing mark start and finish, and pingOnFailure / pingOnSuccess split outcomes. These use Guzzle, which ships with standard Laravel installs.

// routes/console.php (Laravel 11+) or app/Console/Kernel.php on older apps use Illuminate\Support\Facades\Schedule; Schedule::command('billing:generate-invoices') ->dailyAt('01:30') ->timezone('Asia/Karachi') ->onOneServer() ->withoutOverlapping(60) // lock expires after 60 minutes, not 24 hours ->pingBefore(env('HB_INVOICES_URL').'/start') ->pingOnSuccess(env('HB_INVOICES_URL')) ->pingOnFailure(env('HB_INVOICES_URL').'/fail'); 

Two details from real incidents. First, withoutOverlapping() with no argument holds its lock for 24 hours by default, so a killed run can silently skip a full day; set an expiry close to the job's realistic maximum. Second, the start ping is what turns a hung job into an alert: if the monitor sees /start but no finish within the expected duration, it fires.

Also add a heartbeat for the scheduler itself, not just individual jobs:

Schedule::call(fn () => Http::timeout(5)->get(env('HB_SCHEDULER_URL'))) ->everyFiveMinutes() ->name('scheduler-heartbeat'); 

If this one goes quiet, every scheduled task on that app is suspect.

Node.js: wrap the job, report counts

In Node services the same idea works around whatever runner you use, whether that is system cron, a Kubernetes CronJob or a queue-based repeatable job.

// heartbeat.js export async function withHeartbeat(url, fn, { timeoutMs = 15 * 60_000 } = {}) { await fetch(`${url}/start`, { signal: AbortSignal.timeout(5000) }).catch(() => {}); const timer = setTimeout(() => { console.error("job exceeded timeout"); process.exit(2); }, timeoutMs); try { const processed = await fn(); // Report the count so the monitor or a log alert can flag "succeeded with zero rows". await fetch(`${url}?rows=${processed}`, { signal: AbortSignal.timeout(5000) }); return processed; } catch (err) { await fetch(`${url}/fail`, { method: "POST", body: String(err), signal: AbortSignal.timeout(5000) }).catch(() => {}); throw err; } finally { clearTimeout(timer); } } 

The hard exit on timeout is intentional. A job that is stuck forever is worse than one that crashes, because a crash at least produces a signal.

Owning the checker yourself

If you prefer not to add a vendor, a small table and one checker job are enough. The important rule is that the checker must run on different infrastructure from the jobs it watches, otherwise a dead server takes down both the job and its watchdog.

CREATE TABLE job_heartbeats ( job_name text PRIMARY KEY, interval_sec integer NOT NULL, grace_sec integer NOT NULL DEFAULT 600, last_start_at timestamptz, last_success_at timestamptz, last_rows integer ); -- Overdue jobs: never succeeded in interval + grace SELECT job_name, last_success_at FROM job_heartbeats WHERE last_success_at IS NULL OR last_success_at < now() - make_interval(secs => interval_sec + grace_sec); 

Run that query every few minutes from a separate small service or a managed function, and send results to the same paging channel as your error alerts. Store timestamps in UTC; the clock-skew pitfalls from the scheduler post in the previous series apply here too.

Choosing grace windows that do not cry wolf

Alert fatigue is the reason teams switch these monitors off, so start generous and tighten once you have a few weeks of real durations.

Rolling this out across an existing SaaS

When we add this to a client's system at Loop Origin, the order that pays off fastest is:

1. List every scheduled task (php artisan schedule:list, crontabs, CronJobs, repeatable queue jobs).
2. Mark which ones touch money, customer-visible data or compliance.
3. Add a scheduler-level heartbeat first, then per-job pings for the critical list.
4. Add row counts for jobs where "nothing processed" is itself suspicious.
5. Document each monitor next to the job definition so a rebuild cannot forget it.

If you want help wiring this into an existing Laravel stack, our Laravel development work covers scheduler hardening, and our server setup and maintenance service includes keeping crontabs, workers and monitors in your provisioning code.

Diagnostic checklist

• [ ] I have a complete list of scheduled jobs across all servers and runtimes.
• [ ] The scheduler itself sends a heartbeat at least every five minutes.
• [ ] Every money-, data- or compliance-related job pings on start and success.
• [ ] A missing success ping alerts within interval plus grace.
• [ ] The heartbeat checker runs on separate infrastructure from the jobs.
• [ ] withoutOverlapping locks have explicit expiries.
• [ ] Every job has a hard timeout.
• [ ] Jobs report processed counts, and zero triggers a review for jobs where zero is unusual.
• [ ] Crontab or scheduler setup lives in provisioning code, not only on the server.
• [ ] I tested the alert by disabling one job in staging and waiting for it to fire.

FAQ

Isn't uptime monitoring enough?

No. Uptime checks confirm a web endpoint responds. A scheduler can be dead while the website is perfectly healthy.

Where should the heartbeat checker run?

Anywhere that does not share failure modes with the jobs: a different server, a managed function, or a hosted monitoring service.

Should I ping on every run of a job that runs each minute?

You can, but a scheduler-level heartbeat plus per-job pings for important jobs is usually enough and keeps noise down.

What if the ping request itself fails?

Use short timeouts and never let a failed ping fail the job. A missed ping means a false alert, which is far better than a hidden failure.

How is this different from queue monitoring?

Queues need depth and age alerts. Heartbeats answer a different question: did work that should start on a schedule actually start and finish?

Do Kubernetes CronJobs need this too?

Yes. Suspended CronJobs, missed schedules past startingDeadlineSeconds, and image pull failures can all mean no run and no application error.

Next in this series, Part 2 covers webhooks that quietly stop arriving, and how to detect delivery gaps and reconcile against the provider's API.

Follow & Connect

• Website: Loop Origin
• LinkedIn: Loop Origin
• X: Muhammad Ali Mughal (@AliMughal887)
• GitHub: Loop Origin (@LoopOrigin)