We moved a busy local news site to a new server. The migration checklist was long and we were proud of it: files, database, media, DNS, TLS, redirects, image paths, mail, a performance pass. The home page loaded in half a second instead of four. Editors logged in, wrote stories, and published them. Every check we had was green.

Nineteen days later, during a routine look at table sizes, one number stood out. A small table that normally holds a few thousand rows had close to 600,000. Nothing was broken from the outside. The site was fast, pages returned 200, nobody had complained.

The table was big because the job that trims it had not run once since the migration. Neither had any other background job. The application's scheduler, a single crontab line, had never been added on the new server.

This post is about why that kind of failure is invisible, what it actually cost us, and the three checks we now treat as part of every move.

How the scheduler works in this application

The CMS has one scheduler entry point. The operating system runs it every minute from cron:

* * * * * /path/to/php /path/to/app/scheduler.php > /dev/null 2>&1

That one script reads a list of jobs from the database, each with its own interval, and runs whichever are due. The list is long: publishing posts that were scheduled for a future time, rolling raw visit logs up into hourly statistics, deleting old rows from logging and rate-limit tables, rebuilding sitemaps, refreshing small data widgets, sending queued notifications.

This design is convenient. Adding a job is a database row, not a server change. It is also a single point of failure that lives outside the application's code, in the system crontab of whatever user the site runs as. Copy the files and the database, and you have copied the job list but not the thing that runs it.

Why nobody noticed for 19 days

Look at what each job does and ask who would see it fail.

  • Scheduled publishing. Editors who schedule a story for 06:00 would notice when it doesn't appear. On this site the newsroom publishes immediately; nobody had scheduled anything. We checked afterwards: no posts were stuck in a scheduled state.
  • Statistics roll-up. The admin dashboard reads today's numbers from the raw log, so "today" looked fine. Only longer ranges, built from the hourly summaries, were going stale, and nobody had opened a monthly report yet.
  • Cleanup jobs. Their whole effect is that something doesn't happen: a table doesn't grow. A table that is growing gives no signal until it is big enough to slow a query or fill a disk.
  • Data widgets. Currency rates, weather and football fixtures were the only visibly stale things. We had actually noticed those during the migration and fixed them by adding a few dedicated cron lines, one per widget. That fix made the problem harder to see: the visible symptoms went away, so the missing scheduler line looked like it had been handled.

Our uptime monitor checked the home page and an article page. Both were fine. A health check that only asks "does the page render" will never see a job that didn't run, because the page doesn't depend on it today.

There is a general rule hiding here: the failure of a background job is invisible until the job's absence becomes a foreground problem. Cleanup jobs take weeks. Aggregation jobs take until someone opens a report. Publishing jobs take until someone schedules a post.

What it cost

In this case, less than it could have. No content was lost and nothing went down. The damage was:

  • A rate-limit and request log table with roughly 594,000 rows that should have held a few days of data. Inserts into it happen on every request, so it was slowly making every request a little more expensive.
  • Nineteen days without hourly statistics summaries. Summaries can only be rebuilt from raw logs that still exist. Here the raw logs were still there, because the cleanup hadn't run either. If the roll-up had stopped while the cleanup kept running, the raw data behind those reports would have been deleted on schedule.
  • A few days of security log retention that was longer than our own policy says it should be.

The more uncomfortable cost is what would have happened on a different site. On a site that relies on scheduled publishing, the morning's stories would have stayed drafts, and the newsroom would have discovered it at 06:01 on a weekday.

The fix itself was one line 

crontab -u siteuser -l > crontab-before.txt # keep the old state ( crontab -u siteuser -l 2>/dev/null; \ echo '* * * * * /path/to/php -d date.timezone=Europe/Istanbul /path/to/app/scheduler.php > /dev/null 2>&1' \ ) | crontab -u siteuser -

Two details mattered more than the line:

  1. The per-widget lines had to go. They were now duplicates of jobs the scheduler also runs. Two copies of the same job, running at slightly different times, is its own source of subtle bugs.
  2. The site sat behind a CDN proxy, and some jobs call the site's own URLs. From the server, those requests went out to the CDN and back in, picking up rate limits and bot checks on the way. Pointing the site's own hostnames at the local machine in /etc/hosts keeps self-requests local.

Before turning the scheduler on, we checked for posts with a publish time in the past that were still waiting. A scheduler that has been off for weeks will publish all of them in its first minute. On a migrated site those can be old test posts, so look before you enable.

After enabling it, the job table told us within two minutes whether it worked:

SELECT name, last_run, last_status FROM scheduled_jobs ORDER BY last_run DESC LIMIT 10;

The jobs started showing fresh last_run values, and the oversized table was left to the regular nightly cleanup instead of a manual DELETE during business hours.

Three checks we now run on every migration

1. Diff the crontabs, not just the files

Before the move, dump every crontab that touches the site: the site user's, root's, and any system cron file that mentions the site's path.

crontab -u siteuser -l crontab -l | grep -F '/path/to/app' grep -rlF '/path/to/app' /etc/cron.d /etc/crontab 2>/dev/null

After the move, run the same commands on the new server and compare. Any line that exists on the old server and not the new one is either a deliberate decision, written down, or a bug. We had migrated the files with rsync, the database with a dump, and the crontab with memory. Memory lost.

2. Monitor "time since last run", not "is the site up"

The scheduler already wrote a last_run timestamp for every job. Nothing was reading it. We added a check to our regular server health script that asks one question per site:

SELECT COUNT(*) AS stale FROM scheduled_jobs WHERE enabled = 1 AND last_run < NOW() - INTERVAL 10 MINUTE AND run_every_minutes <= 5;

If a job that should run every few minutes has not run in ten, the check warns. It also warns if the site has no scheduler line in any crontab at all. This turns an invisible failure into a visible one within minutes instead of weeks.

If your application doesn't record job runs, the cheapest version is a heartbeat: have the scheduler touch a file or ping a URL every run, and alert when the file's age passes a threshold. Dead-man's-switch services exist for exactly this, and a ten-line script against a timestamp does the same job.

3. Put the scheduler in the definition of "migrated"

Our checklist said "copy cron jobs" somewhere in the middle. It now says, as the last step before anyone signs off:

The scheduler is running on the new server. Proof: the job table shows a last_run within the last two minutes for every enabled job.

A checklist item that needs a timestamp as evidence is hard to tick by accident. "Copy cron jobs" was easy to tick after adding the six widget lines, because cron jobs had, in fact, been copied.

What to take from this

  • A scheduler that lives in the system crontab is part of your deployment, even if every job it runs lives in your database.
  • Background job failures are silent by design. Plan for that by monitoring their timestamps, because their effects will show up late and somewhere else.
  • When you fix a symptom during a migration, ask what produced it. We fixed stale widgets one by one and missed that they all had the same cause.
  • Before re-enabling a scheduler that has been off, look at the queue it will process. Days of backlog can run in one minute.

We run a small fleet of news sites on the same CMS at Alesta WEB, and the stale-job check now runs against all of them as part of regular server maintenance. It is a few lines of SQL, and it would have caught this on day one.