Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

Most backend engineers learn the same rule early: store and schedule everything in UTC. UTC has no daylight saving time, no hour that happens twice, no hour that doesn't exist. Logs line up across servers. Cron on a server set to UTC does exactly what it says.

Then a product manager asks for "a daily digest at 8 a.m. for each user", and the rule stops being obviously right. A digest at 0 8 * * * UTC arrives at 8 a.m. in London in winter, 9 a.m. in summer, 3 a.m. in New York, and 5 p.m. in Tokyo.

So where should the time zone live: in the scheduler, or nowhere near it? Let's argue about it.

The case for UTC everywhere

It's unambiguous. Every UTC time exists exactly once. With local time, 02:30 on a spring-forward day doesn't exist, and 01:30 on a fall-back day happens twice. Every scheduler that supports local time has to choose what to do in those cases, and most of them choose differently.

It's easy to reason about in operations. When an incident happens at 14:00 UTC, everyone can see which jobs ran at 14:00 UTC. With per-user time zones, "what ran at 14:00?" becomes a query across hundreds of zones.

Time zone rules change. Countries change their DST rules, sometimes with only weeks of notice. A UTC schedule doesn't care. A local-time schedule depends on the tz database on every machine being up to date.

Load is predictable. If 70% of your users are in Europe and you schedule everything at "8 a.m. local", you get a huge spike at 07:00 or 08:00 UTC depending on the season. UTC scheduling lets you spread work deliberately.

Convert at the edges. Store UTC, compute the next run time in UTC, convert to local only for display. That's the standard advice, and it works for most systems.

The case for the user's time zone

Users think in local time. "Every Monday at 9" means 9 a.m. on the wall clock, in summer and winter. A UTC schedule drifts by an hour twice a year for anyone in a DST zone, which users see as a bug, because it is one, from their point of view.

"Convert at the edges" doesn't work for recurring schedules. You can convert a single future time to UTC and store it. You can't convert "every day at 08:00 Europe/Berlin" to a single UTC cron expression, because the UTC offset changes during the year. Either the scheduler understands time zones, or something has to recompute every schedule twice a year.

Business rules are local. "Charge the card at the start of the customer's day", "send the reminder the evening before the appointment", "don't send texts at night" are all local-time rules. If the scheduler doesn't know the zone, every job has to re-implement it.

The edge cases are knowable. Nonexistent and repeated hours are a solved problem. Pick a rule (skip forward for the gap, run once for the overlap), document it, test it. It's work, but finite work.

Where I've landed

I think the answer depends on whose schedule it is:

  1. System schedules run in UTC. Backups, cleanups, reports for the ops team, cache warming. Nobody experiences them on a wall clock, and UTC keeps them simple.
  2. User-facing recurring schedules store the user's IANA zone (Europe/Berlin, not +01:00 and not CET) alongside a local rule ("08:00 daily"). The scheduler computes each next run in that zone, then stores the next run time in UTC.
  3. Compute one run at a time. After each run, compute the next one from the rule, the zone and the current tz database. Don't precompute a year of UTC timestamps; DST rules might change before you get there.
  4. Decide the gap and overlap rules explicitly. For a job at 02:30 local on a spring-forward night, either run at 03:00 or skip. For 01:30 on a fall-back night, run once, at the first occurrence. Write the rule down and test both transitions.
  5. Log both times. Every run log should show the UTC time and the user's local time. That one change makes "my report came an hour early" tickets trivial to answer.

If the scheduler doesn't understand time zones, every recurring job ends up re-implementing them. delay.run runs the HTTP job on the schedule you set and logs every run, so you can see exactly when a job fired and compare it with the time the user expected.

The parts I'm still unsure about

  • Users who travel. Should "every morning at 7" follow the phone's current zone or the account's home zone? Calendar apps have gone both ways, and both annoy someone.
  • Zones that change rules. When a country abolishes DST, existing schedules suddenly shift relative to UTC. Is that what users want? Usually yes, but not for the server-side report that feeds a UTC-based partner system.
  • Teams spread across zones. A team digest at "9 a.m." for a team in three countries has no right answer. Some products ask for an "owner's zone", some let each member choose, and both are compromises.
  • Monthly and "last day" schedules. "On the 1st at 00:00 local" is a different instant for every customer, which matters a lot for billing and usage cut-offs. Using UTC for billing boundaries is simpler, but customers in UTC+10 will see invoices dated "yesterday".

A quick test for your system

Pick a recurring user-facing job and answer:

  • What time does it run for a user in America/New_York the week after the clocks change?
  • What happens for a user whose job is scheduled at 02:30 on the night clocks go forward?
  • Can you tell, from your logs alone, what local time a given run happened at for that user?

If any answer is "I'd have to check", that's the argument for making the time zone a first-class part of the schedule.

Related reading

Which side are you on? Do your user-facing schedules store a zone, or does everything run in UTC and users live with the drift?