Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

This is a hypothetical case study, assembled from a pattern that turns up again and again with webhook integrations. The company, timeline and numbers are invented.

The setup

A company runs an online booking platform. A third-party payments provider sends it webhooks for completed payments, refunds and disputes. When a payment webhook arrives, the platform confirms the booking and emails the customer.

The integration had been stable for two years. The handler verified signatures, stored events, returned 200 quickly, and processed in a background job. By most standards it was a good integration.

Friday evening: a deploy

On Friday at 18:40, a routine deploy upgraded the web framework. Among other things, the upgrade changed the default body-parsing middleware. The webhook route, which needed the raw body for signature verification, now received parsed JSON. Every signature check failed, and the handler, written to treat verification errors as unexpected exceptions, returned 500.

The error monitoring tool did catch the exceptions. It grouped them under one issue titled SignatureVerificationError, which an on-call engineer saw on Saturday morning, assumed was someone probing the endpoint with fake requests (that happened sometimes), and snoozed.

Monday: the rollback that wasn't enough

On Monday morning, support noticed customers asking why their bookings were still "pending payment". The team found the deploy, rolled back the middleware change, confirmed that a test webhook from the provider's dashboard succeeded, and closed the incident. Bookings from the weekend were fixed with a manual script that queried the provider's API.

Everyone went back to work.

A weekend of 500s is a retry and alerting problem as much as a code problem. delay.run retries the callback with backoff and records each attempt, so a failing endpoint doesn't fail silently.

What nobody saw

The provider's rules said that an endpoint failing continuously for more than a certain period would be disabled automatically, and an email would be sent to the account's technical contact. The endpoint had failed for about 62 hours. On Sunday night, the provider disabled it and sent the email.

The technical contact on the account was a developer who had left the company eight months earlier. The email went to a deactivated mailbox.

The test webhook on Monday succeeded because the dashboard's "send test event" button delivers to the configured URL regardless of the endpoint's status. It proved the code worked. It didn't prove the provider was sending real events.

From Monday onwards, the platform received no webhooks at all. Payments still succeeded at the provider. Customers were charged. Bookings stayed "pending payment".

How it was found

Nine days later, the finance team ran a monthly reconciliation and found that the number of paid bookings in the platform was about 40% lower than the number of payments in the provider's report. Why only 40%? Because a fallback added years earlier polled the provider's API for bookings left pending for more than six hours and confirmed some of them. It was slow, only covered one payment method, and had been forgotten. It quietly saved about 60% of the bookings.

The remaining customers had been charged without confirmation emails. Many had already emailed support, and those tickets had been handled one at a time as "payment confirmation delayed".

The fixes

The immediate fix took a minute: re-enable the endpoint in the provider's dashboard, then replay the missed events from the provider's event log. The longer list of changes was more interesting.

1. Alert on absence, not just on errors

The team added a metric for "minutes since last webhook received", per event type they depended on. During business hours, payment webhooks normally arrived every few minutes. An alert now fires if none arrives for 45 minutes during the day or three hours at night. In this incident, that alert would have fired early on Monday, after the rollback, while the endpoint was disabled.

2. Monitor the endpoint's status at the provider

The provider's API exposes the status of each webhook endpoint. A scheduled job checks it every 15 minutes and pages if any endpoint isn't enabled. One API call, and it turns the provider's silent decision into an alert.

3. Fix the contact details

Every third-party account now has a shared team address as its technical and security contact, reviewed quarterly. That one stale email address turned a two-day incident into an eleven-day one.

4. Return 400 for verification failures

Signature failures now return 400, not 500, and are logged as a separate metric. A sudden spike in verification failures after a deploy is now its own alert, distinct from random probing. It's also worth knowing that some providers treat 4xx and 5xx responses differently when deciding whether to retry or disable.

5. Make the reconciler the real safety net

The old polling fallback became a proper reconciliation job: every 15 minutes, for every payment method, compare payments at the provider with bookings in the platform, confirm any that are missing, and report how many it fixed. A non-zero number is a signal that the webhook path is broken somewhere.

What I'd take from it

  • "Test webhook succeeded" is not "webhooks are flowing". Check real traffic, not the test button.
  • Senders have rules about failing endpoints. Read them before you need them: retry window, disable threshold, how you'll be told.
  • Silence is an outage. Alert on the absence of expected events.
  • Reconciliation is not optional for money. It saved most of the bookings, even in a neglected form.

Checks like "has a payment webhook arrived in the last 45 minutes?" and "is the endpoint still enabled?" are small scheduled jobs. delay.run can run them, and so can cron on any server you already have.

Related reading

Have you had a sender quietly disable one of your webhook endpoints? How did you find out, and how long had it been off?