I spent two nights watching a nightly job call a model once per queue row, and the failure was never the model. It was arithmetic: a fixed 120-second timeout per attempt, three attempts, and a scheduler window that closed long before the last row was reached. Why do we budget time per request but almost never per job? Here is what I tried, what broke, and the clamp I would repeat.

The setup I actually ran

  1. A queue table with one row per item, where each row needed one model call plus schema validation.
  2. A worker loop that pulled rows in order and wrote a status column after each successful call.
  3. A constant per-call HTTP timeout of 120 seconds, three attempts, and a wall-clock window enforced outside the worker.
  4. A fake-clock test suite that ran before any live call, because I wanted the failure in milliseconds, not hours.

That naive loop looked perfectly reasonable in isolation, and that is exactly why it survived review. Ask yourself what the scheduler sees when every attempt eats its full timeout:

# the version I shipped first, and regretted def process_naive(items, post): for item in items: for attempt in range(3): try: return post(item, timeout=120) # fixed, forever  except TimeoutError: continue return None

If sixty rows each burn three 120-second timeouts, the worst case is six hours inside a window measured in minutes. The process gets killed mid-row, so the row in flight has an unknown outcome while every earlier row looks fine. Nothing downstream can tell those two situations apart without reading logs by hand.

The fix: one deadline, passed down, with a reserve

Instead of a constant timeout, I pass a single monotonic deadline into the worker and derive each call's timeout from what is actually left. Three rules carry the whole change, and none of them is clever:

  • Clamp: timeout = min(cap, deadline - now - reserve).
  • Refuse: if the clamped value drops below a floor, do not start the call at all.
  • Record: log the granted timeout next to each result so you can audit the arithmetic later. 
# budget.py (needs Python 3.10+ for the union type) import time from dataclasses import dataclass @dataclass class Budget: deadline: float # time.monotonic() value for the whole job  reserve: float = 45.0 # seconds held back for checkpoint + clean exit  floor: float = 3.0 # never start a call with less usable time than this  cap: float = 120.0 # no single call may exceed this  def call_timeout(self, now: float) -> float | None: remaining = self.deadline - now - self.reserve if remaining < self.floor: return None return min(self.cap, remaining)

The reserve is the part everyone skips, and it is the part that keeps the run honest under pressure. I hold back enough seconds for the final checkpoint write, the status flush, and a clean exit before the scheduler's axe falls. Ask yourself how long your cleanup really takes when the database is busy, not when it is idle. My first reserve was too small, and the last row never got its status write at all.

The loop that defers instead of overrunning 

# worker.py import time from budget import Budget def run(queue, call_model, checkpoint, budget, clock=time.monotonic): done, deferred = [], [] for item in queue: timeout = budget.call_timeout(clock()) if timeout is None: deferred.append(item["id"]) continue try: result = call_model(item, timeout=timeout) except TimeoutError: checkpoint(item["id"], status="pending", detail="deferred") deferred.append(item["id"]) continue checkpoint(item["id"], status="done", result=result) done.append(item["id"]) return done, deferred

A deferred item is not a failure, and treating it as one is how you end up retrying into a closed window. The next run picks it up because the checkpoint still says pending, so the job converges across windows instead of dying inside one. Is that slower? Yes, deliberately, and it is the only version where the final row count means anything.

A reproducible test with no network at all 

# test_budget.py from budget import Budget from worker import run class FakeClock: def __init__(self, t=0.0): self.t = t def __call__(self): return self.t def advance(self, seconds): self.t += seconds def test_clamp_shrinks_as_the_window_closes(): clock = FakeClock(0.0) b = Budget(deadline=100.0, reserve=40.0, floor=5.0, cap=30.0) assert b.call_timeout(clock()) == 30.0 # 60 usable, cap wins  clock.advance(40.0) assert b.call_timeout(clock()) == 20.0 # 20 usable, remaining wins  clock.advance(20.0) assert b.call_timeout(clock()) is None # 0 usable, refuse  def test_slow_model_defers_instead_of_overrunning(): clock = FakeClock(0.0) b = Budget(deadline=10.0, reserve=4.0, floor=2.0, cap=3.0) granted, writes = [], [] def call_model(item, timeout): granted.append(round(timeout, 2)) clock.advance(timeout) # pretend the call used it all  return {"ok": True} done, deferred = run( [{"id": i} for i in range(5)], call_model, checkpoint=lambda item_id, **kw: writes.append((item_id, kw["status"])), budget=b, clock=clock, ) assert granted == [3.0, 3.0] assert done == [0, 1] assert deferred == [2, 3, 4] assert writes == [(0, "done"), (1, "done")]

Run it with python -m pytest -q test_budget.py, and both tests finish in milliseconds. They fail loudly the moment someone "simplifies" the clamp back into a constant, which is the reason I keep them in the repo. A fake clock is cheaper than a night of watching timestamps scroll past at 3am.

What broke in the first 48 hours

  • I used time.time() first, and one NTP step made the remaining budget jump backwards into absurd timeouts.
  • My reserve was smaller than the checkpoint write under contention, so the last row was processed but never recorded.
  • The deadline was computed once before the loop, so "remaining" never actually moved during the run.
  • Deferred rows were logged as errors, which paged me for a job that was, by its own definition, healthy.

Each of those is a one-line fix and a two-hour investigation, which is roughly the usual ratio for this kind of plumbing.

Where MonkeyCode came in

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I ran the worker on MonkeyCode's free server option and pointed call_model at its free model access, which let me exercise the clamp against a real endpoint before paying for any infrastructure. Both of those availability claims are operator-supplied and can change, so read the current terms yourself rather than planning around my notes. The clamp code does not care which endpoint answers, which is precisely the point of testing it separately.

If you want to rehearse this yourself, start with the fake-clock tests, then do one supervised run against a free endpoint and watch the granted timeouts shrink as the window closes.

Who should not use this approach

  • Workloads with sub-second calls, where a three-second floor never triggers and the extra branch is just noise.
  • Non-idempotent side effects, because deferring a half-applied write is worse than an honest timeout.
  • Jobs that must complete inside a single window for correctness; split the work instead of stretching the budget.
  • Pipelines where the scheduler already hands you a per-item deadline you can read directly.
  • Anything that needs a latency guarantee, since a budget is an accounting tool and not an SLA.

Checklist before the next run

  1. Compute the worst case as items × attempts × cap, then compare it against the window out loud.
  2. Pick the reserve from measured cleanup time, not from optimism about the database.
  3. Assert the clamp and the refusal path in a fake-clock test.
  4. Log granted timeout, elapsed time, and remaining-at-start for every single call.
  5. Treat deferred rows as pending forever, and never as errors that trigger a page.

Two nights of field notes bought me one arithmetic identity, a small test file, and a job that now ends before its window does. Isn't that a better trade than another round of guessing at why the last rows look weird?