Case Study: A Preemption-Safe Job Runner for a Model-Backed Report on a Free Server

The hardest part of running a nightly job on a free server is not CPU, memory, or disk — it is that the machine can stop existing mid-run without telling you. If your job assumes it will finish what it started, you get two bad outcomes: duplicated work when the box wakes up and reruns, or silent gaps when it never comes back. This case study walks through one small project where I designed for preemption first and performance second, and the artifact you can copy is a journal-based runner that survives being killed at any line.

The trend worth stealing from the last few months of AI tooling news is not a new model, it is placement: agent-style work is moving out of chat windows and into scheduled CI jobs. That shift makes boring distributed-systems questions relevant again, because a job that calls a model now has a nondeterministic step inside a pipeline that used to be deterministic. Free compute makes those questions sharper, since a free box is exactly the environment where you cannot assume stability.

Background: what the job actually does

The project is a nightly CI health digest for three personal repositories. It pulls the last 24 hours of failed workflow runs, groups them by a normalized error signature, asks a model to compress each group into a short human sentence, and posts five bullets to a private webhook. I run the model step through MonkeyCode's free model access and host the cron job on its free server option, which is what makes the preemption question load-bearing instead of theoretical.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Both of those are availability claims from the operator, not measurements I can generalize, so budget accordingly and check the current limits on the site before you design around them. The point of this write-up is the runner, which works the same whether your box is free, spot-priced, or a laptop you close at night.

Goal and non-goals

1. Goal: the job must be resumable after an arbitrary kill, with no duplicated webhook posts.
2. Goal: a cold start must not redo work that was already committed.
3. Goal: the whole run must fit a bounded wall-clock budget, whatever the box does.
4. Non-goal: throughput. This job processes tens of items, not millions.
5. Non-goal: exactly-once model calls. The model step is retried cheaply; the side effect is what needs fencing.

The design: three invariants

Everything below follows from three rules, and if you only remember the rules you can reimplement the code in any language.

• Journal before you call. A chunk is claimed in durable storage before the model call starts, so a crash leaves evidence instead of amnesia.
• Key by content, not by time. The chunk ID is a hash of the job name, the target date, the chunk index, and the input payload. Rerunning the same day produces the same IDs, so a resumed run converges on the same set.
• Separate compute from effect. Writing a result is a local atomic operation; delivering it (the webhook post) is a second, fenced step that only reads committed rows.

That third rule is the one people skip. If you post to Slack inside the same function that calls the model, a kill between the two leaves you unable to tell whether the post happened, and you end up choosing between duplicates and silence.

Implementation: a journal, a lease, and an atomic commit

Here is the runner I actually use, trimmed to the parts that matter. It is Python 3.11 with no dependencies beyond the standard library.

# runner.py — preemption-safe chunked job runner import hashlib, json, os, pathlib, sqlite3, sys, time STATE = pathlib.Path(os.environ.get("JOB_STATE", "./state")) STATE.mkdir(parents=True, exist_ok=True) DB = STATE / "journal.sqlite" LEASE_TTL_S = 900 def connect(): conn = sqlite3.connect(DB, timeout=5.0, isolation_level=None) conn.execute("PRAGMA journal_mode=WAL") conn.execute(""" CREATE TABLE IF NOT EXISTS chunk ( id TEXT PRIMARY KEY, status TEXT NOT NULL DEFAULT 'pending', claimed_at REAL, output TEXT, delivered INTEGER NOT NULL DEFAULT 0 )""") return conn def chunk_id(job: str, day: str, index: int, payload: dict) -> str: blob = json.dumps([job, day, index, payload], sort_keys=True).encode() return hashlib.sha256(blob).hexdigest()[:32] 

The lease is crash recovery, not a rate limiter: a claim older than the TTL is presumed dead and can be taken over. If you need concurrency control, that is a different component with different semantics.

def claim(conn, cid: str, ttl: int = LEASE_TTL_S) -> bool: now = time.time() conn.execute("BEGIN IMMEDIATE") try: conn.execute("INSERT OR IGNORE INTO chunk(id) VALUES(?)", (cid,)) cur = conn.execute( "UPDATE chunk SET status='claimed', claimed_at=? " "WHERE id=? AND (status='pending' " " OR (status='claimed' AND claimed_at < ?))", (now, cid, now - ttl), ) conn.execute("COMMIT") return cur.rowcount == 1 except Exception: conn.execute("ROLLBACK") raise def release(conn, cid: str) -> None: conn.execute("UPDATE chunk SET status='pending', claimed_at=NULL WHERE id=?", (cid,)) 

The commit writes bytes to a temp file, renames it into place, and only then flips the row. os.replace is atomic on POSIX and on Windows, so a kill either leaves the old file or the new one — never a half-written JSON document.

def commit(conn, cid: str, payload: dict) -> None: target = STATE / f"{cid}.json" tmp = target.with_suffix(".json.tmp") tmp.write_text(json.dumps(payload, indent=2, sort_keys=True)) os.replace(tmp, target) conn.execute( "UPDATE chunk SET status='done', output=? WHERE id=?", (target.name, cid) ) 

The loop, and the budget clock

Use time.monotonic() for the budget, never time.time(). Wall clock can jump on NTP correction or suspend/resume, and a budget computed from it can go negative or give you an extra hour you do not have.

def run(chunks, budget_s: int = 240, reserve_s: int = 20) -> int: deadline = time.monotonic() + budget_s conn = connect() for cid, payload in chunks: if time.monotonic() >= deadline - reserve_s: print("out of budget, exiting cleanly", file=sys.stderr) return 1 if not claim(conn, cid): continue try: commit(conn, cid, worker(payload)) # may call the model  except Exception as exc: release(conn, cid) print(f"chunk {cid} failed: {exc}", file=sys.stderr) return 0 

Exit code 1 on an exhausted budget is intentional: it tells your cron wrapper that the run was partial, so the wrapper can schedule another pass instead of assuming the night is done. The delivery step is separate and idempotent from the runner's point of view:

def flush(conn, deliver) -> None: rows = conn.execute( "SELECT id, output FROM chunk WHERE status='done' AND delivered=0 LIMIT 50" ).fetchall() for cid, name in rows: deliver(json.loads((STATE / name).read_text())) conn.execute("UPDATE chunk SET delivered=1 WHERE id=?", (cid,)) 

There is still a window between deliver(...) and the delivered=1 write. Close it by making delivery itself idempotent — pass the chunk ID as the webhook's dedupe key, or post to a staging channel and promote. The journal gets you from "probably duplicates" to "one narrow, testable window".

Measuring the box instead of guessing about it

Before tuning chunk sizes, find out whether the machine sleeps or merely goes idle. On Linux you can distinguish the two without any monitoring stack: CLOCK_MONOTONIC stops counting during suspend, while CLOCK_BOOTTIME keeps counting.

# probe.py — drop in cron every 5 minutes, append one line per wake import json, pathlib, time HERE = pathlib.Path(__file__).parent PREV = HERE / ".probe_prev.json" LOG = HERE / "probe.jsonl" def clock(name): try: return time.clock_gettime(getattr(time, name)) except (AttributeError, OSError): return None now = {"wall": time.time(), "mono": clock("CLOCK_MONOTONIC"), "boot": clock("CLOCK_BOOTTIME")} prev = json.loads(PREV.read_text()) if PREV.exists() else None row = {"at": now["wall"]} if prev and prev["boot"] and now["boot"]: wall_gap = now["wall"] - prev["wall"] boot_gap = now["boot"] - prev["boot"] row["wall_gap"] = round(wall_gap, 1) row["suspend_gap"] = round(boot_gap - (now["mono"] - prev["mono"]), 1) row["missing_ticks"] = max(0, round(wall_gap / 300) - 1) with LOG.open("a") as fh: fh.write(json.dumps(row) + "\n") PREV.write_text(json.dumps(now)) 

Read the output by column, not by eye. A large suspend_gap with a small missing_ticks means the box slept between ticks; a small suspend_gap with a large missing_ticks means cron simply did not fire, which is a scheduler problem and not a sleep problem. Those two failures need opposite fixes, and this is the cheapest way to tell them apart.

Results: what the harness changed

I ran the probe and the runner together against the free server option and I am deliberately not printing a benchmark table, because a free tier's wake pattern is a moving target and a copied number would mislead you more than help. What I will say is qualitative and reproducible: in the runs I did, the job's own execution time was consistently shorter than the gaps between observed ticks, so resumability — not speed — was the only property that mattered.

Fill in your own table before you commit to a design:

The decision table for whether a free box is the right host at all is shorter than people expect:

• Fits: idempotent, chunked, no SLA, tolerates a missed night, effects behind a fence.
• Does not fit: anything that must run within a five-minute window, anything that holds a long-lived TCP connection, anything where a duplicate post is a customer-visible incident.

Limitations and who should not use this

This approach adds a state file, a lease, and a delivery step to a job that used to be fifteen lines of cron. That is real cost, and it is only worth paying when interruption is likely. If your job runs on a host you control with a stable schedule, skip the journal and keep the cron line.

Three more honest caveats. The lease can double-run a chunk if your worker is slower than the TTL, so keep the TTL generously above your slowest observed call. SQLite in WAL mode is fine for one writer and hostile to a shared network filesystem, so keep JOB_STATE on local disk. And none of this makes a model call deterministic — it makes the consequences of an unpredictable call survivable, which is a different and much cheaper property to engineer.

What I would keep if I rebuilt it

The runner survived contact with a sleepy box, and the parts I would keep unchanged are the content-hashed chunk IDs, the atomic commit, and the fenced delivery step. The part I would rewrite first is the delivered flag, because a boolean cannot express "posted, but the confirmation never came back"; a three-state column plus a provider-side dedupe key is the honest model of what actually happens.

If you are wiring a scheduled job to a model for the first time, this runner is a reasonable skeleton to start from, and MonkeyCode's free model access and free server option are a low-cost place to shake out the failure modes before you pay for anything. Whatever you pick, run the probe for a week first — the shape of your host's awakeness, not its specs, is what your design has to match.