48-Hour Field Notes: I Moved a Nightly Job Onto a Free Server and Timed What the Cold Boot Ate

The job had been running fine on my laptop for three weeks, which is exactly the kind of evidence that means nothing at all. The moment I moved it onto a host I did not own, every assumption I had made about disks, clocks, and restarts turned into a question I had to answer with a probe instead of a shrug. Two nights later I had a nightly job that reported success and delivered an empty file, and that is the part worth writing down. If your job has never lied to you with a green exit code, you probably have not moved it yet.

Why I moved a nightly job off my own machine

The workload is boring on purpose: pull a batch of prompts, call a model, validate the JSON that comes back, and append a summary row. It takes minutes, it needs no GPU, and it fails in interesting ways when the network or the schema drifts. My laptop was fine as a host right up until it was not — lid closed, kernel updated, wifi dropped mid-run. A cheap always-on box is the obvious answer, and a free server option makes the experiment cost nothing but an evening.

The trap is that "a small Linux box" and "a small Linux box that sleeps, rebuilds, and hands you a fresh filesystem" look identical from a shell prompt. They are not identical, and the difference only shows up on the second run. So I spent the first six hours measuring the host instead of running the job.

Hour 0–6: measure the boot before you trust the job

The first artifact I wrote is a boot probe, and I now run it as the first line of every scheduled job on any host I do not control. It answers four questions in under a second: can I write here, how much room is left, is this the same machine as last night, and which environment variables actually survived.

# boot_probe.py - stdlib only, safe to run as the first step of any job import json, os, platform, shutil, socket, time, uuid from pathlib import Path def probe(workdir: Path) -> dict: started = time.time() workdir.mkdir(parents=True, exist_ok=True) probe_file = workdir / f".probe-{uuid.uuid4().hex}" writable = False try: probe_file.write_text("x" * 4096) writable = probe_file.read_text() == "x" * 4096 except OSError: writable = False finally: probe_file.unlink(missing_ok=True) usage = shutil.disk_usage(workdir) return { "host": socket.gethostname(), "boot_marker": os.environ.get("BOOT_ID", "unset"), "platform": platform.platform(), "python": platform.python_version(), "workdir": str(workdir.resolve()), "workdir_writable": writable, "disk_free_mb": round(usage.free / 1_048_576, 1), "probe_ms": round((time.time() - started) * 1000, 2), "env_keys": sorted(k for k in os.environ if k.startswith(("MONKEY", "JOB_", "RUN_"))), } if __name__ == "__main__": print(json.dumps(probe(Path(os.environ.get("WORKDIR", "./work"))), indent=2)) 

Run it on two consecutive nights and diff the output. A changing host value with a stable workdir string is the signature of a rebuild behind a friendly path. A BOOT_ID that resets tells you the machine genuinely restarted rather than resumed.

I am not going to paste timings from my run as if they were universal, because boot cost depends on image size, region, and what your job imports at startup. Treat the values above as the shape of the measurement, not measurements you can rely on. The point is that you now have a number for the cold path instead of an opinion about it.

The failure: exit code 0 and a file that lied

Here is what actually broke. The job wrote results to a local path, printed a summary, and exited zero. The next morning the summary was in my inbox, the results directory was empty, and nothing in the logs disagreed with itself. Nothing crashed. The filesystem had simply been replaced before I looked.

That is not a bug you can fix with a try/except, because no exception was raised. It is a contract problem: the job treated "wrote to disk" as "durable", and on an ephemeral host those are different words. Ask yourself honestly — after your job exits, how would you prove the output still exists thirty minutes later?

Split the system into two planes

The fix was architectural and small. Everything the job needs is either a control plane (code, config, prompts, schema) that belongs in version control, or a data plane (results, checkpoints, logs) that belongs somewhere durable and external. Ephemeral compute is allowed to be wrong about the second part only if it never claims otherwise.

Once the table is written down, most of the arguments about retries and restarts become trivial. You are no longer debugging a job; you are checking whether each row was honoured.

The receipt pattern: make success falsifiable

A green exit code is an opinion. A receipt is evidence. I now finish every run by writing the output to durable storage and then reading it back to verify the hash, which costs one extra round trip and has caught two silent failures for me.

# receipt.py - write, then verify by reading back import hashlib, json, pathlib, tempfile def write_receipt(payload: dict, remote_dir: str, run_id: str) -> dict: body = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() digest = hashlib.sha256(body).hexdigest() tmp = pathlib.Path(tempfile.mkstemp(suffix=".json")[1]) tmp.write_bytes(body) target = pathlib.Path(remote_dir) / f"{run_id}.json" target.parent.mkdir(parents=True, exist_ok=True) target.write_bytes(tmp.read_bytes()) tmp.unlink() landed = target.read_bytes() # the only check that counts  verified = hashlib.sha256(landed).hexdigest() == digest return {"run_id": run_id, "sha256": digest, "bytes": len(body), "verified": verified} 

The wrapper matters as much as the function: if verified is false, the job must exit non-zero and say so loudly. A receipt that is written but not checked is just another log line nobody reads.

A restart rehearsal you can run in twenty minutes

You do not need to wait for a real cold boot to learn whether your durable path works. Schedule the lesson instead of receiving it.

1. Run the job once, let it finish, and record the run id and the expected receipt hash.
2. Delete the entire local working directory and any cache the job created.
3. Run the job again with the same run id and confirm it resumes or refuses cleanly, never silently duplicates.
4. Kill the job mid-write with SIGKILL and confirm the next attempt does not treat a half-written file as complete.
5. Read the results back through your normal consumer path, not the writer's path, and compare hashes.
6. Run the boot probe after each step and keep the outputs side by side.

If step 3 or step 4 surprises you, stop and fix the data plane before adding features. Nothing downstream can be trusted after a silent half-write, and no amount of retry logic will recover it.

Where the free model tier actually fit

This is the honest part of the notes. The job calls a model, and the model bill was the reason the project stayed a weekend toy. MonkeyCode's free model access is what let me run the loop nightly and, more usefully, replay the same batch repeatedly through the rehearsal above without flinching at cost. The free server option is what let me stop treating my laptop as production infrastructure.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Both of those are availability claims from the project, and availability details have a habit of changing. Quota sizes, instance specs, and sleep policies are exactly the things I refuse to trust from a blog post, including this one, so check the current figures on the official project page before you design around them. What I can say from the workflow side is that a free tier is a fine place for a nightly job that writes receipts, and a terrible place for one that assumes persistence.

If you want to rehearse your own job the same way, point it at the free model access and free server option, run the boot probe first, and keep the receipt check on from the very first run.

Limitations, and who should skip this

• Do not do this if your job needs a real database with strict durability guarantees, or a runtime that must stay warm for sub-second latency.
• Do not do this if your workload is large enough that cold boots dominate total runtime; the measurement will tell you, so measure first.
• Do not do this if you cannot externalise state yet — a job that keeps its only copy of a checkpoint on local disk will fail here regardless of which host you pick.
• Free tiers are shared and can change without notice; a design that cannot tolerate a rebuilt host on short notice is a design that has not been tested.

What has changed for me is smaller than a rewrite and more useful than a migration: I no longer accept "it exited zero" as the end of a run. I want a hash, a hostname, and a boot marker that all agree. What would your nightly job have to prove before you would believe it worked while you were asleep?