Local, Ephemeral, or CI: A Placement Decision Tree for Agent Experiments
Last Friday I had forty prompts to re-run against a changed prompt template. The suite took eleven minutes on my laptop, which meant the fan spun up, two browser tabs died, and I spent the whole time afraid to touch the keyboard. CI could have run it, but pushing forty throwaway iterations through a shared pipeline felt rude. A scratch box somewhere in the cloud was the obvious answer, except I had no idea which of the three places the job actually belonged in, and I had already wasted a morning on that same question the week before.
The DEV front page this week was full of community posts — Hacktoberfest wrap-ups, a goodbye post, a long thread about AI replacing us. Good reading, none of it answered my question. So I sat down and wrote the question out as a tree instead, once, so I would stop re-litigating it every Friday.
This article is that tree. It is about placement, not about models: given a job spec, which execution environment should run it, and why.
Glossary first, because the words do the work
Placement. The environment that executes a step or a batch of steps. Not the framework, not the model — the box.
Ephemeral runner. A machine you create for one job, run, and throw away. No state survives, which is either the point or the problem.
Reproducible from spec. Someone else can re-run the job from committed files alone — no manual steps, no uncommitted environment tweaks, no "oh, I had a .env locally".
Cold-start cost. Everything between "go" and the first useful output: image pull, dependency install, dataset download, credential exchange.
Environment drift. The gap between the machine where a job passed and the machine where it fails. Drift is invisible until you compare two runs.
Billable versus wall-clock time. CI bills queue-plus-run; a laptop bills attention; a scratch box bills neither but may disappear mid-job.
The tree
Q1. Does the job need to reach a host that only exists inside your private network? yes -> LEAF A (local, or CI with a network path) no -> Q2 Q2. Does the job need a GPU, or more than a few GB of input data? yes -> LEAF D (dedicated machine; not a scratch box) no -> Q3 Q3. Can the job be re-run from committed files in under five minutes? no -> LEAF A (local, while you make it reproducible) yes -> Q4 Q4. Does the job need long-lived secrets it should not hold on a throwaway box? yes -> LEAF C (CI with scoped credentials) no -> Q5 Q5. Will the run take more than ~45 minutes of wall clock? yes -> LEAF C (CI; survivable, resumable, logged) no -> Q6 Q6. Is the output something a second person must open, link, or gate on? yes -> LEAF C (CI as the gating run) no -> LEAF B (ephemeral runner) The 45-minute number is a default, not a law. Set it to the point where you would stop watching the terminal. Everything above assumes the job is already spec-reproducible; if it is not, the tree routes you to fix that first, which is the single most useful outcome it can produce.
Worked leaves
Leaf A — local, deliberately
You land here when the job touches an internal host, or when it cannot be re-run from committed files. The second case deserves emphasis: placement is not the bug, reproducibility is. Run it locally, capture the manual steps as you go, and re-enter the tree tomorrow.
# Capture the environment you actually used, not the one you meant to use. python -m pip freeze > runs/manual-$(date +%F)/frozen.txt Leaf B — ephemeral runner
Short jobs, no private network, no long-lived secrets, output for your own eyes. This is the case an ephemeral box was designed for: you want a clean environment, you do not want to keep it, and the whole thing is over before cold-start matters.
# Pseudocode shape; substitute your own access details. rsync -a ./evals ./jobs user@scratch-box:~/work/ ssh user@scratch-box 'cd ~/work && python -m evals.run --spec jobs/eval-small.json' rsync -a user@scratch-box:~/work/runs ./runs/ This is also the leaf where a free tier earns its keep. MonkeyCode advertises free model access and a free server option, which makes leaf B the cheapest place to burn iterations — pull a scratch box, run the suite, pull the artifacts back, destroy it. Check the project's current terms yourself before you move anything you would mind losing; free tiers change, and I am not your source of truth for what is live today.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Leaf C — CI
The gating run. Anything a second person reads, anything that takes long enough that you would close the laptop, anything holding a scoped credential. Keep CI boring: one entry point, one spec file, artifacts uploaded.
# The shape that matters: spec in, artifacts out, no interactive step. - run: python -m evals.run --spec jobs/eval-small.json --out runs/ci.jsonl - uses: actions/upload-artifact@v4 with: name: eval-run path: runs/ci.jsonl Leaf D — dedicated machine
Large data, GPU, or a model too big for a scratch box. Expensive enough that placement should be a scheduled decision, not a reflex. Do not explore here; explore in leaf B and promote the winning spec.
The artifact: a classifier that ends the argument
One file, standard library only. It encodes the tree above and prints the reasons, so the next Friday debate takes four seconds instead of a morning.
#!/usr/bin/env python3 """place.py - classify an experiment job spec into a runner placement. Standard library only. Rules are ordered; the first match wins. The thresholds are defaults; tune them to your own tolerance. """ from __future__ import annotations import json import sys from dataclasses import dataclass LONG_JOB_MIN = 45 BIG_DATA_GB = 5.0 @dataclass class Job: name: str wall_clock_min: int needs_private_network: bool needs_gpu: bool data_gb: float reproducible_from_spec: bool holds_long_lived_secrets: bool shared_artifact: bool def place(job: Job) -> tuple[str, list[str]]: why: list[str] = [] if job.needs_private_network: why.append("needs reachability into a private network") return "A: local-or-network-path-ci", why if job.needs_gpu or job.data_gb > BIG_DATA_GB: why.append(f"hardware/data floor (gpu={job.needs_gpu}, data_gb={job.data_gb})") return "D: dedicated-machine", why if not job.reproducible_from_spec: why.append("cannot be re-run from committed files yet") return "A: local-until-reproducible", why if job.holds_long_lived_secrets: why.append("would need long-lived credentials on a throwaway box") return "C: ci-scoped-creds", why if job.wall_clock_min > LONG_JOB_MIN: why.append(f"wall clock {job.wall_clock_min}m exceeds {LONG_JOB_MIN}m") return "C: ci-long-run", why if job.shared_artifact: why.append("output must be opened or gated on by someone else") return "C: ci-gating-run", why why.append("short, reproducible, self-contained, private output") return "B: ephemeral-runner", why def main() -> int: if len(sys.argv) != 2: print("usage: place.py jobs/eval-small.json", file=sys.stderr) return 2 with open(sys.argv[1], encoding="utf-8") as fh: payload = json.load(fh) job = Job(**payload) placement, why = place(job) print(f"{job.name} -> {placement}") for line in why: print(f" - {line}") return 0 if __name__ == "__main__": raise SystemExit(main()) A spec to feed it:
{ "name": "eval-small", "wall_clock_min": 11, "needs_private_network": false, "needs_gpu": false, "data_gb": 0.2, "reproducible_from_spec": true, "holds_long_lived_secrets": false, "shared_artifact": false } Applying the rules by hand, this spec lands on B: ephemeral-runner with the reason line "short, reproducible, self-contained, private output". Flip shared_artifact to true and it moves to C: ci-gating-run. Set reproducible_from_spec to false and it drops to A regardless of the other fields, which is the tree enforcing its own precondition. The outputs above are derived from the rules, not from a benchmark — the script is small enough that hand-checking is the honest verification.
A test plan for the placement itself
Placement decisions rot quietly. Three checks keep them honest:
- Determinism check. Run the same spec twice on the same leaf and diff the output hashes. A diff means the job was never reproducible and the tree lied to you.
- Drift check. Capture
pip freeze(or your lockfile) on both runs and diff them. Empty diff is the expected result for leaf B. - Cost guard. Abort if a leaf-B run exceeds its wall-clock budget; that job belongs in leaf C next time, and the guard tells you so before the box disappears.
# Sketch: two runs, two hashes, one comparison. for i in 1 2; do python -m evals.run --spec jobs/eval-small.json --out "runs/leaf-b/run-$i.jsonl" done sha256sum runs/leaf-b/run-*.jsonl python -m evals.run is a placeholder for whatever your entry point is called. The two-run comparison is the part that matters.
Limitations, and who should skip this
This tree assumes jobs are batch-shaped and mostly idempotent. If your work is interactive debugging, you want leaf A and no tree at all. If you handle regulated data, scratch boxes are the wrong tool no matter how short the job is — go straight to leaf C with scoped credentials and an audit trail. If the job needs a GPU, the tree just says "not here" and defers to your capacity planning.
And if you only ever run one command a week, ignore all of this. The overhead of choosing is larger than the cost of choosing wrong.
If you do want to try leaf B without provisioning anything, the free model access and free server option in MonkeyCode are the lowest-friction way in — read the current terms on the project page first, then run the classifier against your own spec and see whether the tree agrees with your gut. When it disagrees, the reason line is usually the interesting part.
