One day in July our AI bill reached almost 27 USD in a single day. Nobody had asked for it. This post is the short version of what went wrong and the small guard we run so it does not happen silently again. The code is about 40 lines of Python with no extra packages.

What happened

We had an agent task that woke up every 3 minutes to check an inbox. Each wake-up was a full model call. The scheduler had registered that task 4 times, so it really ran about 80 times an hour.

That alone would have been cheap, because the task used a free model. Then the free model started answering with rate limit errors (HTTP 429). Our model router had a fallback list, and the next entry was a paid model. Every 429 quietly became a paid call. Nothing watched the spend, so we only saw it hours later.

Three mistakes stacked up:

  1. An LLM was used as a poller. Checking "is there anything new?" is a job for a script (a file check, a grep, a curl), not for a model call.
  2. A fallback changed the price class of a call. Free in, paid out, with no log line anyone read.
  3. Nobody priced the recurring task before switching it on. 80 calls an hour times a big context times the most expensive model in the fallback list is the real worst case.

The guard

OpenRouter (the paid gateway we used) has a free endpoint that tells you what a key has spent today: GET https://openrouter.ai/api/v1/key. The response has data.usage_daily, the credits used in the current UTC day, plus limit and limit_remaining.

The guard reads that number every 15 minutes and raises an alert above 2 USD a day and a louder one above 5 USD. It sends each level at most once a day, because in our setup an alert wakes an agent, and an alert every 15 minutes would be its own small burn loop.

#!/usr/bin/env python3 """spend_guard.py: warn when an OpenRouter key spends too much today. Run from cron every 15 min.""" import json import os import sys import time import urllib.request WARN_USD, CRIT_USD = 2.0, 5.0 STATE = os.path.expanduser("~/.spend_guard_state.json") URL = os.environ.get("SPEND_URL", "https://openrouter.ai/api/v1/key") def spent_today(): req = urllib.request.Request(URL, headers={"Authorization": "Bearer " + os.environ["OPENROUTER_API_KEY"]}) with urllib.request.urlopen(req, timeout=20) as r: return float(json.load(r)["data"]["usage_daily"]) def alert(level, usd): # Replace with your own channel: email, Slack webhook, Telegram bot.  print(f"{level}: AI spend today is {usd:.2f} USD", file=sys.stderr) def main(): today = time.strftime("%Y-%m-%d", time.gmtime()) try: state = json.load(open(STATE)) except (OSError, ValueError): state = {} sent = state.get(today, []) try: usd = spent_today() except Exception as e: print("could not read spend:", e, file=sys.stderr) # a guard that cannot see is an alert too  return 2 for level, limit in (("CRIT", CRIT_USD), ("WARN", WARN_USD)): if usd > limit: if level not in sent: alert(level, usd) sent.append(level) break json.dump({today: sent}, open(STATE, "w")) print(f"spent today {usd:.2f} USD, alerts sent today: {sent or 'none'}") return 0 if __name__ == "__main__": sys.exit(main())

Cron line:

*/15 * * * * OPENROUTER_API_KEY=... /usr/bin/python3 /opt/spend_guard.py >> /var/log/spend_guard.log 2>&1

Two details that matter more than the code:

  • The guard itself uses zero model calls. A watcher that calls an LLM to decide whether to alert has the same problem it is meant to catch.
  • Keep the key where only the guard can read it. In our setup a root timer reads the number and writes just the dollar figure to a file the agents can see, so no agent process can borrow the key to spend.

What we changed after

The guard tells you about a leak. It does not stop one. So we also:

  1. Deleted the LLM poller. The inbox check is now a plain watcher script that wakes the agent only when a new message arrives.
  2. Took the paid model out of the fallback list entirely. Now, when the main model fails, the request goes to other free providers or fails. A failed request is cheaper than a surprise bill.
  3. Price every recurring task before it starts: calls per hour times context size times the worst model it can fall back to.
  4. Check the scheduler for duplicate tasks. Our leak was 4 copies of one task.

Since the paid fallback was removed there is nothing left for this guard to watch on our side, so it is switched off. If you still pay per call, it is a small script to run next to a hard cap.

Use something else when

  • Your provider has a hard spending cap you can set. OpenRouter keys can have a credit limit, and OpenAI and Anthropic accounts have spend limits in their consoles. A hard cap stops the spend; this guard only tells you about it. Use both if you can.
  • You need per-feature or per-customer cost numbers. Then you want request-level logging in your app or a tool built for LLM cost tracking, not one daily total.
  • You run many keys or providers. Pull them into one monitoring system you already have instead of a script per key.

We post small automations like this on our Telegram channel, each one from something we actually built or broke: https://t.me/small_automations