Budget the Free Tokens Like Cash: One Wallet per Feature

Last month I burned most of a free AI grant in about forty minutes. Not on a big feature. On a retry loop.

A summarizer kept returning an empty string. My client retried three times per request. The queue had 900 items. Do the math: roughly 2,700 calls for 900 items, and I was watching the wrong number the whole time.

The console showed one number — total tokens. It never told me which feature ate the grant. So I could not separate the runaway loop from real traffic. That was the actual bug. Not the retry. The missing accounting.

This post is the fix I now ship first: a per-feature token wallet that fails closed before the call, not after. It is about 90 lines of stdlib Python. No SDK, no database, no dashboard.

Where I run it: I test these wallets against MonkeyCode's free model access and its free server option. The grant I was quoted was 10M tokens, and quotas move — read the number in your own console, not in this post. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why a total-token counter is useless

A single global counter answers one question: how much is left? It cannot answer the question that matters at 2 a.m.: who spent it?

Ask yourself this. If your free grant halves tomorrow, which feature do you turn off? If you cannot answer in ten seconds, you do not have accounting. You have a thermometer taped to a wall.

A wallet fixes that by giving every feature its own cap. Calls and tokens, both. The feature name is the unit of blame.

Step 1: declare one wallet per feature

Start with a file you can review in a pull request.

{ "summarize": { "max_tokens": 400000, "max_calls": 800 }, "tag-suggest": { "max_tokens": 120000, "max_calls": 2000 }, "eval-suite": { "max_tokens": 250000, "max_calls": 300 } } 

Three rules I follow here.

1. eval-suite is its own wallet. Evaluation traffic must never hide inside product traffic.
2. max_calls is always set. Tokens alone let a cheap infinite loop run forever.
3. The numbers come from a measured dry run, not from optimism. Ten calls, then multiply.

Step 2: check before you spend, never after

Here is the whole tool. Save it as wallet.py.

#!/usr/bin/env python3 """wallet.py - per-feature token wallets that fail closed before spending.""" from __future__ import annotations import json import os import sys import time import uuid from dataclasses import dataclass from pathlib import Path LEDGER = Path(os.environ.get("WALLET_LEDGER", "wallet-ledger.jsonl")) BUDGETS = Path(os.environ.get("WALLET_BUDGETS", "wallets.json")) class BudgetExceeded(RuntimeError): """Raised before a call that would break a wallet.""" @dataclass(frozen=True) class Wallet: feature: str max_tokens: int max_calls: int def load_wallets(path: Path | None = None) -> dict[str, Wallet]: path = path or BUDGETS raw = json.loads(path.read_text()) return {k: Wallet(k, v["max_tokens"], v["max_calls"]) for k, v in raw.items()} def spend(feature: str) -> tuple[int, int]: tokens = calls = 0 if LEDGER.exists(): for line in LEDGER.read_text().splitlines(): row = json.loads(line) if row["feature"] == feature: tokens += row["total_tokens"] calls += 1 return tokens, calls def precheck(wallet: Wallet, estimate: int) -> None: used_tokens, used_calls = spend(wallet.feature) if used_calls + 1 > wallet.max_calls: raise BudgetExceeded( f"{wallet.feature}: call budget spent ({used_calls}/{wallet.max_calls})" ) if used_tokens + estimate > wallet.max_tokens: raise BudgetExceeded( f"{wallet.feature}: token budget would break " f"({used_tokens}+{estimate}>{wallet.max_tokens})" ) def charge(feature: str, estimate: int, call, **kw): precheck(load_wallets()[feature], estimate) resp = call(**kw) usage = resp["usage"] row = { "id": uuid.uuid4().hex[:8], "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "feature": feature, "model": kw.get("model", "unknown"), "input_tokens": usage["input_tokens"], "output_tokens": usage["output_tokens"], "total_tokens": usage["input_tokens"] + usage["output_tokens"], } with LEDGER.open("a") as f: f.write(json.dumps(row) + "\n") return resp def report() -> int: wallets = load_wallets() worst = 0 print(f"{'feature':<14}{'tok':>9}{'cap':>9}{'calls':>7}{'cap':>7} status") for name, w in wallets.items(): t, c = spend(name) pct = max(t / w.max_tokens, c / w.max_calls) status = "STOP" if pct >= 1 else ("WARN" if pct >= 0.8 else "ok") worst = max(worst, 2 if status == "STOP" else 1 if status == "WARN" else 0) print(f"{name:<14}{t:>9}{w.max_tokens:>9}{c:>7}{w.max_calls:>7} {status}") return worst if __name__ == "__main__": sys.exit(report()) 

The important line is precheck(...) running before call(...). A post-hoc budget check is a postmortem, not a gate.

report() returns exit code 0, 1, or 2. That means CI can read it.

python wallet.py; echo "exit=$?" 

Step 3: write the ledger, never rewrite it

The ledger is append-only JSONL. One line per call. You can grep, jq, or load it in pandas without asking anyone for permission.

jq -r 'select(.feature=="eval-suite") | .total_tokens' wallet-ledger.jsonl \ | awk '{s+=$1} END {print "eval tokens:", s}' 

Why JSONL and not SQLite? Because a text file survives git diff, a crashed process, and my own bad decisions.

Step 4: prove it fails closed with a fake client

An accounting layer you have never seen trip is a decoration. Here is the fixture I keep, as test_wallet.py.

import json import pytest import wallet def test_retry_loop_is_stopped_by_call_budget(tmp_path, monkeypatch): budgets = tmp_path / "wallets.json" budgets.write_text(json.dumps({"summarize": {"max_tokens": 1000, "max_calls": 3}})) monkeypatch.setattr(wallet, "BUDGETS", budgets) monkeypatch.setattr(wallet, "LEDGER", tmp_path / "ledger.jsonl") def fake_client(**kw): return {"usage": {"input_tokens": 120, "output_tokens": 80}} attempts = 0 with pytest.raises(wallet.BudgetExceeded) as err: for _ in range(5): # the retry loop from my bad night  wallet.charge("summarize", 200, fake_client, model="free-tier") attempts += 1 assert attempts == 3 assert "call budget spent" in str(err.value) rows = (tmp_path / "ledger.jsonl").read_text().splitlines() assert len(rows) == 3 

My test log after the first real run:

$ pytest -q test_wallet.py . [100%] 1 passed in 0.09s 

And the failure it reproduces is the exact shape of my incident: a loop that wants five calls, a wallet that allows three, and a ledger that stops at three rows.

The bug I hit while writing this

My first version had def load_wallets(path: Path = BUDGETS). The test ignored monkeypatch completely and wrote to my real ledger.

Default arguments bind once, at import. Change the signature to path: Path | None = None and resolve inside the body. That single fix turned the fixture from a lie into a test.

The related one: spend() reads the whole ledger on every call. Fine at hundreds of rows, painful at hundreds of thousands. Rotate the file monthly, or cache a running total. Do not pretend it scales forever.

Free server or paid path: a decision table

A free server option changes where jobs run, not what your users tolerate. I use this table before moving anything.

Two hard rules I follow.

1. A free server job must be idempotent and requeueable. If a restart loses work, it is not eligible.
2. The wallet caps apply on every tier. Free is cheaper, not unlimited.

If you want to try the wallet against free model access, it does not care which provider you point it at. MonkeyCode's free model access and free server option are just two places I have run it; the script is the reusable part.

Limitations, and who should skip this

• The ledger only counts calls that go through charge(). One direct SDK call in a notebook and your numbers drift.
• Streaming responses often hide usage until the end. Measure after the stream closes, not during.
• Cached or short-circuited responses can look free. Count the call anyway, or your call budget lies.
• This is not a billing system. Do not ship invoices off a JSONL file.
• Skip it if you already have per-feature metering in production. Skip it if one feature is your whole product.

What I would add next

The gap I have not closed is retry accounting. A feature that retries three times burns three wallets' worth of calls, but my cap only sees the successful ones.

So here is my question for you. What is your retry multiplier per feature? I want to add it to wallets.json as a third cap and let precheck do the division before the first attempt, not after the third.