Free Tokens Have a Break-Even Failure Rate

Free capacity is cheaper right up until it isn't, and the crossover is arithmetic rather than taste. You can find it in about ten minutes using two numbers you probably already log: attempts per success, and wall-clock per attempt. Below that failure rate you should route the work to the free lane. Above it, you are quietly paying in retries and queue slots that never appear on an invoice.

The trap is that tokens are the only line item with a visible price tag. A retry that costs zero tokens still costs a queue slot, a deadline, and possibly someone's evening. So the honest unit of comparison is cost per successful task, not cost per call.

Attempts are a multiplier, not a rounding error

A lane that fails 20% of the time needs 1.25 attempts per success on average. That single sentence is most of the model. Expected attempts is the reciprocal of the success rate, with one wrinkle: you usually cap retries, and capped retries mean some tasks never succeed even though you paid for every attempt.

def expected_attempts(fail_rate, max_attempts=4): # E[attempts] = sum over k of P(reaching attempt k)  return sum(fail_rate ** (k - 1) for k in range(1, max_attempts + 1)) def success_rate(fail_rate, max_attempts=4): # probability at least one attempt validates  return 1 - fail_rate ** max_attempts 

With fail_rate=0.2 and four attempts you get roughly 1.25 attempts and a 99.84% completion rate. Drop the attempt cap to two and the completion rate falls to 96% while the attempt count barely moves. That is the first honest trade: retry budget buys reliability, not savings.

Put a price on the retry

Now widen the ledger. Each attempt costs tokens, latency, and backoff time, and only completed tasks count as output. I value a minute of queue latency explicitly, because otherwise the free lane can never lose and the whole exercise becomes theatre.

from dataclasses import dataclass @dataclass class Lane: name: str tokens_per_attempt: int price_per_1k: float # 0.0 for a free model lane  latency_s: float fail_rate: float backoff_s: float = 0.0 max_attempts: int = 4 def lane_cost(lane, minutes_value=0.0): attempts = expected_attempts(lane.fail_rate, lane.max_attempts) ok = success_rate(lane.fail_rate, lane.max_attempts) token_cost = attempts * lane.tokens_per_attempt / 1000 * lane.price_per_1k wall_s = attempts * lane.latency_s + max(0.0, attempts - 1) * lane.backoff_s time_cost = wall_s / 60 * minutes_value return { "attempts": round(attempts, 3), "success": round(ok, 4), "wall_s": round(wall_s, 1), "cost_per_success": round((token_cost + time_cost) / ok, 4), } def breakeven_fail_rate(free, paid, minutes_value=0.0): target = lane_cost(paid, minutes_value)["cost_per_success"] lo, hi = 0.0, 0.95 for _ in range(60): mid = (lo + hi) / 2 free.fail_rate = mid if lane_cost(free, minutes_value)["cost_per_success"] > target: hi = mid else: lo = mid return round(lo, 3) 

Run it with synthetic numbers — these are illustrative, not measurements of any particular provider:

free = Lane("free", tokens_per_attempt=1200, price_per_1k=0.0, latency_s=9.0, fail_rate=0.10, backoff_s=3.0) paid = Lane("paid", tokens_per_attempt=900, price_per_1k=0.05, latency_s=6.0, fail_rate=0.02, backoff_s=3.0) print(lane_cost(free, minutes_value=0.30)) # ~0.0517 per success print(lane_cost(paid, minutes_value=0.30)) # ~0.0765 per success print(breakeven_fail_rate(free, paid, 0.30)) # ~0.34 

Read that last line carefully. At a queue-minute value of $0.30, the free lane stays cheaper until its failure rate passes roughly 34%. That is a surprisingly high ceiling, and it is the reason "free is slower but fine" is often correct. It is also why the verdict flips the moment your latency is expensive.

Set minutes_value=0 and the free lane becomes unbeatable at any failure rate. That is not a finding, it is a bug in your accounting.

Measure the failure rate instead of guessing it

One number in that script is unknown, and it is the one people fill in from memory. Replace memory with a replay.

# replay.sh — 50 real prompts, deterministic validator, append to CSV printf 'task,lane,attempts,validated,wall_s\n' > replay.csv for i in $(seq 1 50); do start=$(date +%s) for attempt in 1 2 3 4; do python run_task.py --prompt "prompts/$i.json" --out "out/$i.json" if python validate.py --schema schema.json --file "out/$i.json"; then break fi done end=$(date +%s) python validate.py --schema schema.json --file "out/$i.json" \ && ok=1 || ok=0 printf '%s,free,%s,%s,%s\n' "$i" "$attempt" "$ok" "$((end-start))" >> replay.csv done 

The validator is the load-bearing part. If "looks reasonable" is your acceptance test, you are measuring your own optimism. Schema checks, unit assertions, and exact-match fields are cheap and unforgiving — exactly what you want here. Fifty tasks is enough to see a 30% failure rate; it is not enough to distinguish 3% from 5%, so do not build a procurement decision on a coin flip.

The part people skip: validator cost

A free lane with a high failure rate is only cheap if checking the output is cheap. When validation means a human reading a diff, each rejected attempt costs minutes, and the ledger swings hard. Fold that in as validation_cost * attempts and rerun the break-even search. For tasks where a wrong answer is expensive and detection is slow, the free lane fails on economics long before it fails on quality.

Where this model breaks

Three limits worth stating plainly. Free capacity is shared, so your measured failure rate on a Tuesday morning may not hold during a peak; sample across days, not minutes. Retry storms can turn a 10% failure rate into a self-inflicted rate limit, which looks exactly like a provider outage. And any lane whose availability you do not control needs a fallback path you have actually exercised, not one you documented.

If your SLO is measured in seconds, or your data cannot leave your account, stop here. This workflow is for people whose cost is dominated by volume and whose correctness is machine-checkable. It is not a case for moving regulated workloads anywhere.

One practical place to run the experiment: MonkeyCode offers free model access and a free server option, which is enough to host the replay harness and a small queue worker before you commit a budget line. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not benchmarked it here, and the numbers above are synthetic — the point is the method, and you should measure your own lanes rather than trust anyone's table.

If you want a concrete next step, take the ten lines of lane_cost, plug in your last week of logs, and find the failure rate at which your "free" lane stops being free. Most teams guess it at 5%. The arithmetic usually disagrees.