The green build that teaches you nothing

Your job log says attempt 1 failed, attempt 2 ok. The pipeline is green. Nobody reads that line again.

That line is a bug report you threw away. A retried success means your system spent capacity twice and told you once.

So here is the question I keep asking teams: if the retry is the thing saving you, what exactly is the thing breaking?

This post is a myth-busting FAQ about retries against free model endpoints and restart-prone free servers. It ends with a ledger you can run, not a slogan.

Myth 1: Every error deserves a retry

Most retry helpers retry anything that is not a 2xx. That is a coin flip, not a policy.

Some failures get worse when repeated. Auth failures, schema rejections, and oversized payloads do not heal in 800 ms.

Others are worth one more try. Timeouts, connection resets, and 5xx responses usually are.

Here is the corrected mental model. Classify first, sleep second, retry third.

  • Terminal: fix the request or the credential. Retrying burns your budget.
  • Retryable: transient by nature. Bound it by time, not by hope.
  • Unknown: no status, no body, no clue. Treat as retryable once, then stop.

Myth 2: Retries are free

They are not. Every retry consumes the same shared capacity your first attempt used.

Do the arithmetic on a 30% per-attempt failure rate with three attempts. You still send about 1.4 requests per success.

Now push the failure rate to 60%. Three attempts send roughly 1.9 requests per success, and your tail gets worse.

Wait. That is the trap. Retries amplify load exactly when the endpoint is already unhealthy.

If the provider is throttling you, retries are not resilience. They are a load generator you wrote yourself.

Myth 3: Backoff is a deadline

Exponential backoff bounds the wait between attempts. It says nothing about total time.

Five attempts with jittered backoff can run for a minute. Is that acceptable for your request? Often nobody knows.

Use a wall-clock deadline instead. Pass it down, and refuse to start an attempt you cannot finish.

budget = 8s attempt 1 at t=0.0s -> timeout 6.0s attempt 2 at t=0.4s -> timeout 5.0s attempt 3 at t=1.6s -> only 2.4s left, skip or shrink timeout stop at t=8.0s, report PARTIAL

One more thing. A restart-prone free server can kill in-flight requests. Long backoff makes you more likely to hit that window, not less.

Myth 4: A retry is the same request

At the protocol level, maybe. At the product level, no.

The first attempt may have produced tokens, side effects, or a bill you cannot see. The retry produces a second result.

So ask yourself: can your caller tell the difference between one answer and two?

If the answer is no, you need an idempotency key on the request and a record of what came back. That record is the whole point of the next section.

Myth 5: Retried success means healthy

This is the expensive one. Teams watch error rate and miss retried-success rate.

A service that answers 97% of calls but retries 20% of them is not 97% healthy. It is running at 1.2x load for free and hiding the reason.

Track three numbers instead of one:

  1. Terminal failure rate — real user-visible breakage.
  2. Retried-success ratio — successful calls that needed more than one attempt.
  3. Attempts per success — your true cost multiplier.

The artifact: a retry ledger you can run

Here is the harness. It is stdlib-only Python and I have not run it against your endpoint, so treat it as a starting point you execute yourself.

MonkeyCode's free model access and free server option are one place you could point this at, if you want a cheap endpoint to test against. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not restating quotas, model names, or specs here. Those change, and I have not verified them. Check the provider's own docs.

# retry_ledger.py - stdlib only import json, random, time, urllib.request, urllib.error RETRYABLE = {408, 425, 429, 500, 502, 503, 504} TERMINAL = {400, 401, 403, 404, 413, 422} def classify(status, exc): if exc is not None and status is None: return "retry" if isinstance(exc, (TimeoutError, ConnectionError)) else "unknown" if status in RETRYABLE: return "retry" if status in TERMINAL: return "terminal" return "unknown" def call(url, payload, timeout): req = urllib.request.Request( url, data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"}, method="POST") try: with urllib.request.urlopen(req, timeout=timeout) as r: return r.status, r.read().decode(), None except urllib.error.HTTPError as e: return e.code, e.read().decode(errors="replace"), None except Exception as e: return None, "", e def run(url, payload, budget_s=8.0, max_attempts=3, key=None, ledger="ledger.jsonl"): start = time.monotonic() deadline = start + budget_s attempts = 0 last = None while attempts < max_attempts: remaining = deadline - time.monotonic() if remaining <= 0.2: break attempts += 1 per_try = max(0.2, remaining / (max_attempts - attempts + 1)) status, body, exc = call(url, payload, per_try) verdict = classify(status, exc) last = {"status": status, "body_len": len(body or ""), "verdict": verdict} if verdict == "retry": base = min(2.0, 0.25 * (2 ** (attempts - 1))) time.sleep(base * (0.5 + random.random())) continue break rec = { "key": key, "attempts": attempts, "elapsed_s": round(time.monotonic() - start, 3), "last": last, "ok": bool(last and last["verdict"] != "terminal" and last["status"] and last["status"] < 400), } with open(ledger, "a") as f: f.write(json.dumps(rec) + "\n") return rec

Two details matter more than the code. The per-attempt timeout shrinks as the deadline approaches. And the ledger is append-only JSONL, so a crash costs you one line, not the file.

The gate: make CI read the ledger

A ledger nobody reads is a log file with ambition. Add a gate.

# gate.py - fail when retries are doing the heavy lifting import json, sys rows = [json.loads(l) for l in open("ledger.jsonl") if l.strip()] ok = [r for r in rows if r["ok"]] if len(ok) < 50: print(f"skip: only {len(ok)} successes"); sys.exit(0) retried = sum(1 for r in ok if r["attempts"] > 1) ratio = retried / len(ok) per_success = sum(r["attempts"] for r in ok) / len(ok) print(f"retried-success={ratio:.2%} attempts/success={per_success:.2f}") sys.exit(1 if ratio > 0.05 or per_success > 1.3 else 0)

Tune the thresholds to your traffic. The point is that the number exists, gets reviewed, and can block a merge.

Limitations you should hold onto

  • The harness assumes failures are independent. Shared quota exhaustion breaks that assumption badly.
  • Append-only JSONL grows fast. Rotate it or ship it to real storage.
  • Ephemeral disks lose the ledger on restart. Mount a volume or accept sampling gaps.
  • Clock skew and sleep jitter make elapsed_s approximate, not exact.
  • A 429 tells you almost nothing about how long to wait unless the response says so.

Who should not use this

Skip this approach if you need guaranteed capacity, because retries cannot create it. Skip it for strict latency SLOs, where a deadline of eight seconds is already a violation.

Also skip it for batch work that can be re-queued later. In that case a durable queue beats an in-process retry loop every time.

What I would check first

Pull your last thousand successful calls. How many needed two attempts? If you cannot answer, you do not have a resilience number yet.

Run the ledger for a day. Read the gate output. Then decide whether your retries are protecting users or protecting your dashboard.

If you want a cheap endpoint to point the harness at, MonkeyCode's free model access and free server option can host the experiment without a card on file.

Which one of your green builds is secretly a retry loop wearing a success badge?