Как построить LLM-роутер и проверить экономию без потери качества — a router with budgets, escalation and a cost ledger | Agent Lab Journal
Agent Lab Journal
Guides Glossary Practice · Cost and quality
Как построить LLM-роутер и проверить экономию без потери качества
How to build an LLM router with budget limits, verifier-gated escalation and a cost ledger, and how to prove on a frozen query set that it saves money without quietly getting answers wrong.
Level: advanced ·
Reading time: 45 minutes ·
Updated: 7 October 2026
Most teams start by sending every request to their strongest model. It works, and it costs money on every trivial "extract the invoice number" call. The usual fix is to let a cheap model answer first and stop when it says it is confident. That fix has a known weakness: the model that got the answer wrong is the same one grading itself. When it is confidently wrong, the request exits early and nothing ever checks it. This guide builds something different. A cheap-first LLM router may only stop early when an external check passes. It has hard per-request and per-day budgets, and it writes every model call to a ledger. Then it is tested against an "always use the large model" baseline on a frozen set of queries, with a decision rule written down before the run.
What this article contains and what it does not. It contains complete, dependency-free Python code, a configuration format, an evaluation protocol and an analysis script. It does not contain measured savings, accuracy figures or model prices. We have not run this protocol on your traffic or your models, and any number we printed here would be invented. The results section is a template that your own run fills in. The dry-run mode uses a fake model adapter that only checks the plumbing. Its cost and quality output means nothing.
The case: mixed traffic, one expensive model
Take an internal operations assistant. The traffic below is a hypothetical but typical mix, used here as a worked example and not as data:
Structured extraction: "pull the invoice number, date and total out of this email" and return JSON.
Classification: route a ticket into one of eight queues.
Grounded Q&A: answer a question from a supplied policy excerpt and quote the supporting text.
Small code tasks: write a helper function that has to pass given tests.
Open-ended writing: draft a reply to an unhappy customer.
All of this currently goes to one large model. The goal is to send the easy share to a small model and keep quality where it is. "Keep quality" needs an operational definition, or the project ends with a dashboard showing lower spend and nobody knowing what got worse. In this guide the definition is a non-inferiority test. Before the run you pick a margin, for example "accuracy may drop by at most 2 percentage points". The router is accepted only if the lower bound of the confidence interval for (router − baseline) accuracy sits above minus that margin and the cost reduction clears a target.
Why early exit on self-reported confidence is the wrong gate
A model cascade that asks the small model "how confident are you, 0–100?" and stops above some threshold has three structural problems:
The signal is not independent of the error. If the model misread the task, its confidence is formed from that same misreading. Verbal confidence is often poorly calibrated, and how poorly depends on the model, the prompt and the task, so a threshold that works on one task type need not transfer to another.
The errors it lets through are the expensive kind. Low-confidence wrong answers get escalated and fixed. High-confidence wrong answers exit, and those are exactly the ones a human downstream trusts.
It locks itself in. If you later tune the threshold on production logs that contain only the exited answers, you never see the wrong ones that left early. The system looks better and better on its own data.
We do not assert how large this effect is for your models. The harness below measures it. It still asks the small model for a confidence line and logs it, but never uses it to exit. The analysis then computes what a confidence-threshold policy would have done on the same items, scored against gold answers. If the confident-and-wrong count is zero on your data, you have learned something. If it is not, you have the evidence to drop that design.
Architecture
The router has five parts. Each one is small enough to read in full.
Pre-routing by rules. Cheap, deterministic features pick the starting tier. Task types with no machine-checkable output (open-ended writing) start on the large model. Very long contexts start large. Everything else starts small.
Verifiers. These are reference-free checks per task type. JSON must parse and contain the required keys with the right types. A label must belong to the allowed set. Every quote must appear verbatim in the supplied context. Code must pass the visible tests. A verifier never sees the gold answer.
Escalation. If the verifier fails, the request moves up one tier, at most once per tier. If the top tier also fails, the answer is returned with an explicit status rather than retried in a loop.
Budgets. Before each call the router estimates the worst-case cost (input tokens plus the full output limit). It refuses the call if that would break the per-request or per-day limit.
Ledger. One JSONL line per model call: request id, tier, model, tokens, cost, latency, verifier verdict, confidence and the reason for escalation. Budgets are computed from the ledger, so the spend you enforce and the spend you report come from one source.
The baseline is the same code with the tier list reduced to the large model. Prompts, parsing, verifiers and accounting are identical, so the comparison isolates the routing policy.
Project layout
llm-router/ config.json # tiers, model ids, price table, budgets, routing rules protocol.json # pre-registered margin, cost target, strata (frozen before the run) eval_set.jsonl # frozen queries with gold answers verifiers.py # reference-free checks used by the router router.py # pre-routing, escalation, budgets, ledger grade.py # gold-based grading used only by the evaluation run_eval.py # runs baseline and router on the same items analyze.py # paired statistics and the decision rule blind_pairs.py # blinded A/B export for open-ended items noise.py # baseline-vs-baseline noise floor Everything uses only the Python standard library. Python 3.10 or newer is assumed.
Step 1. Freeze the evaluation set before writing the router
The evaluation set decides what "without loss of quality" means, so build it first and do not change it once you start looking at router output.
Composition
Sample from real traffic, after removing personal data and secrets. If you cannot use real traffic, write the items by hand, say so in the report, and do not generalise the result to production.
Stratify by task type and by a difficulty label assigned before any model sees the items, for example from how long a human took or whether a second human disagreed. Report results per stratum. A router can look fine on average while failing the hard stratum completely.
Size it honestly. With 100 scored items, one item is one percentage point of accuracy. If your margin is 2 points, a 100-item set cannot separate "no change" from "slightly worse" with useful confidence. Use several hundred scored items, and expect the interval to tell you when the set is too small.
Gold answers are not verifiers
This is the most important design rule in the protocol. The router's verifiers must be reference-free, because in production there is no gold answer. The grader uses gold answers that the router never sees. For code tasks, that means two separate test sets: visible_tests, which the router's verifier runs, and hidden_tests, which only the grader runs. If the grader and the verifier are the same check, the router will look perfect by construction.
Item format
The following lines are format examples, not real data:
{"id": "jx-001", "task_type": "json_extract", "difficulty": "easy", "input": "Extract invoice_no and total from: Invoice A-1043, amount due 1250.00 EUR.", "schema": {"required": ["invoice_no", "total"], "types": {"invoice_no": "str", "total": "float"}}, "gold": {"invoice_no": "A-1043", "total": 1250.0}} {"id": "cl-014", "task_type": "classification", "difficulty": "hard", "input": "Ticket: 'Card charged twice after plan downgrade.' Answer with one label only.", "labels": ["billing", "access", "bug", "feature", "legal", "shipping", "account", "other"], "gold": "billing"} {"id": "qa-007", "task_type": "grounded_qa", "difficulty": "medium", "context": "Refunds are issued within 14 days of approval. Approval requires a receipt.", "input": "How long do refunds take? Cite the context with lines starting 'QUOTE: '.", "gold": {"must_include": ["14 days"]}} {"id": "cd-003", "task_type": "code", "difficulty": "medium", "input": "Write a Python function slugify(s) that lowercases and replaces runs of non-alphanumerics with '-'. Return only code.", "gold": {"visible_tests": "assert slugify('A b') == 'a-b'\n", "hidden_tests": "assert slugify(' Hi!!there ') == 'hi-there'\n", "reference_solution": "import re\ndef slugify(s):\n return re.sub(r'[^a-z0-9]+', '-', s.lower()).strip('-')\n"}} {"id": "oe-021", "task_type": "open_ended", "difficulty": "hard", "input": "Draft a short, polite reply to a customer whose delivery is 5 days late."} Note that the visible tests for code live under gold but are passed to the verifier explicitly. The hidden tests never reach the router.
Pre-register the decision
{ "margin_accuracy": 0.02, "target_cost_ratio": 0.6, "critical_strata": ["hard"], "bootstrap_samples": 5000, "seed": 1, "notes": "Margin and target chosen before the run; values are this example's, not recommendations." } Then freeze both files:
sha256sum eval_set.jsonl protocol.json > frozen.sha256 git add eval_set.jsonl protocol.json frozen.sha256 git commit -m "Freeze eval set and protocol before router run" Run sha256sum -c frozen.sha256 before the final analysis. If it fails, you changed the test after seeing results, and you should say so in the report.
Step 2. Configuration and the price table
{ "price_table_date": "YYYY-MM-DD", "price_table_source": "provider pricing page URL, copied by hand on that date", "models": { "small": {"id": "SMALL_MODEL_ID", "price_in_per_mtok": null, "price_out_per_mtok": null, "max_output_tokens": 800}, "large": {"id": "LARGE_MODEL_ID", "price_in_per_mtok": null, "price_out_per_mtok": null, "max_output_tokens": 1500} }, "tiers": ["small", "large"], "budgets": {"per_request_usd": 0.10, "per_day_usd": 25.0}, "routing": { "large_first_task_types": ["open_ended"], "long_context_chars": 24000 } } Prices are deliberately null. The router refuses to run until you fill them in from your provider's current price list and record the date. The budget values are placeholders for this example and should not be read as recommendations. Set the per-request budget high enough that the baseline is never refused. Otherwise the baseline loses quality for budget reasons and the comparison becomes unfair. If your provider bills cached input, batch calls or reasoning tokens at different rates, add those fields and extend price(). A single in/out rate is a simplification.
Step 3. Verifiers
verifiers.py holds one function per task type. Each returns (ok, detail), and the detail string goes to the ledger as the reason for escalation.
import json, os, re, subprocess, sys, tempfile def run_python_tests(code, tests, timeout=10): """Runs model-written code. Do this only inside a container or sandbox with no network and no credentials in the environment.""" with tempfile.TemporaryDirectory() as d: path = os.path.join(d, "t.py") with open(path, "w", encoding="utf-8") as f: f.write(code + "\n\n" + tests) try: p = subprocess.run([sys.executable, path], capture_output=True, text=True, timeout=timeout, env={"PATH": os.environ.get("PATH", "")}) except subprocess.TimeoutExpired: return False, "timeout" return p.returncode == 0, "ok" if p.returncode == 0 else "tests_failed" def v_json_extract(req, answer): try: obj = json.loads(answer) except json.JSONDecodeError as e: return False, f"json_parse:{e.msg}" if not isinstance(obj, dict): return False, "not_object" missing = [k for k in req["schema"]["required"] if k not in obj] if missing: return False, "missing:" + ",".join(missing) for k, typ in req["schema"].get("types", {}).items(): if k in obj and type(obj[k]).__name__ != typ: return False, f"type:{k}" return True, "ok" def v_classification(req, answer): ok = answer.strip() in req["labels"] return ok, "ok" if ok else "label_not_allowed" def v_grounded_qa(req, answer): quotes = [l[len("QUOTE: "):].strip() for l in answer.splitlines() if l.startswith("QUOTE: ")] if not quotes: return False, "no_quote" bad = [q for q in quotes if len(q) < 8 or q not in req.get("context", "")] return (not bad), "ok" if not bad else "quote_not_in_context" def v_code(req, answer): return run_python_tests(answer, req["gold"]["visible_tests"]) VERIFIERS = { "json_extract": v_json_extract, "classification": v_classification, "grounded_qa": v_grounded_qa, "code": v_code, # "open_ended": no reference-free check -> routed large-first, never exits cheaply } Verifiers vary a lot in strength. The JSON and code checks catch many errors. The classification check catches almost none: any allowed label passes. That is intentional here, because it makes the weak spot visible in the per-task-type results. If classification shows harms, the fix is a stronger check, such as agreement between two cheap samples or a rule-based keyword cross-check, or sending that task type large-first. Lowering a threshold does not fix it.
One caveat about v_code: req["gold"]["visible_tests"] is read from the item for convenience. In production the visible tests come from the request itself. The router must never read hidden_tests.
Step 4. The router
import datetime, json, re, time, uuid from dataclasses import dataclass from verifiers import VERIFIERS @dataclass class Usage: input_tokens: int output_tokens: int def price(cfg, tier, usage): m = cfg["models"][tier] if m["price_in_per_mtok"] is None or m["price_out_per_mtok"] is None: raise ValueError(f"price table for tier {tier!r} is empty; fill it from the provider price list") return (usage.input_tokens * m["price_in_per_mtok"] + usage.output_tokens * m["price_out_per_mtok"]) / 1_000_000 def rough_tokens(text): # Deliberately pessimistic estimate for budget checks only. # Replace with your provider's token counter; actual cost always uses reported usage. return len(text) // 3 + 1 CONF_INSTR = "After the answer, add one final line exactly in the form CONFIDENCE: <integer 0-100>." def build_messages(req): system = req.get("system", "Answer the task. Follow the required output format exactly.") user = req["input"] if not req.get("context") else f"CONTEXT:\n{req['context']}\n\nTASK:\n{req['input']}" return [{"role": "system", "content": system + " " + CONF_INSTR}, {"role": "user", "content": user}] def parse_confidence(text): t = text.strip() m = re.search(r"CONFIDENCE:\s*(\d{1,3})\s*$", t) if not m: return t, None return t[:m.start()].strip(), min(int(m.group(1)), 100) class Ledger: def __init__(self, path): self.path = path def append(self, rec): with open(self.path, "a", encoding="utf-8") as f: f.write(json.dumps(rec, ensure_ascii=False) + "\n") def spent_on(self, day): total = 0.0 try: with open(self.path, encoding="utf-8") as f: for line in f: r = json.loads(line) if r["day"] == day: total += r["cost_usd"] except FileNotFoundError: pass return total class Router: def __init__(self, cfg, adapter, ledger, clock=None): self.cfg, self.adapter, self.ledger = cfg, adapter, ledger self.clock = clock or (lambda: datetime.datetime.now(datetime.timezone.utc)) def pre_route(self, req): r = self.cfg["routing"] if r.get("force_tier"): return r["force_tier"], "forced" if req["task_type"] in r["large_first_task_typesSource：https://dev.to/_862f933aa9477a9d2d/kak-postroit-llm-routier-i-provierit-ekonomiiu-biez-potieri-kachiestva-3ljm Related posts
DeepSeek's Open-Source "Super-Agent" Codes Without Being Asked
All-in-one platform or a stack you assemble? Where do coaches and educators land?
I Wanted to Live on Coruscant. PAT Scared Me. Now My Alexa Lives on Mute.
Detect What You Can Name: PaliGemma 2 Object Detection with Transformers
When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation
