An input price per million tokens is a useful filter, but it is not the cost of getting a correct result. If a cheaper route fails the task or requires retries, the bill for an accepted answer can be different from the price sheet.

We ran a deliberately small document-QA check on two model routes through one API on 11 November 2026. This is a method example, not a benchmark or a claim that either model is generally better.

Define the answer gate first

We wrote a synthetic five-line document and four questions before sending any request. The questions tested:

  1. Grounding: Can the model answer a fact actually stated in the document?
  2. Abstention: Does it say unknown when the document does not specify a region?
  3. Cache math: Does it reject the claim that an 80% input-cache hit necessarily cuts the entire bill by 80%?
  4. Rate arithmetic: Can it choose the lower token subtotal from rates provided inside the synthetic document?

We requested a JSON object with one short answer field, then compared that answer with a fixed expected string. Each route received each question once. No automatic retry was allowed.

Both Qwen3.8 Flash and GLM-5.3 Flash returned four HTTP 200 responses with parseable JSON and matched all four expected short answers. Eight calls and invented questions cannot establish production quality, repeatability, long-context behavior, uptime, or a latency distribution. Your own sanitized task examples are the next test.

Record attempts, not just successful answers

For every request, keep the route ID, prompt version, expected answer, observed answer, HTTP status, elapsed time, retry count, and the complete usage object. An attempt that fails still belongs in the cost of the workflow. In a decoded Chat Completions JSON response, if completion_tokens_details reports reasoning_tokens, treat it as a subset of completion_tokens; adding the two would double-count output.

usage = response.get("usage") or {} completion = int(usage.get("completion_tokens") or 0) details = usage.get("completion_tokens_details") or {} reasoning = int(details.get("reasoning_tokens") or 0) if reasoning > completion: raise ValueError("Inspect the provider's usage fields") billable_output_from_response = completion # Do not add reasoning again.

That variable records a response field, not an independently verified invoice. Keep cached input, uncached input, cache creation, tool fees, and any retry or batch condition in separate columns if the route exposes them. The OpenAI token-counting guide also explains why reported output can exceed visible text; check the exact provider's semantics before applying the same logic to another API.

Make the comparison only after reconciliation

For each route, divide the total bill for all attempts by the number of accepted tasks. Do that only after the acceptance rule represents the work you actually need. We have not independently reconciled the response cost fields with a final account ledger, so we are not presenting a measured dollar saving or a model ranking here. The final account charge and tariff version still need checking.

The exact route names and current conditions should be rechecked in the API's live model catalog before rerunning this example. The document, refund window and example rates in our test were synthetic inputs, not real customer records or company policy.