Four cost gates that decide your token bill (none of them is the price per token)

I spend a lot of time in comment threads arguing about what a token costs. After enough of those, a pattern showed up: most "cost" questions are not pricing questions. They are questions about a gate somewhere else in the pipeline.

Four gates, in the order they usually bite.

1. Caching has a break-even, and it is later than you think

A cached prefix is not free. Providers normally charge a premium to write the cache and a discount to read it. So the question is not "does caching help", it is "how many times will this exact prefix be read before it dies".

If the write premium is 25% and the read discount is 90%, a prefix read once is a loss. Read twice, roughly break-even. From the third read on it is free money. That is the whole model, and it means the hit rate is the only number that matters - below roughly 60% you are paying to write something nobody reads.

A real example I keep coming back to: a game loop where the 1-hour cache cost 3.13 cents a turn and running with no cache cost 2.94 cents. Caching was 6% more expensive than not caching. Not a bug - the prefixes were not being reused.

What to log, if you want to act on this: prefix id, hit/miss, and tokens read from cache, per request. Three fields.

2. Where you put the schema decides your hit rate

Structured output is the usual reason caching quietly stops working. If the JSON schema is injected after the variable part - tool definitions appended last, schema string pinned to the tail - then the shared prefix ends before the schema, and every request forks at that point. Nothing after it can be shared.

Move the schema and the static instructions to the very front, and keep per-request content (session state, turn count, the user's message) strictly at the tail. On the structured-output loads I have logged, that reordering moved prefix hit rate by 40-70 points.

The "cache write premium" people complain about is the same effect seen from the invoice: you paid to write a prefix that was never read twice.

3. "It reproduced" is not a test result

If you compare two runs - two GPUs, two builds, two batch sizes - a diff-based comparison will call a truncated run "identical" when one output is a strict prefix of the other. Two outputs that end early at the same word hash the same.

So keep two assertions separate: equality and completion.

• equality: hash the full output. Any length change fails immediately.
• completion: read the finish reason from the metadata. Not from the decoded text - most decoders strip the EOS token, so a clean finish has no visible marker to look for.

Why bother: on a 300-question suite, one truncation silently scored as "equal" is 0.33 points. A zero-failure run can only claim an upper bound of about 1% at 95% confidence. The aggregate will never show you that error. Only the gate record will.

4. Billed-but-empty requests are invisible by construction

Timeouts after the prompt was tokenized, policy blocks that fire late, retries that fail on the third attempt - all of them appear on the invoice and none of them appear in your token counts.

There is no counter for this in most stacks, so it hides inside the per-turn average and reads as model price. The fix is one integer per request: billed_but_failed. Put it next to the token count you already log.

Why "cheapest model" tables are mostly noise

I rebuild a price feed for 109 Chinese models. The distribution is the argument:

Only 14% of the catalogue prices per token. A leaderboard that sorts all 109 by a single number is sorting strings: one row answers "what does an image cost", another answers "what does a million tokens cost".

You can check that distribution yourself, no key:

curl -s https://aiapi-pro.com/china-ai-leaderboard.json \ | jq -r '[.categories[].models[].unit] | group_by(.) | map({unit: .[0], n: length}) | sort_by(-.n)' 

Where this stops

All of the above is about measuring cost, not about a model being cheap or good. Pinning the setup makes two runs comparable; it does not make either one correct. Normalising a billing unit makes two prices comparable; it does not tell you which one fits your workload.

The useful version of every one of these questions is per-workload: "what does 10,000 product images with one retry each cost" has an answer. "Which model is cheapest" does not.

I keep the normalised numbers and the per-unit price table here: Chinese AI model prices, per unit - 109 models, billing unit next to every price, and a generated_at on the feed so you can tell how stale it is.