Everyone is talking about LLM inference cost. The price per token keeps falling, yet bills keep rising, especially for AI agents. This article explains one cause that gets little attention, shows the arithmetic, and reports what happened when we removed it in a measured test on Gemini 3.8 Flash.

The short version

An agent that keeps its history in the prompt pays for the same tokens again at every step. The total grows with the square of the number of steps. Cheaper chips and cheaper tokens make each token cost less, but they don't change that curve. Holding the exact state of the job on the host and sending a fresh short prompt at every step makes the cost grow in a straight line instead.

Where the cost comes from

Most agents work like a chat. At every step the model receives the whole history again: the instructions, every earlier request, every earlier answer. Step k sends about k units of history. Add up N steps and you get about N²/2 units. Double the length of a job and the history cost quadruples.

This has a consequence people feel but rarely price: the more an agent is used, the worse the cost per task gets. Under a flat-price plan, the heaviest users cost the most. Under per-token pricing, the same job costs more than it did the week before simply because it ran longer.

What the cure has to change

There are three common ways to reduce the bill, and it helps to see what each one changes:

Cheaper hardware or cheaper tokens reduce the price of each token by a constant factor. The curve keeps its shape.
Prefix caching makes re-reading tokens cheaper by a fixed factor. Again, a constant, and the curve keeps its shape.
Summaries and compaction shrink the history, but a summary is a guess about what matters, and a lost detail stays lost.

To change the shape, the history itself has to stop growing.

The approach we tested

The architecture, AXI, is by Rajnish Choubey. The exact state of the job lives on the host. At every step:

The host builds a short prompt from the exact state.
The model proposes the next move.
A deterministic gate checks the proposal in about 2 microseconds (measured in Python).
A rule-based repair fixes a slip.
The host updates the state and the loop repeats.

There is no training and no change to the model. It works from the first step, on short jobs and long ones.

A worked example from the measurements

Over 1,500 steps on Gemini 3.8 Flash, the kept chat's mean prompt was 476,835 tokens (peak 949,591). With host-held state, every prompt was 2,824 tokens.

Total input over the job, from those means:

Kept chat: 476,835 × 1,500 ≈ 715 million tokens
Host-held state: 2,824 × 1,500 ≈ 4.2 million tokens

That is about 169 times fewer input tokens.

The cost figure is smaller than that token ratio, because it counts every call at typed prices per correct step: $0.0012 against $0.0052, 4.5 times lower, and 9.3 times lower on a second seed.

Latency, not just money

Past 500k tokens of context, a whole call took 10.66 s against 3.07 s (3.5x), and the first token arrived after 9.13 s against 2.27 s (4.0x). Exact steps per hour: 1,280 against 488 (2.6x). A smaller prompt is cheaper and also faster.

Accuracy is part of cost

Wrong steps cost money twice: once to make them and again to fix them. In the same 1,500-step run, the host-held approach was exact on 1,500 of 1,500 steps, while the kept chat started slipping at turn 223 and was wrong from turn 287.

In a separate 4,000-request run, 393 of 400 replies were right (98%), but only 85 of 400 steps ended in the correct state (21%) when the replies were applied as written. A small error rate per reply becomes a large error rate per job, because each slip is carried into everything after it.

Questions people ask

Doesn't prefix caching already solve this?
Caching cuts the price of re-reading, by a fixed factor. It doesn't stop the history from growing, doesn't remove the latency we measured past 500k tokens, and doesn't stop drift. A direct head-to-head comparison on the same job is the right next experiment, and I would welcome help running it.

Does it need fine-tuning or a special model?
No. The model is unchanged. We tested Gemini 3.8 Flash and small Qwen models. With Qwen, 12 of 12 kept-chat episodes ended corrupted and 0 of 360 did with host-held state.

Is this only for long jobs?
No. It works from the first step. The saving grows with the number of steps, but the exactness holds from the start.

How do I know the numbers are real?
Each test page was locked with a SHA-256 hash before the run, with probabilities on every prediction. In the latest test, 12 of 13 predictions held and the one that missed is reported as wrong. A test in the repo recomputes every figure from the saved results.

Compare it on your own workload

If you run agents at scale, the quickest way to see what this would save you is to try it on your own jobs. The repo has a Colab notebook that runs the test with your own Gemini key and a spend limit. If you try it, please open an issue with what you find, good or bad.