At turn 223, a Gemini 3.8 Flash chat started making mistakes. By turn 287 it was wrong. The same model, on the same job, with the same 1,500 steps, made none.

The difference was not the model. It was what the model was handed at each step.

The cost nobody prices in

Most LLM agents work like a chat: every step, the whole history goes back into the prompt. Step k sends about k units of history, so a job of N steps sends about N²/2 units in total. Double the job and the cost of the history quadruples. Faster chips make each token cheaper by a constant factor. They don't change that curve.

We measured it. Over 1,500 steps the kept chat's mean prompt was 476,835 tokens, peaking at 949,591. The alternative held the exact state of the job on the host and sent a fresh prompt of 2,824 tokens at every step. That is 169 times smaller on average.

What that did

  • Exactness: 1,500 of 1,500 steps exact, against a chat that slipped from turn 223.
  • Latency: past 500k tokens, a call took 10.66 s against 3.07 s, and first token 9.13 s against 2.27 s.
  • Throughput: 1,280 exact steps per hour against 488 (2.6x).
  • Cost per correct step at typed prices: $0.0012 against $0.0052 (4.5x), and 9.3x on a second seed.

The error problem is worse than the cost problem

In a separate 4,000-request run, 393 of 400 replies were right, 98%. Applied exactly as the model wrote them, only 85 of 400 steps ended in the correct state, 21%. A small error rate per reply becomes a large error rate per job, because each slip is carried into everything after it.

In a fuzzy-input test, one slip at turn 191 stayed in the state for 140 turns, while 99.5% of replies were individually fine.

On a small open model (Qwen), 12 of 12 episodes with the history kept in the chat ended corrupted. With host-held state, 0 of 360 did.

How it works

The architecture, AXI, is by Rajnish Choubey. The exact state lives on the host. At each step the model gets a short prompt built from it, and proposes the next move. A deterministic gate checks the proposal in about 2 microseconds (measured in Python), a rule-based repair fixes a slip, and the host updates the state.

There is no training and no change to the model, and it works from the first step, on short jobs and long ones. The cost grows linearly with steps, not with their square.

How we tested

Every test page was locked with a SHA-256 hash before the run, with probabilities attached to each prediction. In the latest test, 12 of 13 predictions held and one missed; we report it as wrong. A test in the repo recomputes every figure from the saved results.

Limits

These are specific jobs on Gemini 3.8 Flash and Qwen. The repair applies where a clean rule exists, and how widely that holds across other jobs is still being measured. On a TPU v6e-1, memory headroom allowed 927 sessions with host-held state against 5 with a kept chat, but that is for a 0.5B cache shape and is not end-to-end throughput.

Tests built and run by Shalendra Gaur, published with Rajnish Choubey's permission.

Try it yourself

ai #llm #machinelearning #python #gemini #aiagents #tokencost #kvcache #inference #devops