This article was created with the help of AI. The AXI architecture is by Rajnish Choubey. The tests were built and run by Shalendra Gaur.
An agent that keeps its whole chat has to hold that chat in memory for as long as the session is live. How many sessions fit on one chip is a memory question, so I measured it on a Google TPU chip.
The test
- Chip: one Google TPU v6e in Colab. It reported a memory limit of 33,546,042,880 bytes (about 33.5 GB). JAX 0.11.2. Run on 31 October 2026.
- What was put on the chip: real cache-shaped arrays (the shape of Qwen2.5-0.5B, bf16), session after session, until the chip reported out of memory. No model was run.
- Lengths: the prompt of one AXI step (2,824 tokens), and the lengths of a kept chat from a 457-turn test on Gemini 3.8 Flash (mean 476,835 tokens, largest 949,591), plus 50,000, 100,000 and 250,000.
- The plan and six yes/no lines were written and hash-locked before the run, with a probability on each.
Result
| Prompt kept per session | Tokens | Sessions held on a TPU v6e-1 | Sessions held on an NVIDIA T4 (earlier test) |
|---|---|---|---|
| AXI step | 2,824 | 927 | 430 |
| Kept chat at 50k | 50,000 | 54 | 23 |
| Kept chat at 100k | 100,000 | 27 | 12 |
| Kept chat at 250k | 250,000 | 10 | 4 |
| Kept chat, mean of the long run | 476,835 | 5 | 2 |
| Kept chat, largest of the long run | 949,591 | 2 | 1 |
- The ratio between the two ways is about 169 in memory (476,835 / 2,824). Memory grows in a straight line with tokens, so this is close to the token ratio by construction. What the test adds is the chip itself: its real memory limit and the real bytes a session takes.
- A kept chat at the long run's mean length takes the memory of about 169 AXI-sized sessions, and at its largest length about 336.
- Of the six lines, five read YES and one read NO. The NO: I guessed the real bytes would be within 2% of the plain count at every length. At the AXI length they were 4.5% over (36,277,760 against 34,701,312 bytes). The extra is about what rounding the token axis up to a tile size would give, but I did not confirm that cause.
What this does not show
- This is memory only. Nothing here shows speed, quality, throughput or price. Real servers evict, offload, share prefixes and page memory, and the count means sessions in flight at once.
- No model was run, the cache shape is a small model's, and there is one run per length.
- The TPU holds more sessions than the T4 mainly because it has more memory (33.5 GB against 15.5 GB), not because of any speed.
AXI is a layer on the host side by Rajnish Choubey: the model is unchanged and the claims are about the state the host holds. The test page is docs/TPU_SESSIONS_TEST.md in the repository: https://github.com/gaurshalendra2412/axi-runtime
White paper, "AXI: The Cure for the Cost Crisis in AI" (Rajnish Choubey and Shalendra Gaur): https://doi.org/10.5281/zenodo.23958737