Most multi-model comparisons never tell you what they cost to run. Ours did, because in MultiO every message carries a receipt — so the study's own bill is part of its method, and we published it.

The programme: the same 24 questions, unchanged, sent to five mid-tier models — GPT, Claude, Gemini, Grok and Llama — then a Council read per question to flag where the answers materially split, plus a rerun control (two questions, three fresh samples each through one model). The total metered cost was $0.87 — 866,245 micro-USD on the study account's own receipts.

The breakdown is the interesting part:

  • Panel answers: 177,440 micro — the five models answering 24 questions each was the smaller half of the bill.
  • Council runs: 594,750 micro — judging where the models split cost more than three times the answers themselves. Reading for disagreement is the expensive step, not generating the answers.
  • Rerun control: 15,896 micro — cheap, and it earned its keep: the model never changed its verdict across fresh samples, which is how we know the splits live between models, not inside one model's sampling.
  • First attempt: 78,159 micro — a first run whose data was lost to a service restart. It is in the total anyway, disclosed in the study's Limitations, because a cost you hide is a cost you are asking readers to trust you on.

Two honesty details sit inside those numbers. Ten of the panel calls (the first two questions in the completed run) were exact-cache replays of the lost first attempt's answers — same models, same questions, generated minutes earlier — and were charged 0; the Council judging on them ran fresh. And the judging itself needed a metered extraction follow-up in 17 of 24 runs when the first pass missed the structured format. Neither fact changes the findings — 52 material divergences, models materially agreeing on only 8 of 24 questions — but both are in the published method because the receipt is the point: if a tool meters usage, the study run on it should be auditable down to the micro.

That is also the practical takeaway for anyone cross-checking models on a budget: the answers are not the expensive part. Asking five models a question is cheap; the cost is in the structured read that tells you where they split — and that read is exactly the part a single-model workflow skips entirely, at $0 and at the price of never seeing the disagreement.

Full study, method, cost breakdown and all caveats (including that one judge was also a panel member, the rotating judging sets, and a memory-bleed incident we found and disclosed): https://multio.ai/research/model-divergence

Disclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.

— @getmultio / hello@multio.ai