Comparing 6 LLM APIs on identical prompts: a 30x latency gap and 4 silent failures

This post was written with AI assistance. Every number below comes from the raw responses I kept for each request, and the script that scored them is the same one I used to compute the expected answers first.

Six models, three prompts, eighteen requests, one API key. I expected the interesting part to be accuracy. It was not. Five of the six models are interchangeable on "can it do the task", and what actually separates them is how long you wait, how many output tokens they burn, and whether the response comes back with content in it at all.

Tested on 12 November 2026: claude-sonnet-5-5, gpt-5.3-codex, gemini-3.1-pro, deepseek-v4-pro, doubao-seed-2-1-turbo-260628, and hy4 (Tencent Hunyuan).

TL;DR, if you are about to write a client against one of these:

• Accuracy is not where they differ. One model missed a calendar question by a day; the other five were perfect.
• Latency differs by roughly 30x on the same three questions, and output tokens by about the same factor. Both hit your bill and your timeout budget.
• Four integration problems will do more damage in week one than model choice: empty 200s, rate limits that look like flakes, no web access, stale knowledge.
• The fastest model was also the one that ignored an explicit word count. Cheap and obedient are not the same axis.

How the test was run

Reasoning. Three questions with checkable answers: how many integers from 1 to 1000 are divisible by 3 or 5 but not 15; what weekday 1 March 2027 falls on given that 12 November 2026 is a Thursday; the conditional probability that the first five of ten coin flips were tails given at least one head. I computed the expected answers with a script before running anything, so I was not grading on impression.

Writing. A 140–160 character promotional blurb for a robot vacuum, family audience. Three required selling points, four banned phrases, no subheadings, no emoji. The constraint is the test — anyone can write a blurb, not everyone holds a character count.

Code. Implement parse_duration(s) in Python, turning 1h30m and 90s into seconds, returning None on bad input, with at least six assertions. I ignored the assertions each model wrote and ran their code against 13 hidden cases of my own: repeated units, bare units, missing units, truncated input, negatives, out-of-order but legal input.

The four things that break, in the order I hit them

1. A 200 response with an empty body. Hunyuan and DeepSeek, given a 4,096 output-token cap, spent the entire budget on reasoning and returned empty content with finish_reason: "length". Nothing in the HTTP layer tells you anything went wrong. I had to raise the cap to 16,384 before an answer appeared. If your client checks response.status == 200 and moves on, you will ship a feature that returns nothing, and it will look like a model problem rather than a config problem.

The check that catches it belongs next to your status check: when finish_reason is length and the content is empty, retry with a larger output budget instead of parsing the response.

2. Rate limits that present as instability, not as errors. Gemini's 429s appeared only at the higher output cap. Drop the cap and they vanish; raise it and they come back. From your logs this looks like an intermittent failure with no pattern, which is the worst kind. Backoff with jitter is not optional on this path — and neither is a fallback model, because a retry storm against a 429 is just a slower outage.

3. None of these APIs search the web. I asked all six for the previous day's US market close. All six said they could not retrieve it, and none invented numbers, which I appreciated. Then I declared a web_search tool in the request:

The one 200 is the dangerous one. A client that trusts HTTP status would have taken that response as a successful search. If you need current information, you build the retrieval layer and put results in the prompt. There is no flag for it.

4. Current model names, stale knowledge. Asked for their training cutoffs: Codex and DeepSeek said June 2024, Hunyuan and Doubao said July 2024, Gemini said January 2025, Claude would only say "around 2025". A model released this year can be eighteen months behind on facts. Anything time-sensitive needs a retrieval step regardless of what the model card says.

Accuracy: one miss, and it was not the slow model

Five models answered all three questions correctly. Codex answered two.

The miss was the calendar question. Counting days from 12 November 2026 to 1 March 2027, Codex came up one day short, divided by seven, and landed on Sunday instead of Monday. The write-up was immaculate: numbered steps, clean formatting, a confident conclusion. Nothing in the output hints that you should double-check it. I only caught it because the answer was already in a script.

That is the failure mode that costs money in production: not a model that declines, but one that hands you the wrong weekday with a confident walkthrough.

Two models answer in about six seconds, two around forty, two take three minutes. For a chat window that is an annoyance. For a nightly batch of a few thousand questions it is queue length and timeout rate.

Note the token column: the slowest model spent 30x the output tokens of the fastest one on the same three questions, and it was all reasoning tokens you pay for and mostly do not read.

Writing: the fastest run was the one that failed

Five models delivered publishable copy: 145–150 characters, all three selling points, zero banned phrases. The tone differences are channel differences, not quality differences.

Codex came back at 117 characters. Selling points present, banned phrases absent, length requirement simply dropped — in 3.0 seconds, on 104 output tokens, the cheapest run in the entire test. Fast and obedient turn out to be separate axes. Validate its output mechanically, then let it run.

Cost on identical output length, at vendor list price:

A 27x spread for the same 150 characters. Almost none of it comes from the per-token price — it comes from how many tokens the model spends thinking about a short piece of copy. Cost models built on "price per million" alone are off by an order of magnitude on writing tasks.

Code: five models passed the same hidden tests, one never answered

Claude, Codex, Hunyuan, DeepSeek and Doubao all passed 13/13. Repeated units, missing numbers, negatives, mixed order — every one of them handled it. At this difficulty, "which model writes better code" is not a useful question. Delivery speed is:

A 29x gap on a function whose spec fits in four lines. Gemini never returned usable code: repeated 429s at a 4,096 cap, and a 200 at 2,048 with the function truncated mid-body. That is not proof it cannot write the function; it is proof that through this route and at that budget it did not reliably produce one, which is the thing you actually need to know before routing traffic to it.

One endpoint is what made this comparison possible

I did not sign up for six accounts to write this. All six models sat behind a single OpenAI-compatible endpoint with one key and one request format, so switching models was a field change rather than a new SDK, a new invoice, and a new auth flow. That matters for the test itself: everything except the model stayed fixed between calls, which is the only reason the numbers are comparable at all.

https://api.crazyrouter.com/v1 

Send the same body, change model, and the comparison stays honest. Current per-model rates are on the price list, and a run this size costs well under a dollar, so you can reproduce it before committing to anything. If you would rather not run six separate signups to get there, one key covers all six.

The checks I now run on every response

None of these are model-specific, and all of them came out of the four failures above.

1. Treat finish_reason as part of the result, not as metadata. length with empty content is a failure, not a success with a short answer.
2. Log usage on every call. Prompt and completion tokens separately. The 30x spread in this test was invisible in the latency numbers alone, and it is the line that predicts your invoice.
3. Back off on 429 with jitter, and cap the retries. A retry storm against a rate limit is an outage with extra steps. Have a fallback model configured before you need it.
4. Never let model output be the only copy of a computed fact. Dates, counts, money: compute them in code, then ask the model to explain. Codex's calendar miss cost nothing here because the answer already existed in a script.

Which model for which job

What this test does not prove

One sample per task, one day, one endpoint. Repeat any of these prompts twenty times and the failure pattern shifts, especially the reasoning one — a single calendar miss is a data point, not a ranking. Long context, vision and tool calling were not tested at all. Prices are vendor list rates converted at published prices, not what you would pay on a contract.

What does hold: at this difficulty, correctness stopped being the differentiator. Latency, output tokens, and the defaults on your request budget are. Pick on those, and measure them against your own prompts before you commit.