Model leaderboards measure a model's ceiling under official or single-host conditions. Uptime checks tell you whether an HTTP endpoint is reachable. Neither answers the question that shows up in production: when a model is served through a third-party API relay, do its reasoning depth, constraint following, and model-level behavior still match the direct official API?
An endpoint can keep returning HTTP 200, answer a short prompt fluently, and even look faster because it thought less. That surface health can hide a quieter class of failures: a cheaper model or older snapshot on the backend, a lowered reasoning budget with the same model name, or a truncated context window. This post is a condensed version of Folkbench's method write-up on how Integrity Monitor tries to surface that drift. The canonical article, including figures, is here:
Introducing Folkbench Model Degradation Detection
Why uptime and naive checks miss the problem
In a typical relay path, a gateway sits between the caller and the model lab. Most of the time it just forwards requests and meters usage. Under cost pressure or peak routing, though, three kinds of degradation barely show up in network telemetry:
- Silent model substitution. The client requests a flagship model; peak-hour or long-context traffic is sent to a cheaper model or an older snapshot.
- Reduced reasoning budgets. The model family name stays the same, but reasoning effort is lowered or internal thinking steps are cut short.
- Context or output truncation. Earlier turns are dropped, or output caps are tightened, so multi-step tasks start missing constraints.
Figure 1. HTTP 200, low initial latency, and fluent short replies can still sit on top of a routing swap, a trimmed reasoning budget, or a chopped history.
Sampling randomness makes this harder. A couple of bad replies are not a diagnosis — as discussed in Did the Model Really Get Dumber?, isolated screenshots cannot separate normal variance from systemic drift.
Two manual checks fail for structural reasons:
- Asking the model to name itself. A language model cannot see the proxy in front of it. A gateway system prompt, or stock phrasing memorized by a smaller model, will still produce a confident flagship identity.
- Reusing well-known static puzzles. Those prompts are already in public training data, and a repeated monitoring script is easy to cache or whitelist onto a clean upstream while everyday traffic stays degraded.
Figure 2. Left: a gateway prompt can dictate the self-identified model name. Right: repeated static prompts can be cached or sent down a whitelist path.
A useful check therefore needs prompts that are hard to pre-cache, and signals that go beyond whether one final answer happens to look right.
Dual-dimension probes in the same window
Folkbench splits the work into two complementary probe types, collected together in each evaluation window.
Current-events probes fold recent public timelines and newly introduced constraints into structured tasks, and they keep changing. That makes static caches and frozen answer keys less useful, and it tends to expose outdated snapshots (knowledge-cutoff errors) as well as over-compressed substitutes that drop conditions.
Model-fingerprint probes look at behavioral patterns that stay relatively stable across model families: multi-step constraints, formatting boundaries, and structured reasoning paths. Alongside output alignment, the monitor records:
- Probe agreement — how consistently behavior matches the target model's characteristic patterns.
- Input tokens and reasoning tokens — context ingestion and pre-answer thinking cost.
- Model fingerprints and cache checks — API-returned fingerprint fields, plus cache-check observations when data exists.
If reasoning effort is quietly lowered, the prose may still read well while reasoning-token volume and probe agreement fall together. If traffic is moved to a different model series, fingerprint and formatting deviations often show up before a simple factual question fails.
Calibrate against a live official baseline
A drop in a relay's score is not self-explanatory. It might be the relay. It might also be an official snapshot update, a short lab-side fluctuation, or a harder new prompt set whose official pass rate was already in that range. Without a synchronized reference, an absolute number cannot tell you what moved.
Every probe suite is therefore also run through isolated direct official API keys in the same window. Those keys talk to the official provider only — no third-party relay in the path — and they execute the same current-events and fingerprint tasks. The official result is a live baseline, not a fixed pass mark.
| Observation | Uncalibrated relay monitoring | Calibrated with direct official keys |
|---|---|---|
| Official model shows a minor update or variance | Easy to misread a lab-wide shift as relay degradation | The live official baseline moves in parallel, separating upstream noise from relay drift |
| Official baseline holds while a relay drops | No simultaneous reference for magnitude | Probe agreement and reasoning tokens diverge relative to the baseline |
| Service groups inside one relay diverge | A strong pool can mask a weaker budget pool | Each model × service group route is compared on its own |
Figure 4 is a mechanism diagram, **not measured telemetry. Do not read the curves as results for a particular vendor.
Interpretive boundaries matter. As the Integrity Monitor page states, a single response or isolated percentage cannot establish model substitution on its own. Read sample size, consecutive windows, token distributions, and test conditions together. The output is a reviewable early-warning signal, not a verdict.
Results are also kept at model × service group granularity. Discount pools, dynamic-multiplier routes, and higher-cost official-key pools under one domain can be completely different backends. A strong premium group should not stand in for the cheaper group you are about to buy.
Where this sits in the rest of the toolkit
Degradation detection is one layer. Folkbench also publishes four operational rankings — availability (including “fake 200” payloads and mid-stream cutoffs), stability, latency (TTFT separated from generation rate), and session cost — plus side-by-side relay comparison.
The open-source desktop client Folkbench Switch (Tauri and Rust; macOS and Windows) can preview config diffs for Claude Code, Codex, Gemini CLI, Aider, OpenCode, and Kimi CLI, and it parks official Codex OAuth credentials when you move to a third-party endpoint. Integrity monitoring itself remains a website evaluation feature; it is not a local check built into the client.
A practical sequence is: shortlist a specific model-and-group route on the operational rankings; check Integrity Monitor against the official baseline over several windows; compare the exact group you plan to purchase; keep a backup route ready if telemetry drifts.
The longer method post, with the full figures and source notes, is here: https://folkbench.com/en/blog/how-folkbench-detects-model-degradation?utm_source=devto
