Your LLM provider can quietly downgrade your model, and nothing will tell you
In October 2026, Chinese AI communities went through repeated waves of "degradation panic." Users on paid tiers discovered their models had been quietly swapped or throttled — no notice, no dashboard indicator, no changelog. People invented their own detection rituals: asking the model to draw a pelican riding a bicycle as SVG, candy-counting questions, "who is the current prime minister of Japan" fingerprint tests, even typing error messages into support chats to check for hidden account flags.
One-off fingerprint websites exist. But nobody was doing continuous canary monitoring with a real baseline and cost accounting. So I built model-canary: a zero-dependency Python watchdog that probes your endpoint on a schedule and tells you when something changed.
How it works
Four canary probes, each catching a different failure mode:
| probe | kind | what it catches |
|---|---|---|
identity-svg | identity | Pelican-on-a-bicycle SVG — the community gold standard. A silently swapped model draws a recognizably different picture. |
candy-arithmetic | reasoning | "3 candies, take 1" — a degraded model fumbles even trivial reasoning. |
instruction-repeat | instruction | Repeat "blue" five times — catches instruction-following decay. |
factual-anchor | factual | Stable factual question — signal is drift against your baseline, not absolute correctness. |
pip install model-canary model-canary init --endpoint https://api.openai.com/v1 --model gpt-6.1-sol model-canary run # probes run, baseline written on first run model-canary check # exit code tells you the verdict model-canary report # markdown drift report Put run + check on cron (hourly is plenty). check exits non-zero when something's wrong:
| exit | verdict | meaning |
|---|---|---|
| 0 | ok / warn / baselining | fine, or still collecting baseline samples |
| 2 | degraded | high-confidence signal: model swapped, quality dropped, or account flagged |
| 3 | probe_failed | transport broke — never reported as degradation |
| 4 | throttled | provider rate-limited you — also not degradation |
Every run logs token usage too, so report shows exactly what the watchdog itself costs. A monitor you can't afford to run is a monitor you'll turn off.
Design decisions that matter
Fail-open on alerting. A dead network is not a degraded model. Transport failures get their own verdict and exit code, and throttling gets its own too (exit 4). Only explicit account-level markers (like subscription_sharing_user_not_eligible) raise an account_flag. I match structured error.code/error.type fields, not substrings — matching "capacity" as a substring would flag half the internet's 503s.
No single-sample decisions. run --samples 5 (default) takes multiple samples; check looks at a window of recent runs and needs k-of-n confirmation. temperature=0 doesn't guarantee determinism on most providers, and some routers send a fraction of traffic to smaller models — one sample can't catch either case.
Baseline hygiene. The baseline needs n≥5 samples before it's trusted (before that, check reports baselining). Baselines carry model/endpoint/prompt-hash metadata — if you switch models, you get baseline_stale, not a false drift alarm. The representative sample is the medoid, not the last run. Rebuilding a baseline excludes runs that already showed drift, so a degraded period can't poison its own reference.
Atomic ledger. The ledger is append-only JSONL with O_APPEND + flock, single-write records, and the baseline is written to tmp + os.replace. A crashed or overlapping cron run can't corrupt either.
The review that caught real bugs
I had Claude do a full code review before publishing. It found three critical issues I'd missed, all false-positive factories:
- Guaranteed fingerprint false positive. The baseline sample was truncated to 500 chars while run records kept 4000. With
SequenceMatcher, any SVG longer than ~1320 chars mathematically could not score above the 0.55 threshold — every real pelican SVG would have been flagged as "model swapped." Fixed: identical normalization on both sides,autojunk=False, and a dynamic threshold derived from baseline sample similarity instead of a hardcoded 0.55. - Rate limits reported as degradation. The old code appended a HIGH
error_signalwhenever an error body contained "rate_limit" or "capacity" — a normal 429 would page you at 3am as "model degraded." Now throttling is its own verdict. - In-stream errors treated as success. Some relays return HTTP 200 with
{"error": ...}inside the SSE stream. The old code readchoices, got empty text, and reported quality/fingerprint/length drift. Now a stream error raises immediately, and a stream without[DONE]or with a non-stopfinish reason is a probe failure, not a degradation.
All 21 tests pass, including regression tests for each of these.
Limitations, stated plainly
- Probes cost tokens every run. Hourly, not minutely.
- Fingerprint heuristics are statistical — alerts mean "go look," not "verdict."
- If a provider degrades everyone identically, the identity probe won't trip it. That's what the reasoning probes are for.
- Your API key is read from env and never written to disk.
MIT licensed. If you're paying for API access through any channel you don't fully control, a canary is cheaper than the alternative: finding out from your users.
