Deleting your losers buys 4.1 points of hit rate: an audit protocol for model retirement

Every prediction system eventually grows a lifecycle manager: models get born, promoted, demoted and killed. Ours has 8,789 model rows and a retirement pipeline that runs every two hours. For a long time the number we published was computed over whatever models happened to still exist at query time — a mistake whose size is measurable.

46,160 graded predictions carry a boolean hit. The all-cohort rate is 57.05% (26,334 correct). The rate over models still in a live lifecycle state is 61.17% (19,281 of 31,520). Nothing about the models changed between those queries — only the WHERE clause did. A 4.12-point swing produced entirely by bookkeeping. In a system that retires models automatically, survivorship bias is not an edge case; it is the default output. What follows is the audit protocol we now run before any retirement decision.

Where a model actually goes when it dies

There is no single "deleted" state. A model leaves the visible cohort through six different doors, and the aggregate rate treats each differently.

Three things stand out. First, the worst cohorts are also the smallest. demoted holds 2,937 model rows but only 1,014 graded predictions — 0.35 per model. pending_retrain holds 56 models and 318 predictions at 28.30%. These are not measured failures; they are barely-measured ones. A rule that fires on 0.35 observations per model is not selecting on quality, it is selecting on noise.

Second, eliminated is a real signal. 801 models, 7,950 graded predictions, 45.11% — 11.98 points below the evolving cohort on a sample worth reading.

Third: 4,854 graded predictions reference 525 model_id values that no longer exist in ai_models at all. Their score, 56.61%, is close to the pool average, and any metric written as JOIN ai_models silently drops every one of them. We also keep an ai_models_archive table with 4,054 rows, while archived_at is NULL on all 8,789 live rows — the primary table cannot tell you when anything was retired.

The audit query: publish two numbers, never one

The protocol: publish two numbers from one query so they cannot drift apart — the headline, and its complement over the full historical cohort including rows whose model row is missing.

-- Cohort audit: the same graded rows, seen through two different filters. WITH graded AS ( SELECT p.id, p.model_id, p.hit, p.created_at, COALESCE(m.lifecycle_status, 'hard_deleted') AS lc FROM ai_predictions p LEFT JOIN ai_models m ON m.id = p.model_id WHERE p.settled = TRUE AND p.hit IS NOT NULL ) SELECT 'all_cohort' AS view, COUNT(*) AS n, SUM(hit::int) AS hits, ROUND(100.0 * SUM(hit::int) / COUNT(*), 2) AS hit_rate FROM graded UNION ALL SELECT 'live_only', COUNT(*), SUM(hit::int), ROUND(100.0 * SUM(hit::int) / COUNT(*), 2) FROM graded WHERE lc IN ('evolving', 'active', 'incubating', 'validating') UNION ALL SELECT 'retired_cohort', COUNT(*), SUM(hit::int), ROUND(100.0 * SUM(hit::int) / COUNT(*), 2) FROM graded WHERE lc IN ('eliminated', 'demoted', 'archived'); 

 view | n | hits | hit_rate ---------------+-------+-------+---------- all_cohort | 46160 | 26334 | 57.05 live_only | 31520 | 19281 | 61.17 retired_cohort| 8964 | 3931 | 43.85 

The rule we now enforce: the delta between all_cohort and live_only is itself a monitored metric. Call it the retirement gap. When it grows, the pipeline is deleting measured failures faster than unmeasured models — flattering the published number without improving the system. Month by month:

The gap is not constant. It is smallest in the month when almost nothing had been retired and largest when a quarter of the pool was already gone. The full series runs between 50% and 62%, and part of the recent stability is the deletion of 23% of the evidence.

Why the survivors look better than they are

Hard deletion destroys the denominator. 525 model ids vanished from ai_models while their predictions stayed, so every JOIN-based metric silently excludes 4,854 rows — worse than a wrong number, because nothing in the output declares it.

Status filters are applied at read time, not write time. The same physical rows produce 57.05% or 61.17% depending on a WHERE clause chosen by a dashboard author, and no schema-level signal marks that the choice exists.

The leaderboard adds a minimum-sample filter on top. Of 4,681 models with any graded prediction, only 660 reach 20 and 170 reach 50; the median is 6. A "top models by hit rate" view is therefore almost entirely models whose whole record is under ten games, and sorting it descending returns 100% records: 388 with fewer than five graded predictions, 18 more between five and nine.

Here is the part that should end the argument. If every model were exchangeable at the pool rate of 0.5705, the expected number of perfect records is Σ p^n over the actual sample sizes:

models with n < 5 : expected 634.5 perfect records, observed 388 models with n >= 5 : expected 25.3 perfect records, observed 18 

Observed is below the coin-flip expectation in both bands. A 4-for-4 record is not evidence of skill here — it is less impressive than chance would produce. Any rule that ranks on raw rate ranks a mixture of noise and sample size, and the models it rewards are the ones that guessed few times.

Make retirement append-only and auditable

The fix is not "stop filtering" — some views legitimately want the live cohort. The fix is to make the filter explicit, permanent and reversible, so excluded rows stay addressable. Retirement becomes an event, never a delete.

"""Retirement ledger: retirement is an event, never a delete.""" import hashlib, json, datetime as dt def _h(prev: str, payload: dict) -> str: body = json.dumps(payload, sort_keys=True, separators=(",", ":")) return hashlib.sha256((prev + body).encode()).hexdigest() def retire(cur, model_id: int, new_state: str, reason: str, evidence: dict): """Append one row. Never UPDATE ai_models.status, never DELETE.""" cur.execute( "SELECT state, entry_hash FROM model_lifecycle_ledger " "WHERE model_id = %s ORDER BY id DESC LIMIT 1", (model_id,)) row = cur.fetchone() prev_state, prev_hash = (row if row else ("born", "0" * 64)) payload = { "model_id": model_id, "from_state": prev_state, "to_state": new_state, "reason": reason, # 'eliminated_low_n' "evidence": evidence, # n, hits, rate, window "at": dt.datetime.utcnow().isoformat(timespec="seconds"), } cur.execute( "INSERT INTO model_lifecycle_ledger " "(model_id, from_state, to_state, reason, evidence, entry_hash, prev_hash) " "VALUES (%s,%s,%s,%s,%s,%s,%s)", (model_id, prev_state, new_state, reason, json.dumps(evidence), _h(prev_hash, payload), prev_hash)) # projections may change; history may not cur.execute("UPDATE ai_models SET lifecycle_status = %s WHERE id = %s", (new_state, model_id)) 

With that ledger, the cohort view stops being a guess:

CREATE OR REPLACE VIEW v_model_cohort AS SELECT m.id AS model_id, m.lifecycle_status AS current_state, COALESCE(f.first_retired_at, TIMESTAMPTZ 'infinity') AS retired_at, f.retirement_reason FROM ai_models m LEFT JOIN LATERAL ( SELECT MIN(at) AS first_retired_at, (ARRAY_AGG(reason ORDER BY at))[1] AS retirement_reason FROM model_lifecycle_ledger l WHERE l.model_id = m.id AND l.to_state IN ('eliminated', 'demoted', 'archived') ) f ON TRUE; 

Now "as of date D" is a temporal join instead of a status filter: a model retired in September still contributes its September predictions to every point-in-time metric, and the historical series can no longer be rewritten by a later decision. Three invariants belong in CI, each of which our bad state violated:

1. COUNT(*) FROM ai_predictions p WHERE NOT EXISTS (SELECT 1 FROM ai_models m WHERE m.id = p.model_id) must be zero. Ours was 4,854.
2. archived_at must be non-NULL for every row whose state is archived. Ours was 0 of 170.
3. Every state change needs a ledger row, and the chain must verify from genesis. Ours had 1,185 rows with reactivated_at set and no recorded reason.

What a retirement decision actually needs

Judge a retirement rule on the predictions it would have removed, not on the leaderboard it produces afterwards. Report four numbers together for every candidate cohort: n, hits, rate, and the rate of the cohort that survives. A rule removing 8,964 rows at 43.85% from a pool running at 57.05% raises the published number by 3.18 points while deleting 19.4% of the evidence. Whether that trade is defensible depends on whether the rule fired on measured failure — and with demoted averaging 0.35 graded predictions per model, ours mostly did not.

The honest summary of our pool: 63.6% of model rows (5,587 of 8,789) have never produced a single graded prediction, the median model has six, only 170 clear fifty, and 253 models were fathered by parents that were later eliminated. A system that kills a parent but keeps its descendants runs a selection loop it cannot audit, because the descendants inherit a training word whose measured performance was negative and nothing records the inheritance.

Retirement is a data operation, not a UI operation. If deleting a row changes the headline metric, that metric is a function of your bookkeeping rather than your model quality — and the cheapest way to improve it is always to delete more.

Analytics and research content. Not betting advice. 18+

Disclaimer: This content is provided for data analysis and research purposes only and does not constitute any decision-making advice.