Your leaderboard is ranking luck: 3,736 models, 46,284 graded calls, and a split-half correlation of 0.04

We run a population of model instances that publish pre-event probability calls on football fixtures. Every call is written to ai_predictions at creation, graded exactly once after the event, and stored as a boolean. As of this audit: 46,884 rows, 46,284 with a non-null grade, 26,415 hits — a global rate of 57.07%.

Our public leaderboard reads ai_models.hit_rate, sorts descending, renders the top. Standard design, structurally broken. The failure is not the sort comparator; it is a property of sorting point estimates computed from wildly unequal denominators.

Here is the board as it renders today:

Four rows, four perfect records, none above seven graded calls. Across the table, 327 models sit at exactly 100.0%, and 204 of those have a single graded call. Of the 584 models with at least 20 graded calls, zero are at 100%. The first hundred rows of the board contain no model with ten or more calls.

A leaderboard topped by 204 coin flips is not a leaderboard. It is a sampling artifact rendered as a ranking.

Two populations, one table

The first thing an audit has to separate is how many models exist versus how many have enough evidence to be compared.

-- population split: board cache vs. the fact table SELECT count(*) AS board_rows, count(*) FILTER (WHERE total_predictions > 0) AS with_evidence, percentile_cont(0.5) WITHIN GROUP (ORDER BY total_predictions) AS median_n FROM ai_models; WITH t AS ( SELECT model_id, count(*) AS n, count(*) FILTER (WHERE hit) AS h FROM ai_predictions WHERE hit IS NOT NULL AND model_id IS NOT NULL GROUP BY 1 ) SELECT count(*) AS models, sum(n) AS rows, sum(h) AS hits, count(*) FILTER (WHERE n >= 20) AS n20, count(*) FILTER (WHERE n >= 50) AS n50, count(*) FILTER (WHERE n >= 200) AS n200, percentile_cont(0.5) WITHIN GROUP (ORDER BY n) AS median_n, max(n) AS max_n FROM t; 

Output, unedited:

board_rows 8942 | with_evidence 3229 | median_n 7.0 models 3736 | rows 46284 | hits 26415 | n20 661 | n50 171 | n200 7 | median_n 6.0 | max_n 381 

Three numbers matter. The median model has six graded calls, 171 models clear fifty, seven clear two hundred. Sorting 3,229 estimates where the median sample is 7 is not ranking models — it is ranking the draws.

The test: split each model's history in half

If a hit rate reflects a durable property of the model, the rate on the first half of its own history must predict the rate on the second half. If it reflects noise, the halves are independent and the ranking carries no forward information.

The split is by sequence position, not calendar time, so a model that started last week is treated like one that started in June. Both halves need at least ten graded calls, restricting the test to the 661 models with n ≥ 20.

WITH ordered AS ( SELECT model_id, hit, row_number() OVER (PARTITION BY model_id ORDER BY id) AS rn, count(*) OVER (PARTITION BY model_id) AS n FROM ai_predictions WHERE hit IS NOT NULL AND model_id IS NOT NULL ), halves AS ( SELECT model_id, max(n) AS n, avg(CASE WHEN rn <= n/2 THEN hit::int END) AS h1, avg(CASE WHEN rn > n/2 THEN hit::int END) AS h2, count(*) FILTER (WHERE rn <= n/2 AND hit) AS k1, count(*) FILTER (WHERE rn <= n/2) AS n1 FROM ordered WHERE n >= 20 GROUP BY model_id ) SELECT count(*) AS models, corr(h1, h2) AS pearson, avg(h1) AS mean_h1, stddev_samp(h1) AS sd_h1, avg(h2) AS mean_h2, stddev_samp(h2) AS sd_h2 FROM halves; 

Result over 661 models:

models 661 | pearson 0.0443 | mean_h1 0.6291 | sd_h1 0.0991 | mean_h2 0.5656 | sd_h2 0.1135 

A correlation of 0.0443. Spearman on the same pairs is 0.0261. For practical purposes these are zero — the first half of a model's record tells you almost nothing about its second half.

Note the drift, not just the flatness: first half averages 62.91%, second half 56.56%. Instances are not stationary across their own lifetime. One tuned against recent fixtures starts hot and mean-reverts, and a board built on the early record is systematically reading the hot streak.

What the top of the board is worth

Take the 661 pairs, sort by first-half rate, cut the top 66, look at what that group did next.

The top decile displays 80.85% and delivers 60.36%. The surviving edge over the rest of the population is 3.8 points, not the 24.7 the display implies. The bottom decile displays 46.14% and delivers 56.29% — the global rate within noise. Both tails converge on 56–57% because that is where the population lives; the spread at the tails was estimation error, not skill.

This is what rewarding luck means numerically. A 100.0% display on seven calls is not a stronger signal than a 60% display on 380 calls — it is a weaker one wearing a better font.

The multiplicity check

Under the null that every model draws from the same 57.07% process, the probability of a clean sweep at n calls is 0.5707^n:

n=5 6.05e-02 n=7 1.97e-02 n=10 3.67e-03 n=20 1.34e-05 

At n=7 a perfect record happens once in fifty-one. With 3,229 board rows and thousands of small-n models, dozens of perfect records are expected, not remarkable. Summing 0.5707^n over every model's actual n gives an expected 661.75 perfect records. We observe 401.

Observed is below expected, and the reason is informative: the smallest buckets underperform. Models with 1–9 graded calls hit 49.02% across 8,606 rows; the 100–199 bucket hits 61.01%. Fresh instances start worse than the population they are compared against, so perfect records are not over-represented relative to chance — they are the surviving right tail of many short, slow-starting records.

Correct for multiplicity and the picture gets worse. With 3,229 candidates at α = 0.05 the per-test threshold is 1.55e-5, which requires n ≥ 20 with a clean sweep. We have 584 models at n ≥ 20 and none is perfect. No defensible 100% exists on this board.

Trying to fix the ranking function

The obvious next move is replacing the point estimate with something that penalises small samples — a Wilson lower bound, or a Beta posterior shrunk toward the population rate. Implement both, then re-run the same split-half evaluation at the same 66-model cut.

import math P0 = 0.570716 # 26,415 hits / 46,284 graded rows STRENGTH = 20.0 # pseudo-count mass pulling toward P0 def wilson_lb(k, n, z=1.96): if n == 0: return 0.0 p = k / n denom = 1 + z * z / n centre = p + z * z / (2 * n) margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) return (centre - margin) / denom def shrunk(k, n, prior=P0, strength=STRENGTH): return (k + prior * strength) / (n + strength) def top_slice(rows, key, k): """rows: dicts with h1, k1, n1, n, h2""" chosen = sorted(rows, key=key, reverse=True)[:k] return {"second_half": round(sum(r["h2"] for r in chosen) / len(chosen), 4), "mean_n": round(sum(r["n"] for r in chosen) / len(chosen), 1)} 

Run over the same 661 pairs, k = 66:

The honest result: none of them fixes the forward rate. Every variant lands within 3.8 points of the population mean, and the raw sort is nominally best — not because raw sorting is good, but because the top decile of a positively-correlated noisy signal still concentrates some genuine signal, and shrinkage trades that sliver for conservatism.

The problem is not the ranking function. It is the object being ranked. With a median of six graded calls, no monotone transform of (k, n) can recover forward skill, because there is no forward skill in the estimator to recover.

What actually works

Never render a point estimate without its denominator and interval. 60.4% (n=62, 95% CI 47.9–71.7) is honest. 100.0% is a lie of omission when n=4. The interval is one line of Python; the bare number has no excuse.

Freeze the board as a historical artifact. A live query whose ordering reshuffles as rows arrive can be gamed by waiting. Materialise a weekly snapshot of (as_of, model_id, k, n, rank) and publish that. A model falling from rank 1 to rank 90 then becomes a visible fact instead of a silent overwrite.

Report the panel, not the total. The 171 models at n ≥ 50 average 59.66% with a standard deviation of 6.17 points. That spread is the real content: the credible population sits in a band roughly ±12 points wide at two sigma. A ranked list of 3,229 rows destroys that information; a distribution preserves it.

Set the threshold by what the estimate can support. At n = 50 the standard error of a 0.6 estimate is 0.069, so a 95% interval spans ~27 points. Nobody should compare two models across a 27-point interval. Either label the board as a discovery surface below fifty calls, or restrict ranking to the 171 models where the interval is narrower than the effect being claimed.

The generalisable lesson

Any system ranking entities by a ratio of two counters fails this way once the denominators vary by more than an order of magnitude — recommender A/B tests, creative scoring, per-shard error rates, per-endpoint latency percentiles, analyst dashboards. The tell is always the same: the top of the list is held by the smallest denominators, and the list collapses into ties at the boundary value (100%, 0%, perfect).

The diagnostic is cheap: split each entity's own history in half, correlate the halves, and check what the top decile delivers next. Near-zero correlation plus a twenty-point mean reversion means the ranking is measuring sample size, not the entity.

We still show the board. We now show it with n, with intervals, with a weekly frozen snapshot, and with a hard label that below fifty graded calls the ordering is not a comparison. The models did not get worse. The display stopped flattering the ones that had barely played.

Analytics and research content. Not betting advice. 18+

本内容仅供数据分析与研究参考，不构成任何决策建议。