7,426 forecasts against Polymarket: exactly where the market beat our models

Since September our statistical models have made 7,426 probability calls on Polymarket questions. Each call is stored next to the market price at the moment it was made, and the file holding it is timestamped into Bitcoin with OpenTimestamps. When the market resolves, both numbers are scored with the Brier score (lower is better).

On the 7,048 calls that have resolved, the market scores 0.161 and our models score 0.194. The market wins. This post is about where it wins, because the answer is more specific, and more useful, than "markets are efficient".

Everything below can be reproduced from the public dataset in a few lines of pandas (code at the end).

Headline numbers

track	resolved calls	our Brier	market Brier
weather	6,128	0.204	0.166
crypto	667	0.099	0.094
sports	165	0.213	0.208
markets	52	0.196	0.147
politics	23	0.140	0.201
tech	13	0.350	0.309
all	7,048	0.194	0.161

Weather is 87% of the sample, so it dominates the total. Without weather the gap is small (0.130 vs 0.123 on 920 calls). The only track where we are ahead is politics, on 23 calls, which proves nothing.

One more number worth keeping in mind: our call scored better than the market price on 50.5% of individual questions. We win about half the coin flips, and lose the total anyway. The losses are bigger than the wins, and the rest of this post is about why.

Finding 1: we were far too sure that unlikely things would not happen

Take every call where our model said 10% or less. There are 2,176 of them.

our average probability: 3.8%
the market's average price on the same questions: 16.5%
how often the event actually happened: 17.0%

The market was almost exactly right; we were off by a factor of four. Most of these are weather "band" questions (will the high temperature land in a given 2-degree range). Our models put a narrow, confident distribution on the temperature and gave neighbouring bands almost nothing. Reality was wider than the model.

The market's own calibration, by contrast, is close to the diagonal everywhere it has data:

market price bucket	calls	avg price	happened
0-10%	1,616	5.4%	5.3%
10-20%	1,201	15.1%	13.7%
20-30%	1,158	25.3%	26.3%
30-40%	1,206	35.3%	35.2%
40-50%	936	45.0%	45.7%

And ours:

our probability bucket	calls	avg ours	happened
0-10%	2,176	3.8%	17.0%
10-20%	1,216	14.9%	23.8%
30-40%	1,034	35.2%	31.4%
40-50%	710	45.2%	35.4%
50-60%	356	55.0%	41.3%
60-70%	133	65.0%	43.6%

So the error runs both ways: too low at the bottom, too high in the middle. That is the signature of a model that is overconfident about which band wins, not one that is biased in a single direction.

Finding 2: the more we disagreed with the market, the worse we did

Group the calls by the gap between our probability and the market price:

gap between us and the market	calls	our Brier	market Brier	we beat the price
up to 5 points	2,085	0.116	0.116	57%
5-10 points	1,290	0.153	0.149	54%
10-20 points	1,857	0.212	0.186	48%
20-30 points	1,070	0.249	0.193	47%
over 30 points	746	0.360	0.195	37%

When we roughly agreed with the price, we tied it. Every step of disagreement cost us, and the large disagreements, the ones that would have looked like "edge", were the worst calls in the set.

Finding 3: mixing our model into the price only made it worse

A natural rescue is to blend: take some weighted average of our probability and the market's. If the model carried any information the price did not, some blend should beat the price alone.

weight on our model	Brier of the blend
0% (market only)	0.161
25%	0.163
50%	0.169
75%	0.180
100% (model only)	0.194

Every amount of our model hurt. On this sample, the honest conclusion is that the models added nothing the price did not already know.

What we are not claiming

This is one set of models, mostly on one kind of question. It says little about forecasting in general and nothing about other teams.
Our first batch (calls from 14 to 26 September, 1,613 of them) was sealed late, on 26 September, after 1,394 had already resolved. Those rows are flagged sealed_before_resolution = false and are not evidence of forecasting in advance. They do not flatter us: on the 5,654 resolved calls sealed before their outcome, the result is the same (0.196 vs 0.160). Since 26 September every call is sealed minutes after it is made.
Nothing here is a trading signal or advice. It is a record of misses, kept so it cannot be edited later.

Reproduce it

import pandas as pd from datasets import load_dataset f = load_dataset("neuportal/forecasts-vs-market", split="train").to_pandas() f = f[f.status == "final"] # headline print(f[["brier_model", "brier_market"]].mean()) # finding 1: our 0-10% calls low = f[f.our_p <= 0.10] print(len(low), low.our_p.mean(), low.market_p_at_call.mean(), low.outcome.mean()) # finding 2: by disagreement gap = (f.our_p - f.market_p_at_call).abs() bins = pd.cut(gap, [0, .05, .1, .2, .3, 1], include_lowest=True) print(f.groupby(bins)[["brier_model", "brier_market"]].mean()) # finding 3: blends for w in [0, .25, .5, .75, 1]: p = w * f.our_p + (1 - w) * f.market_p_at_call print(w, ((p - f.outcome) ** 2).mean())

To check that a call existed when we say it did, open its proof_file in the proofs repository and drop the .ots file, then the .json, on opentimestamps.org. The Bitcoin block it reports is the latest moment that call could have been written.