A Permutation Test for Backtests That Searched Too Hard (Python)

AI disclosure: this post was written by an AI agent (Claude) for the AI Crypto Secrets team. Every number below comes from running the code shown, on synthetic data.

Pick the best of 355 trading rules on a price series that contains no edge at all, and about a third of the time the winner clears an annualized Sharpe ratio of 1.0. A threshold can't tell that winner apart from a real strategy, because the winner was chosen for looking good. A permutation test can, and it fits in a few dozen lines of NumPy.

What question does the test ask?

A backtest is the survivor of a search. If you tried 355 variants, the number you report is the maximum of 355 scores, and the maximum of many noisy numbers is large even when every one of them is noise.

So don't ask "is a Sharpe of 1.2 good?". Ask: how often would the best of the same 355 rules reach 1.2 on data that has no structure? To find out, shuffle the returns, rerun the entire search on the shuffled data, and record the best score. Repeat that a few hundred times. The p-value is the share of shuffles whose best beat your real best.

That is the same idea behind White's Reality Check, which tests whether the best model found in a specification search has any real superiority, rather than treating the winner as if it had been the only candidate. What follows is a simpler cousin: a shuffle instead of a bootstrap, and no benchmark model. It is enough to see how the correction works.

The setup

The data is 6,000 synthetic 8-hour bars, about 5.5 years. The first 4,200 are the research sample, which is the only part the search sees. The rules are 355 moving-average crossovers: go long when the fast average is above the slow one, short otherwise, with fast windows from 5 to 100 bars and slow windows from 20 to 400. Signals act on the next bar, and every change of position costs 5 basis points.

For the "does it work" half of the experiment we also generate markets with a weak real effect: a drift that fades with a half-life of about 34 bars (roughly 11 days) and flips sign unpredictably, sized as a fraction of one bar's volatility.

The code

import numpy as np BARS_PER_YEAR, N_BARS, SPLIT = 3 * 365, 6000, 4200 # 8-hour bars; first 70% is the research sample SIGMA, COST = 0.012, 0.0005 # 1.2% volatility per bar; 5 bps per unit of position change GRID = [(f, s) for f in range(5, 105, 5) for s in range(20, 420, 20) if f < s] # 355 crossover rules WINDOWS = sorted({w for rule in GRID for w in rule}) FAST = np.array([WINDOWS.index(f) for f, s in GRID]) SLOW = np.array([WINDOWS.index(s) for f, s in GRID]) WARMUP = np.array([s for f, s in GRID]) MASK = (np.arange(SPLIT)[None, :] >= WARMUP[:, None]).astype(float) # skip each rule's warm-up bars  def rule_returns(rets): """Net returns of all 355 rules at once, shape (355, T). Signals act on the NEXT bar: no look-ahead.""" close = 100 * np.exp(np.cumsum(rets)) c = np.cumsum(np.insert(close, 0, 0.0)) sma = np.full((len(WINDOWS), len(close)), np.nan) for i, n in enumerate(WINDOWS): sma[i, n - 1:] = (c[n:] - c[:-n]) / n pos = np.roll(np.nan_to_num(np.sign(sma[FAST] - sma[SLOW])), 1, axis=1) pos[:, 0] = 0.0 return pos * rets - COST * np.abs(np.diff(pos, axis=1, prepend=0.0)) def sharpe(r): """Annualized in-sample Sharpe ratio of every rule.""" r = r[:, :SPLIT] rm = r * MASK n = MASK.sum(axis=1) mean = rm.sum(axis=1) / n var = (rm * r).sum(axis=1) / n - mean ** 2 return mean / np.sqrt(np.maximum(var, 1e-18)) * np.sqrt(BARS_PER_YEAR) def p_value(rets, n_perm=199, seed=0): """How often does the BEST of all 355 rules, run on shuffled in-sample returns, beat the real best?""" rng = np.random.default_rng(seed) best = sharpe(rule_returns(rets)).max() shuffled = rets[:SPLIT].copy() null = np.empty(n_perm) for b in range(n_perm): rng.shuffle(shuffled) # same returns, no time structure left  null[b] = sharpe(rule_returns(shuffled)).max() # search the SAME 355 rules again  return (1 + np.sum(null >= best)) / (n_perm + 1) def make_market(seed, edge=0.0): """Zero-drift noise, plus (if edge > 0) a weak persistent drift that flips sign every few weeks.""" rng = np.random.default_rng(seed) rets = rng.normal(0.0, SIGMA, N_BARS) if edge: rho = 0.98 eta = rng.normal(0.0, edge * SIGMA * np.sqrt(1 - rho ** 2), N_BARS) mu = np.zeros(N_BARS) for t in range(1, N_BARS): mu[t] = rho * mu[t - 1] + eta[t] rets += mu return rets 

Two details matter. The shuffle runs the whole 355-rule search again, so the null distribution contains the same selection effect as the real result. And the + 1 in the p-value is the standard correction that stops a permutation test from reporting a p-value of exactly zero.

Two markets, one threshold

for seed, edge in ((2000, 0.0), (2001, 0.1)): rets = make_market(seed, edge) r = rule_returns(rets); sc = sharpe(r); k = int(np.argmax(sc)) print(f"seed {seed} edge {edge}: best rule {GRID[k]}, in-sample Sharpe {sc[k]:.2f}, p = {p_value(rets):.3f}") 

seed 2000 edge 0.0: best rule (55, 80), in-sample Sharpe 1.23, p = 0.145 seed 2001 edge 0.1: best rule (35, 220), in-sample Sharpe 2.17, p = 0.020 

The first market is pure noise, and its best rule scores a Sharpe of 1.23. That would pass most informal screens. The test says that a result at least that good appears in about 14.5% of shuffles, so there is no reason to believe it. The second market contains a real drift, and its winner is a clear outlier against the shuffled searches. (We picked the first noise market whose winner crossed 1.2, which is itself a little cherry-picking; the aggregate below is the honest view.)

What happens across 100 markets?

The chart at the top of this post shows the whole experiment: on the left, where the p-values land; on the right, how often each method calls a market "real".

On pure noise the threshold raises a false alarm in nearly one market out of three. The permutation test raises one in about twenty-five, in line with the 5% it was built for (4 of 100 is well inside the margin you'd expect from a sample of that size). When the effect is real but weak, the test often declines to call it, flagging 9% at the weakest level and 49% at the stronger one. That is the price of controlling false alarms: with a short sample and a small effect, "can't tell from luck" is the honest answer.

For context on whether the effects are real, the rule picked in-sample and then evaluated once on the untouched final 1,800 bars averaged a Sharpe of -0.03 on noise, 0.13 with the weak trend and 0.62 with the stronger trend. Real effects persist out of sample and noise does not. Each individual out-of-sample figure is itself noisy, with a standard error of roughly 0.8 for a window of that length, so the averages carry the message and single markets don't.

Where this test stops being enough

It only corrects for the search you included. Every variant you tried and discarded, every parameter you nudged by eye before running the script, has to be part of the 355. If you tuned for a week and then listed three finalists, the honest count is far larger than three.

Shuffling destroys all serial structure, including volatility clustering. For real returns a block bootstrap, which resamples chunks of consecutive bars, is the more defensible null, and the null you choose should match the claim you are testing.

And it doesn't replace out-of-sample data. It tells you the winner is hard to explain by selection luck inside one sample. Whether it keeps working on data it never saw is a separate question, which we cover in this explainer on out-of-sample validation.

Disclosure

We build AI Crypto Secrets, a systematic, non-custodial trend-following bot for crypto perpetual futures. The crossover rules above are a textbook example for this experiment, not our strategy, and all data is synthetic, so nothing here describes the performance of any real product. Trading leveraged crypto derivatives carries a real risk of loss, including the loss of all capital.

Reference

• White, H. (2000). A Reality Check for Data Snooping. Econometrica, 68(5), 1097-1126.