I built a Python paper-trading bot for crypto spot markets. The backtest looked fine. Then I tested it the strict way, and it fell apart. This is what the strict way looks like, and what it found.

The rule: lock the criteria before you look

Before each test I write down the hypothesis, the data, the metric and the pass/fail thresholds. I hash the file (sha256). The test script refuses to run if the hash does not match. After that I get one run. No tweaking the threshold after seeing the result.

It sounds bureaucratic. It is the only thing that stopped me from fooling myself, because I tried a lot of ideas and some of them looked good at first glance.

Result 1: the bot had no edge before costs

I replayed the bot's engine on history: 3,314 simulated trades.

  • Profit factor before any costs: 0.979
  • After slippage: 0.949
  • After fees: 0.894

So the signal itself was already slightly negative (about -0.07% per trade), and costs of about 0.30% per round trip did the rest. About 81% of the total loss was cost, 19% was the signal.

The shape of the trades was also a trap. Only 25.3% of trades won, with a payoff of 2.63 to 1. Breakeven needed 27.5% winners. We were 2.2 points short, and no amount of stop tuning changes that if entries are no better than random.

Result 2: the "random entries beat the bot" finding was my own bug

At one point a control showed random entries doing much better than the bot: profit factor 1.69 against 0.89. I almost wrote it up as a discovery.

I locked a follow-up test to find out why. Answer: the random entries were drawn from a window of plus or minus 14 days around each bot entry. The bot buys after a price rise by construction, so random entries placed before it were buying the same rise cheaper. Random entries placed after the bot entry:

  • Random before: average +0.277 R, profit factor 1.69
  • Random after: average -0.063 R, profit factor 0.875
  • The bot: average -0.068 R, profit factor 0.894

The advantage was look-ahead in my control group, not a flaw in the bot. A placebo test is only useful if the placebo is clean.

Result 3: survivorship explained every false positive

Momentum, rotation and several other ideas looked good on today's list of coins. On a point-in-time universe that includes coins that were later delisted, they vanished. In my own post-mortem, the choice of universe explained all of the false positives I found.

Result 4: the effect sits in a handful of episodes

When the "good" result survived a basic test, I removed the best 5% of days or the top few episodes. In several cases the effect went to zero or flipped sign. The number of independent events was in the tens, even though the table showed thousands of rows, because rows from the same day are not independent.

What I took from it

  1. Write the pass/fail criteria first. Hash them. One run.
  2. Test against a placebo, and check that the placebo is clean.
  3. Check the result without its best days.
  4. Count independent events, not rows.
  5. Include delisted instruments.
  6. Stress the costs. Most small effects are smaller than the fee.

I turned this method into a small command-line toolkit, called BotGauntlet: you lock your criteria, run the tests on your own trade list, and get a plain PASS/FAIL report. It will most likely tell you your idea does not work, which is the point. It is not signals and not financial advice. I am collecting a waitlist at https://botgauntlet.com and I would like to hear which of these checks you already run.

Disclosure: the code and analysis were developed with AI assistance (Claude). All numbers above come from my own pre-registered runs on historical data, a simulation, not live trading. Trading involves risk of loss.