A backtest's headline number is an average. Averages hide two common problems. Both have cheap checks.
1. Is it just one stock?
A portfolio of 20 names can look great because one of them went up tenfold. That isn't a strategy. That's one lucky pick, and it won't repeat.
The check: report what share of the total gain came from the top three names.
contrib = (weights * moves).sum(axis=0) # each stock's total contribution top3_share = contrib.nlargest(3).sum() / contrib[contrib > 0].sum() If most of the result comes from three names, you're looking at concentration, not skill. Our portfolio page shows this number for each of our hypothetical portfolios.
2. Is it just one year?
What worked in 2021 failed in 2022. Interest rates, inflation and what investors fear all change, and a model can be tuned, without anyone meaning to, to one kind of market.
The check: show results year by year, not one average. Better still, mark the different market periods.
A model that's strong in one year and weak in all the others hasn't found something stable. One that's mildly positive in most years is more interesting than one with a high average driven by a single year.
What we do
- The top-three share on the portfolio pages.
- Year-by-year results on every model page.
- Both reported whether they look good or bad.
The general idea
Before trusting any average, break it down: by stock, by year, by market condition. If the result only lives in one slice, it's probably luck.
(Our portfolios are hypothetical paper portfolios, not a claim of real-world results, and they could easily do worse than their benchmarks.)
From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/portfolios
Educational, not investment advice.
