What is overfitting?

Try enough ideas and one will look brilliant by accident. Overfitting is when a strategy is shaped (by you, or by a search) to fit the past so well that it is mostly fitting noise, so it falls apart on data it hasn't seen.

The cure isn't cleverness. It is bookkeeping: write the rule down first, count every try, and judge the result on years you didn't peek at.

What we do

1. Pre-register

The rule, the windows and the pass criteria are written into a file and committed in version control before the run. The runner refuses to start on a file that has uncommitted changes. If we change the rule after seeing results, that is a different idea and the page says so.

2. Split the years

  • Training: 2005–2016. Searching and tuning are allowed here.
  • Validation: 2017–2021, a strongly trending market.
  • Second window: 2022–2023, when stocks and bonds fell together.
  • Reserved: from 2 January 2024 on. Never used for any published result. (Honest footnote: a few early demo runs, before we enforced this, did see some of it.)

3. Count every try

Each backtest, parameter combination and variant goes into a trial registry. A result with no trial count is just a pretty line.

4. Judge against what chance would do

  • Deflated Sharpe ratio (DSR) (Bailey and López de Prado, 2014): given how many things we tried, how likely is the best Sharpe ratio to be real? Our bar is 0.95.
  • Probability of backtest overfitting (PBO) (Bailey, Borwein, López de Prado and Zhu, 2017): how often does the in-sample winner land below the median out of sample? Our general bar is 0.20; for one family we set a looser 0.50 in advance and it still failed.
  • False-discovery control (Benjamini and Hochberg, 1995) when running many tests at once.
  • Null comparisons: shift the indicator in time, pick random entry dates, or train on shuffled answers. If scrambled data does as well, the "edge" isn't one.

5. Compare fairly

A strategy that is invested 50% of the time looks safer than SPY for free. So we also compare against SPY held at the same invested level (rest in T-bills) and at the same volatility, plus an equal-weight basket of the same funds.

6. Trade it realistically

Decide at the close, fill at the next open. Spreads, impact and fees included. Cash settles the next day, like a retirement account, with no leverage. Idle cash earns the 3-month T-bill rate.

How things die

Type Meaning
Overfit The winner doesn't survive an honest count of tries, or is no better than a shuffled null.
Not tradable The effect may exist but can't be captured with our realistic execution.
Edge decayed It worked in one window and failed a later one we fixed in advance.
Works but not worth it It does something real (often smaller drops) but fails its bar or gives up the return.
Passed, unproven live It cleared its pre-registered tests. That is not the same as working with real money.
Untestable Not enough data to test it honestly. We say so instead of inventing a p-value.

What this can't tell you

  • We used 23 liquid ETFs and some other free-data baskets. Results may not carry to single stocks.
  • The data is free-tier and not survivorship-free.
  • Everything is paper. Simulated costs are not a brokerage statement.
  • A failed test says "we couldn't find it with this data and rule", not "can never work".
  • Sharpe ratio = average excess return per unit of volatility. It's a ratio, and it has its own failure modes.

Research and education only. Not investment advice. See Legal.