1,192 indicator combinations: the winner was no better than luck
Failure type: overfit. Verdict: nothing promoted.
The idea
Combine up to three indicators into a ranking score, hold the top 5 or 8 ETFs, rebalance every 5 or 21 sessions. Search everything.
What we did
- 12 indicators, all 1–3 indicator subsets, 2 rebalance speeds, 2 portfolio sizes: 1,192 configurations.
- Choose on training years only (2006–2016). Validation (2017–2021) was reported, not used to choose.
- A null: run 200 complete searches on time-shifted indicators and see what the best of each looks like.
- Deflated Sharpe and probability of backtest overfitting on the whole set.
What happened
| Result | |
|---|---|
| Best on training years | Sharpe 0.744 (a three-indicator combo, rebalanced every 5 sessions) |
| Typical config on training | median Sharpe 0.50; 90th percentile 0.61 |
| Best of a scrambled-data search | median 0.83; 95th percentile 1.03 |
| How often scrambled data beat our winner | empirical p = 0.73 |
| Deflated Sharpe (1,192 tries) | 0.910 (bar 0.95) |
| Probability of backtest overfitting | 0.78 (bar 0.20) |
Rebuilt in a realistic engine (next-open fills, settled cash, costs), the finalist got Sharpe 0.70 on training and 0.77 on 2017–2021, about 9.2% a year against SPY's 18.2% in that window. It traded about 41–45 times its capital a year.
Why it failed
Picking the training winner added nothing out of sample: the top ten on training averaged 0.77 on validation, against a 0.76 median for all configurations. Searching more than a thousand combinations on 23 funds finds something that looks good, and that is all it did.
Receipts: 1,192 distinct configurations were recorded in our trial registry.
Research and education only. These are paper results from historical data on a limited set of funds. Not investment advice, not a recommendation, not a prediction. See Legal.