If you want the bookkeeping behind the tests, see How we test. For a general walkthrough of overfitting, see Investopedia’s explainer (external).

Sharpe ratio

The Sharpe ratio is return per unit of bumpiness. You take a strategy's yearly return above T-bills and divide it by how much its returns swing around (that swing is often called volatility). A Sharpe of 1.0 means you earned one unit of extra return for each unit of risk you carried.

Rough scale for a yearly Sharpe on a simple daily strategy:

  • Below 0.3: basically noise. You'd do as well holding cash.
  • 0.3 to 0.7: ordinary. The stock market over long periods sits around here.
  • 0.7 to 1.0: good. In a calm stretch, plain buy-and-hold can land here too.
  • Above 1.0 for years: rare, and worth checking hard.
  • Above 2.0 on daily ETF data: almost always a bug or overfitting.

Our own results show the range. Plain SPY scored 0.92 in 2017–2021, which is the bar a lot of our tests had to beat. Weekly reversal looked fine on the training years, then scored only 0.02–0.11 in that later window. The best of 32 trend variants reached 0.70, and the best of 28 indicator families reached 0.62 — both on the training years, both above SPY's 0.41 in those same years, and both still failed our overfitting checks. The drawdown brake scored 1.00 in 2017–2021 against 0.97 for the plain 200-day rule (SPY itself was 0.92 in that window). It earned about 8% a year against 18% for SPY; its real benefit was shallower drops, not extra return.

One catch: a single Sharpe number says nothing about how many tries it took to find it. That is why we also report the deflated Sharpe and the probability of overfitting.

Drawdown

A drawdown is the fall from a peak to the lowest point that followed, before the account made a new high. When we say worst drop, we mean the largest of those falls in the window. A strategy can have a fine average return and still be hard to live with if the worst drop is huge.

CAGR (yearly return)

CAGR stands for compound annual growth rate. It is the constant yearly rate that would take you from the starting value to the ending value. On this site we usually call it yearly return. It is a useful summary of the whole window, not a promise that every year looked like that.

Training window

The training window is the earlier stretch of history where we are allowed to search, compare variants, and tune. On this site that is 2005–2016. Some tests start in 2006 once a long lookback, such as a 200-day average, needs time to fill in. A result that only looks good here has not yet been tested on years we didn't peek at.

Validation window

The validation window is a later stretch of history we set aside in advance to judge a rule we already wrote down. We do not get to retune after seeing it.

On this site there are two such windows:

  • 2017–2021, a strongly trending stock market.
  • 2022–2023, when stocks and bonds fell together.

If an idea works in one and fails in the other, we do not call it confirmed.

Pre-registration

Pre-registration means we write the rule, the years, and what counts as a pass into a file and save it before the test runs. Our runner refuses to start if that file has unsaved changes. If we change the rule after seeing results, that is a different idea, and the page says so.

Holdout

A holdout is data you do not use to invent or tweak the rule, so the test can be fair. Our reserved slice — from 2 January 2024 on — has not been used for any published result. (Honest footnote: a few early demo runs, before we enforced this, did see some of it.) The validation windows are also a kind of holdout, used only after the rule is written down.

Deflated Sharpe

The deflated Sharpe (sometimes called DSR) is a Sharpe ratio that has been marked down for how many ideas you tried. A high Sharpe is easier to get by chance when you search a lot. The number asks how believable the best result still is after that search. Our bar is 0.95. Below that, we do not treat the winner as proven. This measure comes from Bailey and López de Prado (2014).

Probability of overfitting

The probability of backtest overfitting (sometimes called PBO) is a check for a familiar failure: you pick the winner of a big search, and it falls apart later. It asks how often the in-sample champion is only average, or worse, on data it didn't see. Lower is better. Our usual bar is 0.20. For one family we set a looser 0.50 bar in advance, and it still failed.

The paper behind this test is Bailey, Borwein, López de Prado and Zhu, The Probability of Backtest Overfitting (PDF, Journal of Computational Finance, 2017).

p-value

A p-value is a rough answer to “how often would luck alone look this good?” A low number is more surprising. It is not the chance that the idea is true. It also gets shaky if you ran many tests, because luck has more chances to look clever. That is why we often pair it with false-discovery control.

False-discovery control

When you run many tests, some will look like winners by chance. False-discovery control (we use Benjamini and Hochberg, 1995) is a way to keep those accidental winners in check. The related q-value is a false-discovery rate: a rough “if we call this a winner, how often would that be a mistake?” We often use a bar of 0.10. Missing that bar, even closely, still means the idea did not pass.

Exposure-matched benchmark

An exposure-matched benchmark is a fairer comparison. One version holds SPY only on the days the strategy is invested, and sits in T-bills the rest of the time. Another scales SPY to the same bumpiness as the strategy. A rule that’s in cash half the time can look “safer” than buy-and-hold for free if you only compare it to a fully invested market.

T-bills and cash

T-bills are short-term U.S. government bills. When a rule is out of the market, we credit idle cash at the 3-month T-bill rate (from the Fed’s DTB3 series), not zero. Sitting in cash is not “earning nothing,” and it also means you miss whatever the market did on those days.

Overfit

We use overfit when the winner doesn’t hold up once you count how many ideas you tried, or when it is no better than luck — including a test on scrambled or time-shifted data. A pretty line from a big search is the usual example.

Not tradable

We use not tradable when the effect may be real, but we couldn’t have captured it with our realistic trading rules. We decide at the close and fill at the next open, with next-day cash settlement, like a retirement account. An overnight jump you can only see after it has happened is the usual example.

Edge decayed

We use edge decayed when an idea worked in one period and failed a later window we had already fixed in advance. We do not retune after seeing that. The training years loved it; the test years did not.

Works but not worth it

We use works but not worth it when the idea does something real (often smaller drops) but misses a bar we already set, or gives up too much of the return to get there. Cutting the pain is not the same as being worth the trade-off we said we’d accept.

Passed, unproven live

We use passed, unproven live when an idea cleared the tests we wrote down in advance. Paper results are not a brokerage statement. It has not been proven with real money, and the caveats on that page still apply.

Untestable

We use untestable when there isn’t enough data to test the idea honestly. We would rather say that than invent a p-value for a chart that only makes a handful of calls.

Research and education only. Not investment advice. See Legal.