Skip to main content

Quant Analytics: from scratch to pro · Backtesting done right · Advanced

Overfitting, trial counts and the deflated Sharpe ratio

Try enough variants of a rule and the best one looks good even when none has an edge. The expected best Sharpe ratio of N unskilled trials, and the deflated Sharpe ratio built on it, put a number on how much of a result is selection.

4 min read Reviewed Revision 1

In this lesson

  • Explain why the best of N trials is biased upward
  • Compute the expected maximum Sharpe ratio of N unskilled trials
  • Compute a deflated Sharpe ratio and read it as a probability
  • Describe the probability of backtest overfitting and why trial counts must be recorded

Read first: What a backtest is, and what it is not, Multiple testing: why the best of many tries looks good

Expected best annual Sharpe ratio of N trials with no skill (5 years)

0.00 0.25 0.50 0.75 1.00 1.25 1.50 210502001000 Number of independent trials N
Expected best annual Sharpe ratio. Computed from the approximation in Bailey and López de Prado (2014) for independent trials whose true Sharpe ratio is zero, with 60 months of data. A real result must beat this line, not zero.

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
chance that at least one of 20 null rules passes at 5%1 - 0.95**2064.15%
sample length in months6060 months
noise of a monthly Sharpe estimate under zero skill1 / sqrt(t_months)0.129
Euler-Mascheroni constant0.57721566490.5772
expected maximum of 100 standard normal trials(1 - euler) * phi_inv(1 - 1/100) + euler * phi_inv(1 - 1/(100 * e))2.53
expected best monthly Sharpe of 100 trialssd_m * br1000.327
the same, annualisedsr0_m * sqrt(12)1.13
observed annual Sharpe ratio1.21.2
observed monthly Sharpe ratiosr_a / sqrt(12)0.346
non-normality adjustment, skew -0.5, kurtosis 5sqrt(1 - (-0.5) * sr_m + (5 - 1) / 4 * sr_m**2)1.137
deflated Sharpe z-score(sr_m - sr0_m) * sqrt(t_months - 1) / adj0.133
deflated Sharpe ratiophi(z_dsr)55.3%
z-score against zero, one trialsr_m * sqrt(t_months - 1) / adj2.34
probabilistic Sharpe ratio against zerophi(z_psr)99.0%
expected best annual Sharpe, 10 trials, 5 yearssd_m * sqrt(12) * ((1 - euler) * phi_inv(1 - 1/10) + euler * phi_inv(1 - 1/(10 * e)))0.70
expected best annual Sharpe, 1000 trials, 5 yearssd_m * sqrt(12) * ((1 - euler) * phi_inv(1 - 1/1000) + euler * phi_inv(1 - 1/(1000 * e)))1.46
expected best annual Sharpe, 100 trials, 20 yearsbr100 / sqrt(20)0.57

Check yourself

Twenty independent rules with no edge are each tested at the 5% level. What is the chance that at least one looks significant?

1 - 0.95^20 = 64.15%. A significant-looking rule is the expected outcome of searching, not a discovery.

Why does a longer sample reduce the expected best Sharpe ratio of unskilled trials?

The noise in an estimated Sharpe ratio falls with the square root of the sample length. With 20 years instead of 5, the noise halves, and so does the best of 100 trials: 0.57 against 1.13.

A strategy shows an annual Sharpe ratio of 1.2 over 5 years. It was picked from 100 trials. What does the deflated Sharpe ratio say?

The benchmark rises from 0 to 1.13 and the probability that the true Sharpe ratio is above it is 55.3%. The same result from a single trial gives 99.0%.

A researcher reports only the best of many variants, and does not say how many. What is missing?

The trial count N, which sets the benchmark. Without it, no one can deflate the result. Count every variant, including those discarded.

Key terms

OverfittingDeflated Sharpe ratioSharpe ratioMultiple testing

Checked facts

Every figure below carries how it was checked: Primary checked (read in the cited primary, official, academic or data source), Two sources agree, or Approximate (an order of magnitude only).

  • Primary checked

    Harvey, Liu and Zhu argue that a newly claimed factor should clear a t-statistic of 3.0 rather than the conventional 2.0, given how many factors have been tried.

    As of 1 Jan 2016. Source: ... and the Cross-Section of Expected Returns.

Sources

Informational content. Not investment advice.