Quant Analytics: from scratch to pro · Backtesting done right · Advanced
Overfitting, trial counts and the deflated Sharpe ratio
Try enough variants of a rule and the best one looks good even when none has an edge. The expected best Sharpe ratio of N unskilled trials, and the deflated Sharpe ratio built on it, put a number on how much of a result is selection.
In this lesson
- Explain why the best of N trials is biased upward
- Compute the expected maximum Sharpe ratio of N unskilled trials
- Compute a deflated Sharpe ratio and read it as a probability
- Describe the probability of backtest overfitting and why trial counts must be recorded
Read first: What a backtest is, and what it is not, Multiple testing: why the best of many tries looks good
Every backtest answers the question asked of one rule. Most research asks it of many: other lookbacks, other markets, other thresholds. The best answer among many is biased upward, and the bias grows with the number tried. This is overfitting by selection.
Why the best of many looks good
Take 20 rules with no edge and test each at the 5% level. The chance that at least one passes is 1 - 0.95^20, which is 64.15%. A pass is the normal outcome of the search. Multiple testing covers the general case. Harvey, Liu and Zhu argue that, given the number of factors tried in the literature, a new claim should clear a t-statistic of 3.0, not 2.0. White's reality check and the work of Sullivan, Timmermann and White test whether the best rule beats a benchmark after accounting for the whole search.
The expected best Sharpe ratio of N trials
The Sharpe ratio is mean excess return divided by its standard deviation. Estimated from a sample, it carries noise. Over T periods, with zero true skill, the estimate has a standard deviation of about 1 / sqrt(T) in per-period units.
For N independent trials with true Sharpe ratio zero, Bailey and López de Prado (2014) give the approximate expected maximum:
E[max SR] = sd x ((1 - g) x phi_inv(1 - 1/N) + g x phi_inv(1 - 1/(N x e)))
Here sd is the standard deviation of the trial Sharpe estimates, g is the Euler-Mascheroni constant (0.5772), and phi_inv is the standard normal quantile. The result is in per-period units, so annualise by multiplying by sqrt(12) for monthly data.
With 5 years of monthly data (60 months), sd is 0.129 a month. The table gives the best annual Sharpe ratio that luck alone is expected to produce:
| Trials N | Expected best annual Sharpe |
|---|---|
| 10 | 0.70 |
| 100 | 1.13 |
| 1000 | 1.46 |
A longer sample helps. With 20 years instead of 5, the noise halves, and the best of 100 falls to 0.57.
The deflated Sharpe ratio
The probabilistic Sharpe ratio is the probability that the true Sharpe ratio exceeds a benchmark SR*, given the sample length and the skewness g3 and raw kurtosis g4 of returns (3 for a normal distribution):
PSR = phi((SR - SR*) x sqrt(T - 1) / sqrt(1 - g3 x SR + (g4 - 1)/4 x SR^2))
The deflated Sharpe ratio is this probability with SR* set to the expected best Sharpe ratio of the N trials. It asks whether the winner beats what selection alone would produce.
Worked example
A rule shows an annual Sharpe ratio of 1.2 over 60 months, which is 0.346 a month. Returns have skew -0.5 and kurtosis 5 (raw kurtosis, where a normal distribution has 3). The adjustment term is 1.137.
One trial. Against zero, the z-score is 2.34 and the probability is 99.0%. This looks convincing.
One hundred trials. The rule was the best of 100. The bracket in the formula is 2.53 for N = 100, so the benchmark becomes 0.129 x 2.53 = 0.327 a month, or 1.13 a year. The z-score is (0.346 - 0.327) x sqrt(59) / 1.137, which is 0.133, and the deflated Sharpe ratio is 55.3%. That is barely better than a coin toss. The same number moved from convincing to doubtful with one fact: how many were tried.
The probability of backtest overfitting
A related test splits the data into many blocks, picks the best configuration in some blocks, and checks its rank in the others. The probability of backtest overfitting is the share of splits in which the in-sample winner ranks below the median out of sample. A high value says that picking the best in sample does not choose a good rule.
Common mistakes
- Not counting trials. Every variant, parameter value and market tried counts, including those dropped.
- Treating correlated trials as independent. Ten near-identical variants are fewer than ten trials. The effective N is lower, and it is hard to estimate.
- Using the formula on a sample too short for the skew and kurtosis to be measured.
- Annualising before applying the formula. Use per-period values for
SRandT. - Reading the deflated Sharpe ratio as proof. It corrects one bias; leaks and costs still need checking.
In practice
Keep a log of every trial run, with its settings and result, and report N next to the best result. Hold back data that was never used in the search (see walk-forward and the locked holdout). Research on FinUties is where studies built on methods like these are published, with their method and limits stated.
Expected best annual Sharpe ratio of N trials with no skill (5 years)
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| chance that at least one of 20 null rules passes at 5% | 1 - 0.95**20 | 64.15% |
| sample length in months | 60 | 60 months |
| noise of a monthly Sharpe estimate under zero skill | 1 / sqrt(t_months) | 0.129 |
| Euler-Mascheroni constant | 0.5772156649 | 0.5772 |
| expected maximum of 100 standard normal trials | (1 - euler) * phi_inv(1 - 1/100) + euler * phi_inv(1 - 1/(100 * e)) | 2.53 |
| expected best monthly Sharpe of 100 trials | sd_m * br100 | 0.327 |
| the same, annualised | sr0_m * sqrt(12) | 1.13 |
| observed annual Sharpe ratio | 1.2 | 1.2 |
| observed monthly Sharpe ratio | sr_a / sqrt(12) | 0.346 |
| non-normality adjustment, skew -0.5, kurtosis 5 | sqrt(1 - (-0.5) * sr_m + (5 - 1) / 4 * sr_m**2) | 1.137 |
| deflated Sharpe z-score | (sr_m - sr0_m) * sqrt(t_months - 1) / adj | 0.133 |
| deflated Sharpe ratio | phi(z_dsr) | 55.3% |
| z-score against zero, one trial | sr_m * sqrt(t_months - 1) / adj | 2.34 |
| probabilistic Sharpe ratio against zero | phi(z_psr) | 99.0% |
| expected best annual Sharpe, 10 trials, 5 years | sd_m * sqrt(12) * ((1 - euler) * phi_inv(1 - 1/10) + euler * phi_inv(1 - 1/(10 * e))) | 0.70 |
| expected best annual Sharpe, 1000 trials, 5 years | sd_m * sqrt(12) * ((1 - euler) * phi_inv(1 - 1/1000) + euler * phi_inv(1 - 1/(1000 * e))) | 1.46 |
| expected best annual Sharpe, 100 trials, 20 years | br100 / sqrt(20) | 0.57 |
Check yourself
Twenty independent rules with no edge are each tested at the 5% level. What is the chance that at least one looks significant?
1 - 0.95^20 = 64.15%. A significant-looking rule is the expected outcome of searching, not a discovery.
Why does a longer sample reduce the expected best Sharpe ratio of unskilled trials?
The noise in an estimated Sharpe ratio falls with the square root of the sample length. With 20 years instead of 5, the noise halves, and so does the best of 100 trials: 0.57 against 1.13.
A strategy shows an annual Sharpe ratio of 1.2 over 5 years. It was picked from 100 trials. What does the deflated Sharpe ratio say?
The benchmark rises from 0 to 1.13 and the probability that the true Sharpe ratio is above it is 55.3%. The same result from a single trial gives 99.0%.
A researcher reports only the best of many variants, and does not say how many. What is missing?
The trial count N, which sets the benchmark. Without it, no one can deflate the result. Count every variant, including those discarded.
Key terms
OverfittingDeflated Sharpe ratioSharpe ratioMultiple testing
Checked facts
Every figure below carries how it was checked: Primary checked (read in the cited primary, official, academic or data source), Two sources agree, or Approximate (an order of magnitude only).
- Primary checked
Harvey, Liu and Zhu argue that a newly claimed factor should clear a t-statistic of 3.0 rather than the conventional 2.0, given how many factors have been tried.
Sources
- The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality. The Journal of Portfolio Management 40(5), 94-107 (David H. Bailey, Marcos López de Prado), 2014. Accessed 9 Nov 2026
- The Probability of Backtest Overfitting. Journal of Computational Finance 20(4), 39-69 (Bailey, Borwein, López de Prado, Zhu), 2017. Accessed 9 Nov 2026
- ... and the Cross-Section of Expected Returns. The Review of Financial Studies 29(1), 5-68 (Campbell R. Harvey, Yan Liu, Heqing Zhu), 2016. Accessed 9 Nov 2026
- A Reality Check for Data Snooping. Econometrica 68(5), 1097-1126 (Halbert White), 2000. Accessed 9 Nov 2026
- Data-Snooping, Technical Trading Rule Performance, and the Bootstrap. The Journal of Finance 54(5), 1647-1691 (Ryan Sullivan, Allan Timmermann, Halbert White), 1999. Accessed 9 Nov 2026