Quant Analytics: from scratch to pro · Statistics for markets · Advanced
Multiple testing: why the best of many tries looks good
Test enough rules and some will pass a 5% test by luck alone. Corrections such as Bonferroni and Benjamini-Hochberg raise the bar according to how many things were tried, and the same logic is why published market anomalies need a higher hurdle than a t-statistic of 2.
In this lesson
- Compute the chance that at least one of N noise tests looks significant
- Apply a Bonferroni correction and the Benjamini-Hochberg procedure to a list of p-values
- Explain the difference between controlling any false positive and controlling the share of false discoveries
- Translate a t-statistic hurdle into the Sharpe ratio it needs over a given span
Read first: Hypothesis tests and p-values for market data
Give someone a hundred trading rules and a price history, and some of the rules will look good. This is not a defect of any one rule. It is arithmetic. The more things are tried, the more likely the best of them clears a fixed bar by luck. Any statement about the best result is a statement about the maximum of many numbers, and a maximum behaves differently from a single draw.
The arithmetic of many tries
A 5% test on a rule with no real edge falsely passes 5% of the time. Try N unrelated rules and the chance that at least one passes is 1 - 0.95^N.
- 5 tests: 22.6%.
- 20 tests: 64.15%.
- 100 tests: 99.4%.
With 20 tests the expected number of false positives is 1. With 100 tests, one in a hundred noise rules has a t-statistic above 2.58. The chart shows how fast the curve rises. In real research the tries are not tidy: parameters, markets, timeframes, filters and variants each multiply the count, and many are never recorded.
Controlling the error: two goals
Family-wise error is the chance of at least one false positive in the whole set. False discovery rate (FDR) is the expected share of false positives among the results you call discoveries. The first is strict and suits a case where one false finding is costly. The second suits screening many ideas, where you accept some noise among the survivors as long as its share is known.
Bonferroni
Divide the significance level by the number of tests: 0.05 / N. With 20 tests the threshold is 0.25% per test, and the chance of any false positive stays at 4.88% for independent tests. For 100 tests, the t-statistic needs to be 3.48. Bonferroni is simple and safe but conservative, particularly when the tests are correlated, as rule variants usually are.
Benjamini-Hochberg
Benjamini and Hochberg (1995) control the FDR. Sort the m p-values from smallest to largest (the procedure is proved for independent or positively dependent tests). Find the largest rank k with p(k) <= k x q / m, then call the first k results discoveries.
Worked example
Ten sorted p-values: 0.001, 0.008, 0.012, 0.021, 0.034, 0.042, 0.20, 0.35, 0.60, 0.88. Take q = 0.05.
| Rank | p | BH line k x 0.05 / 10 |
Passes |
|---|---|---|---|
| 1 | 0.001 | 0.005 | yes |
| 2 | 0.008 | 0.010 | yes |
| 3 | 0.012 | 0.015 | yes |
| 4 | 0.021 | 0.020 | no |
| 5 | 0.034 | 0.025 | no |
| 6 | 0.042 | 0.030 | no |
- Raw 5% rule: 6 results look significant.
- Bonferroni, threshold 0.005: 1 result.
- Benjamini-Hochberg: 3 results, since rank 3 is the last at or under its line.
Each step is stricter than the one before, and the right one depends on how costly a false find is.
What counts as a try
The correction needs the true number of tests. A researcher who tries 100 variants, keeps 5 and reports 5 has run 100. Dependence between tests lowers the effective number, but it needs to be estimated, not assumed. White (2000) built a bootstrap "reality check" that tests the best of a family of rules against the whole family, so the search itself is accounted for. See measuring an indicator honestly for an application.
A higher hurdle for new findings
Harvey, Liu and Zhu (2016) collected the factors proposed in the research literature to explain returns and applied multiple-testing corrections to them. Their conclusion is that a newly proposed factor should clear a t-statistic of more than 3.0, not 2.0, and that many published claims are likely false. Harvey (2017) argues the same point in his presidential address to the American Finance Association.
What a t-statistic of 3 asks for: since t = SR x sqrt(years), a t-statistic of 3 over ten years needs an annual Sharpe ratio of 0.95. A t-statistic of 2 needs only 0.63. The higher bar removes ideas that were marginal to begin with.
Common mistakes
- Reporting only the winner and forgetting the others.
- Counting tests per paper rather than per idea tried by anyone on the same data.
- Applying Bonferroni to correlated variants of the same rule and concluding nothing is left.
- Treating the BH output as certainty: a share of the survivors are still false.
- Choosing the correction after seeing which one gives the answer wanted.
In practice
Keep a log of every variant tried, including the failures, and divide or adjust by the full count. Prefer a pre-specified idea with a reason behind it to a search through parameters. When a result comes from a search, ask what the test statistic of the best of N noise rules would be, and compare.
Chance of at least one false positive against the number of tests
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| chance of at least one false positive in 20 tests | 1 - 0.95**20 | 64.15% |
| same for 5 tests | 1 - 0.95**5 | 22.6% |
| same for 100 tests | 1 - 0.95**100 | 99.4% |
| expected false positives in 20 tests | 20 * 0.05 | 1 |
| Bonferroni threshold for 20 tests | 0.05 / 20 | 0.25% |
| overall false-positive chance with that threshold, independent tests | 1 - (1 - bonf20)**20 | 4.88% |
| t-statistic needed for p = 0.05/100, two-sided | phi_inv(1 - 0.05 / 100 / 2) | 3.48 |
| t-statistic that 1 in 100 noise tests exceeds | phi_inv(1 - 0.01 / 2) | 2.58 |
| Benjamini-Hochberg line at rank 3 of 10 | 3 * 0.05 / 10 | 0.015 |
| Benjamini-Hochberg line at rank 4 of 10 | 4 * 0.05 / 10 | 0.02 |
| Bonferroni threshold for 10 tests | 0.05 / 10 | 0.005 |
| Sharpe ratio for a t-statistic of 3 over 10 years | 3 / sqrt(10) | 0.95 |
| Sharpe ratio for a t-statistic of 2 over 10 years | 2 / sqrt(10) | 0.63 |
Check yourself
You test 20 unrelated rules on pure noise at the 5% level. What is the chance that at least one looks significant?
1 - 0.95^20 = 64.15%. The expected number of false positives is 20 x 0.05 = 1.
What is the Bonferroni threshold for 20 tests at an overall 5% level?
0.05 / 20 = 0.25% per test. It keeps the chance of any false positive near 5% (for independent tests, 4.88%).
Ten sorted p-values are 0.001, 0.008, 0.012, 0.021, 0.034, 0.042, 0.20, 0.35, 0.60, 0.88. How many pass at the raw 5% level, under Bonferroni, and under Benjamini-Hochberg at 5%?
Raw: 6. Bonferroni (0.005): 1. Benjamini-Hochberg: 3, because p(3) = 0.012 is at or below 3 x 0.05 / 10 = 0.015, while p(4) = 0.021 is above 0.02.
Over ten years, what annual Sharpe ratio gives a t-statistic of 3?
t = SR x sqrt(years), so SR = 3 / sqrt(10) = 0.95. A t-statistic of 2 needs 0.63.
Key terms
Sources
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society, Series B 57(1), 289-300 (Yoav Benjamini, Yosef Hochberg), 1995. Accessed 9 Nov 2026
- ... and the Cross-Section of Expected Returns. The Review of Financial Studies 29(1), 5-68 (Campbell R. Harvey, Yan Liu, Heqing Zhu), 2016. Accessed 9 Nov 2026
- A Reality Check for Data Snooping. Econometrica 68(5), 1097-1126 (Halbert White), 2000. Accessed 9 Nov 2026
- Why Most Published Research Findings Are False. PLoS Medicine 2(8), e124 (John P. A. Ioannidis), 2005. Accessed 9 Nov 2026
- Presidential Address: The Scientific Outlook in Financial Economics. The Journal of Finance 72(4), 1399-1440 (Campbell R. Harvey), 2017. Accessed 9 Nov 2026