Skip to main content

Quant Analytics: from scratch to pro · Statistics for markets · Advanced

Multiple testing: why the best of many tries looks good

Test enough rules and some will pass a 5% test by luck alone. Corrections such as Bonferroni and Benjamini-Hochberg raise the bar according to how many things were tried, and the same logic is why published market anomalies need a higher hurdle than a t-statistic of 2.

4 min read Reviewed Revision 1

In this lesson

  • Compute the chance that at least one of N noise tests looks significant
  • Apply a Bonferroni correction and the Benjamini-Hochberg procedure to a list of p-values
  • Explain the difference between controlling any false positive and controlling the share of false discoveries
  • Translate a t-statistic hurdle into the Sharpe ratio it needs over a given span

Read first: Hypothesis tests and p-values for market data

Chance of at least one false positive against the number of tests

0% 20% 40% 60% 80% 100% 138153050100 Number of independent tests on noise
Chance of at least one p < 5%. Computed from the formula for independent tests, not from market data. With 14 tests the chance is already above 50%.

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
chance of at least one false positive in 20 tests1 - 0.95**2064.15%
same for 5 tests1 - 0.95**522.6%
same for 100 tests1 - 0.95**10099.4%
expected false positives in 20 tests20 * 0.051
Bonferroni threshold for 20 tests0.05 / 200.25%
overall false-positive chance with that threshold, independent tests1 - (1 - bonf20)**204.88%
t-statistic needed for p = 0.05/100, two-sidedphi_inv(1 - 0.05 / 100 / 2)3.48
t-statistic that 1 in 100 noise tests exceedsphi_inv(1 - 0.01 / 2)2.58
Benjamini-Hochberg line at rank 3 of 103 * 0.05 / 100.015
Benjamini-Hochberg line at rank 4 of 104 * 0.05 / 100.02
Bonferroni threshold for 10 tests0.05 / 100.005
Sharpe ratio for a t-statistic of 3 over 10 years3 / sqrt(10)0.95
Sharpe ratio for a t-statistic of 2 over 10 years2 / sqrt(10)0.63

Check yourself

You test 20 unrelated rules on pure noise at the 5% level. What is the chance that at least one looks significant?

1 - 0.95^20 = 64.15%. The expected number of false positives is 20 x 0.05 = 1.

What is the Bonferroni threshold for 20 tests at an overall 5% level?

0.05 / 20 = 0.25% per test. It keeps the chance of any false positive near 5% (for independent tests, 4.88%).

Ten sorted p-values are 0.001, 0.008, 0.012, 0.021, 0.034, 0.042, 0.20, 0.35, 0.60, 0.88. How many pass at the raw 5% level, under Bonferroni, and under Benjamini-Hochberg at 5%?

Raw: 6. Bonferroni (0.005): 1. Benjamini-Hochberg: 3, because p(3) = 0.012 is at or below 3 x 0.05 / 10 = 0.015, while p(4) = 0.021 is above 0.02.

Over ten years, what annual Sharpe ratio gives a t-statistic of 3?

t = SR x sqrt(years), so SR = 3 / sqrt(10) = 0.95. A t-statistic of 2 needs 0.63.

Key terms

Multiple testingFalse discovery rateP-valueT-statistic

Sources

Informational content. Not investment advice.