Skip to main content

Quant Analytics: from scratch to pro · Statistics for markets · Intermediate

Hypothesis tests and p-values for market data

A p-value measures how surprising a result would be if nothing were going on. It does not say how likely the idea is to be true, and for market data it is easy to misread, overstate and mis-size.

4 min read Reviewed Revision 1

In this lesson

  • State a null hypothesis and compute a t-statistic and two-sided p-value for a mean return
  • Say what a p-value does and does not mean
  • Explain statistical power and why most tests of small edges are underpowered
  • Estimate how likely a significant result is to be real, given a base rate

Read first: How sure is an average? Standard error and confidence intervals

Two-sided p-value against the t-statistic

0% 20% 40% 60% 80% 100% 00.751.52.2534 t-statistic
Two-sided p-value. Computed from the normal distribution, not from market data. A t-statistic of 2 corresponds to a p-value near 5%; a t-statistic of 3 to about 0.3%.

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
standard error of the daily mean, 2,520 days at 1% volatility0.01 / sqrt(2520)0.0199%
t-statistic of a 0.03% daily mean0.0003 / se_d1.51
two-sided p-value2 * (1 - phi(t_d))13.2%
two-sided p-value at t = 2.02 * (1 - phi(2.0))4.55%
two-sided p-value at t = 3.02 * (1 - phi(3.0))0.27%
effective observations with autocorrelation 0.12520 * (1 - 0.1) / (1 + 0.1)2,062
t-statistic with the effective count0.0003 / (0.01 / sqrt(neff))1.36
p-value with the effective count2 * (1 - phi(t_adj))17.3%
chance of rejecting zero when the true t-statistic is 1.511 - phi(1.96 - t_d) + phi(-1.96 - t_d)33%
days for 80% power((1.96 + phi_inv(0.8)) * 0.01 / 0.0003)**28,721
years for 80% power at 252 days a yearn_power / 25234.6
share of significant results that are real0.1 * 0.8 / (0.1 * 0.8 + 0.9 * 0.05)64%
chance at least one of 20 independent noise tests reaches p < 5%1 - 0.95**2064%

Check yourself

A mean has a t-statistic of 2.0. What is the two-sided p-value, using the normal approximation?

2 x (1 - phi(2.0)) = 4.55%. About 1 time in 22 a mean this far from zero arises by chance alone if the true mean is zero.

A p-value of 0.03 is found. Is there a 97% chance the rule works?

No. It says that if the true mean were zero, a result this extreme would occur 3% of the time. It is not the probability that the idea is true.

Why can a test fail to find a real edge?

Low power. With a true t-statistic of about 1.5 the test rejects zero only about a third of the time, so a real edge is usually missed in a short sample.

10% of ideas tried are real, tests have 80% power and a 5% significance level. What share of significant results are real?

0.1 x 0.8 / (0.1 x 0.8 + 0.9 x 0.05) = 64%. More than a third of the 'discoveries' are noise.

Key terms

P-valueT-statisticStandard errorConfidence interval

Sources

Informational content. Not investment advice.