Quant Analytics: from scratch to pro · Statistics for markets · Intermediate
Hypothesis tests and p-values for market data
A p-value measures how surprising a result would be if nothing were going on. It does not say how likely the idea is to be true, and for market data it is easy to misread, overstate and mis-size.
In this lesson
- State a null hypothesis and compute a t-statistic and two-sided p-value for a mean return
- Say what a p-value does and does not mean
- Explain statistical power and why most tests of small edges are underpowered
- Estimate how likely a significant result is to be real, given a base rate
Read first: How sure is an average? Standard error and confidence intervals
A rule earned a positive average return in the past. Was that skill, or the kind of result noise produces on its own? A hypothesis test is the standard way to put a number on that question. It is also one of the most misread tools in research.
The set-up
Start with a null hypothesis, the dull explanation: the true mean return is zero. The alternative is that it is not. Compute the t-statistic, t = mean / SE (see standard error), and ask how often a t-statistic this far from zero would occur if the null were true. That frequency is the p-value.
For large samples the t-statistic is close to normal, so p = 2 x (1 - phi(|t|)), where phi is the standard normal cumulative distribution. Small p means the data are hard to explain by chance alone.
Worked example
Ten years of daily returns, 2,520 days. The mean is 0.03% a day and the daily volatility is 1%.
- Standard error:
0.01 / sqrt(2520)= 0.0199%. - t-statistic:
0.0003 / 0.000199= 1.51. - Two-sided p-value: 13.2%.
At the usual 5% level the null is not rejected. The result is quite compatible with a true mean of zero.
For reference, a t-statistic of 2.0 gives a p-value of 4.55%, and 3.0 gives 0.27%.
What a p-value is not
It is the probability of data this extreme if the null is true. It is not:
- the probability that the null is true,
- the probability that the rule works,
- a measure of how large or useful the effect is.
The American Statistical Association's 2016 statement on p-values says the same: a p-value does not measure the probability that the hypothesis is true, and a conclusion should not rest on whether a result crosses a threshold alone. A tiny effect can have a tiny p-value in a large sample, and a large effect can have a large one in a small sample. Report the size of the effect and its interval together.
Market data bends the assumptions
Returns are not independent coin flips. If daily returns have an autocorrelation of 0.1, the effective number of observations is about n x (1 - rho) / (1 + rho), which is 2,062 instead of 2,520. The t-statistic falls to 1.36 and the p-value rises to 17.3%. Ignoring the correlation makes results look better than they are. Heavy tails and changing volatility have a similar effect, and robust standard errors or the bootstrap are the usual repairs.
Power: the other error
A test can wrongly reject a true null (a false positive) or fail to reject a false one (a miss). Power is the chance of detecting an effect that is real.
If the true t-statistic of the edge in the example is 1.51, a 5% test rejects zero only about 33% of the time. To have 80% power for a mean of 0.03% a day takes about 8,721 days, or 34.6 years. Small edges in noisy data are close to undetectable, so a non-significant result usually means "not enough data", not "no edge".
Significant does not mean real
Suppose 10% of the ideas you test are real, your tests have 80% power and you use a 5% level. Of all significant results, 0.1 x 0.8 / (0.1 x 0.8 + 0.9 x 0.05) = 64% are real. More than a third are noise, and this is before any data-snooping. Ioannidis (2005) makes this argument for published research findings in general. The fewer true effects there are to find, the worse the ratio gets.
Run 20 independent tests on pure noise and the chance that at least one gives p below 5% is 64%. This is the subject of multiple testing.
Common mistakes
- Reading 1 minus the p-value as the probability the rule works.
- Treating p = 0.049 as a discovery and p = 0.051 as nothing.
- Concluding "no edge" from a non-significant result in a short sample.
- Ignoring autocorrelation and overlapping returns when computing the standard error.
- Reporting only the tests that worked.
In practice
Write the null and the test before looking at the data. Report the estimate, its interval, the p-value, the number of observations and the number of things tried. Treat p-values as one input next to the economics of the idea and the size of the effect. Research on FinUties is where studies built on methods like these are published, with their method and limits stated.
Two-sided p-value against the t-statistic
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| standard error of the daily mean, 2,520 days at 1% volatility | 0.01 / sqrt(2520) | 0.0199% |
| t-statistic of a 0.03% daily mean | 0.0003 / se_d | 1.51 |
| two-sided p-value | 2 * (1 - phi(t_d)) | 13.2% |
| two-sided p-value at t = 2.0 | 2 * (1 - phi(2.0)) | 4.55% |
| two-sided p-value at t = 3.0 | 2 * (1 - phi(3.0)) | 0.27% |
| effective observations with autocorrelation 0.1 | 2520 * (1 - 0.1) / (1 + 0.1) | 2,062 |
| t-statistic with the effective count | 0.0003 / (0.01 / sqrt(neff)) | 1.36 |
| p-value with the effective count | 2 * (1 - phi(t_adj)) | 17.3% |
| chance of rejecting zero when the true t-statistic is 1.51 | 1 - phi(1.96 - t_d) + phi(-1.96 - t_d) | 33% |
| days for 80% power | ((1.96 + phi_inv(0.8)) * 0.01 / 0.0003)**2 | 8,721 |
| years for 80% power at 252 days a year | n_power / 252 | 34.6 |
| share of significant results that are real | 0.1 * 0.8 / (0.1 * 0.8 + 0.9 * 0.05) | 64% |
| chance at least one of 20 independent noise tests reaches p < 5% | 1 - 0.95**20 | 64% |
Check yourself
A mean has a t-statistic of 2.0. What is the two-sided p-value, using the normal approximation?
2 x (1 - phi(2.0)) = 4.55%. About 1 time in 22 a mean this far from zero arises by chance alone if the true mean is zero.
A p-value of 0.03 is found. Is there a 97% chance the rule works?
No. It says that if the true mean were zero, a result this extreme would occur 3% of the time. It is not the probability that the idea is true.
Why can a test fail to find a real edge?
Low power. With a true t-statistic of about 1.5 the test rejects zero only about a third of the time, so a real edge is usually missed in a short sample.
10% of ideas tried are real, tests have 80% power and a 5% significance level. What share of significant results are real?
0.1 x 0.8 / (0.1 x 0.8 + 0.9 x 0.05) = 64%. More than a third of the 'discoveries' are noise.
Key terms
Sources
- The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician 70(2), 129-133 (Ronald L. Wasserstein, Nicole A. Lazar), 2016. Accessed 9 Nov 2026
- Why Most Published Research Findings Are False. PLoS Medicine 2(8), e124 (John P. A. Ioannidis), 2005. Accessed 9 Nov 2026
- The Econometrics of Financial Markets. Princeton University Press (John Y. Campbell, Andrew W. Lo, A. Craig MacKinlay), 1997. Accessed 9 Nov 2026