Quant Analytics: from scratch to pro · Professional practice and failure case studies · Intermediate
The research process: question, hypothesis, test, decision
Good quantitative research runs in a fixed order: a question, a hypothesis written down, a test whose pass mark is set in advance, and a decision that includes the option to stop. The order protects against fooling yourself.
In this lesson
- Turn a loose idea into a question, a hypothesis and a pre-set pass mark
- Estimate how much data a test needs before it can tell a small edge from zero
- Use a prior probability to judge how much a single positive result is worth
- Write kill criteria that end a line of work without argument
Read first: Reading a research claim: ten questions to ask
Research without a process produces results that feel convincing and are not. The cure is not more cleverness but a fixed order of steps, each one written down before the next begins. The order is: question, hypothesis, test, decision.
Step 1: a question that can be answered
"Does momentum work?" cannot be answered. "Do equity indices that rose most over the past 12 months beat the others over the next month, after costs, in this period and these markets?" can. A usable question names the universe, the horizon, the measure and the comparison.
Step 2: a hypothesis and a reason
State the hypothesis as something that can fail: "The top third by past return earns more per month than the bottom third." Add the reason it should be true and who is on the other side of the trade. A pattern with no reason is more likely to be noise. This also settles which result would count as a failure.
Step 3: fix the test before running it
Write down, before looking at any results:
- the data, dates and any data held back;
- the exact definitions and the parameters you will use;
- the measure and the pass mark, for example a t-statistic above a stated level after costs;
- the kill criteria: the results that end the work.
Writing this down is the core of pre-registration. It fixes the number of tries and stops the pass mark moving to fit the data (Nosek et al., 2018).
Worked example: how much is one pass worth?
Suppose a test passes a real edge 80% of the time (its power) and passes a useless idea 5% of the time. Suppose 10% of the ideas you test are truly good. Out of 1,000 ideas, 100 are good and 80 of them pass. Of the 900 useless ideas, 5% pass, which is 45. So 80 of the 125 passes are real: 0.8 x 0.10 / (0.8 x 0.10 + 0.05 x 0.90), which is 64%.
Now let only 2% of ideas be truly good. The same test gives 0.8 x 0.02 / (0.8 x 0.02 + 0.05 x 0.98), which is 24.6%. Most passes are now false. The same logic sits behind Ioannidis (2005): a positive result is worth less when the prior is low and many ideas are tried. This is the base rate of good ideas.
Worked example: how much data?
For independent returns the t-statistic is about SR x sqrt(years), where SR is the annual Sharpe ratio. To reach t = 2 you need years = (2 / SR)^2. An edge with a Sharpe ratio of 1.0 needs 4 years. An edge of 0.5 needs 16 years. Halving the edge quadruples the data, as the chart shows. If the data you have is shorter than this, the honest outcome of the test is "cannot tell".
Step 4: decide, including to stop
The decision has three outcomes: adopt, reject, or collect more data. Kill criteria make rejection routine: "stop if the out-of-sample t-statistic is below 1, or if costs remove more than half the gross edge." Without them, every failed test gets one more tweak, and the tweaks are tries that nobody counts. Record the outcome either way. Negative results are what stop the same idea being tried again.
Common mistakes
- Choosing the pass mark after seeing results.
- Counting only the variant that worked.
- Moving the dates or markets until the effect appears.
- Treating a short record as a negative result, or as a positive one. It is usually neither.
- Skipping the reason. A pattern nobody can explain is hard to trust and hard to stop trusting.
- Failing to record rejected ideas.
In practice
Keep one page per idea with the question, hypothesis, reason, pass mark and kill criteria, dated before the test runs, then add the result and the decision. Research on FinUties is where studies built on methods like these are published, with their method and limits stated.
Years of data needed for t = 2, by Sharpe ratio
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| power: chance a real edge passes | 0.8 | 80% |
| false-positive rate | 0.05 | 5% |
| share of ideas that are truly good, optimistic | 0.10 | 10% |
| chance a passing idea is good, prior 10% | power * prior_hi / (power * prior_hi + alpha * (1 - prior_hi)) | 64% |
| share of ideas that are truly good, sceptical | 0.02 | 2% |
| chance a passing idea is good, prior 2% | power * prior_lo / (power * prior_lo + alpha * (1 - prior_lo)) | 24.6% |
| years to reach t = 2 at a Sharpe ratio of 0.5 | (2 / 0.5)**2 | 16 years |
| years to reach t = 2 at a Sharpe ratio of 1.0 | (2 / 1.0)**2 | 4 years |
Check yourself
Why write the hypothesis and the pass mark down before looking at results?
Because a pass mark chosen after seeing the data can be set to whatever the data allows. Fixing it first keeps the test honest and keeps the number of tries countable.
A test has 80% power and a 5% false-positive rate. One idea in ten is truly good. A test passes. How likely is the idea good?
0.8 x 0.10 / (0.8 x 0.10 + 0.05 x 0.90) = 64%. A pass is encouraging, not proof.
The same test is used where only one idea in fifty is good. What changes?
0.8 x 0.02 / (0.8 x 0.02 + 0.05 x 0.98) = 24.6%. Most passes are now false. The prior matters as much as the test.
An edge has an annual Sharpe ratio of 0.5. About how many years of independent data give a t-statistic of 2?
t is about SR x sqrt(years), so years = (2 / 0.5)^2 = 16. Small edges need long records, which is why short tests prove little.
Key terms
Sources
- Why Most Published Research Findings Are False. PLoS Medicine 2(8), e124 (John P. A. Ioannidis), 2005. Accessed 9 Nov 2026
- Presidential Address: The Scientific Outlook in Financial Economics. The Journal of Finance 72(4), 1399-1440 (Campbell R. Harvey), 2017. Accessed 9 Nov 2026
- The preregistration revolution. Proceedings of the National Academy of Sciences 115(11), 2600-2606 (Brian A. Nosek et al.), 2018. Accessed 9 Nov 2026