Skip to main content

Quant Analytics: from scratch to pro · Professional practice and failure case studies · Intermediate

The research process: question, hypothesis, test, decision

Good quantitative research runs in a fixed order: a question, a hypothesis written down, a test whose pass mark is set in advance, and a decision that includes the option to stop. The order protects against fooling yourself.

3 min read Reviewed Revision 1

In this lesson

  • Turn a loose idea into a question, a hypothesis and a pre-set pass mark
  • Estimate how much data a test needs before it can tell a small edge from zero
  • Use a prior probability to judge how much a single positive result is worth
  • Write kill criteria that end a line of work without argument

Read first: Reading a research claim: ten questions to ask

Years of data needed for t = 2, by Sharpe ratio

0 20 40 60 0.250.50.7511.251.52 Annual Sharpe ratio
Years. Computed from t = SR x sqrt(years) for independent returns. Halving the edge quadruples the data needed.

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
power: chance a real edge passes0.880%
false-positive rate0.055%
share of ideas that are truly good, optimistic0.1010%
chance a passing idea is good, prior 10%power * prior_hi / (power * prior_hi + alpha * (1 - prior_hi))64%
share of ideas that are truly good, sceptical0.022%
chance a passing idea is good, prior 2%power * prior_lo / (power * prior_lo + alpha * (1 - prior_lo))24.6%
years to reach t = 2 at a Sharpe ratio of 0.5(2 / 0.5)**216 years
years to reach t = 2 at a Sharpe ratio of 1.0(2 / 1.0)**24 years

Check yourself

Why write the hypothesis and the pass mark down before looking at results?

Because a pass mark chosen after seeing the data can be set to whatever the data allows. Fixing it first keeps the test honest and keeps the number of tries countable.

A test has 80% power and a 5% false-positive rate. One idea in ten is truly good. A test passes. How likely is the idea good?

0.8 x 0.10 / (0.8 x 0.10 + 0.05 x 0.90) = 64%. A pass is encouraging, not proof.

The same test is used where only one idea in fifty is good. What changes?

0.8 x 0.02 / (0.8 x 0.02 + 0.05 x 0.98) = 24.6%. Most passes are now false. The prior matters as much as the test.

An edge has an annual Sharpe ratio of 0.5. About how many years of independent data give a t-statistic of 2?

t is about SR x sqrt(years), so years = (2 / 0.5)^2 = 16. Small edges need long records, which is why short tests prove little.

Key terms

Base rateTrial ledgerPre-registration

Sources

Informational content. Not investment advice.