Skip to main content

Quant Analytics: from scratch to pro · Professional practice and failure case studies · Beginner

Reading a research claim: ten questions to ask

A headline result such as 'this rule returned 18% a year' says little until you know how many things were tried, how much data stood behind it and what it cost. Ten questions turn a claim into something you can weigh.

4 min read Reviewed Revision 1

In this lesson

  • Ask ten standard questions of any performance or research claim
  • Turn a quoted Sharpe ratio and track length into a rough confidence range
  • Work out how likely it is that the best of many tries looks good by luck
  • Tell a claim that survives scrutiny from one that only reports a result

Read first: Risk and return

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
quoted annual Sharpe ratio1.21.2
length of the record in years33 years
standard error of the Sharpe ratiosqrt((1 + 0.5 * sr**2) / years)0.76
lower end of the rough 95% rangesr - 1.96 * se0.28
upper end of the rough 95% rangesr + 1.96 * se2.68
t-statistic of the quoted Sharpe ratiosr / se1.58
chance that at least one of 20 tries passes a 5% test by luck1 - 0.95**2064.2%
same for 5 tries1 - 0.95**522.6%
accuracy above the base rate(0.58 - 0.57) * 1001 percentage point

Check yourself

A report says a rule had a Sharpe ratio of 1.2 over 3 years. Is that clearly better than zero?

No. The standard error is sqrt((1 + 0.5 x 1.2^2) / 3) = 0.76, so the rough 95% range runs from -0.28 to 2.68 and includes zero.

Twenty variants of a rule are tried and the best is reported. Each variant has a 5% chance of looking significant by luck. How likely is at least one lucky winner?

1 - 0.95^20 = 64.2%. A reported winner among twenty tries is closer to expected than surprising.

A model predicts 'up' days with 58% accuracy. Up days are 57% of all days. Is it useful?

Barely. Always saying 'up' scores 57%, so the model adds one percentage point. Accuracy means nothing until it is set against the base rate.

Which single question is the most often skipped?

How many things were tried. A result without a count of trials cannot be judged, because the best of many tries always looks good.

Key terms

Trial ledgerBase rateMultiple testing

Sources

Informational content. Not investment advice.