Quant Analytics: from scratch to pro · Professional practice and failure case studies · Beginner
Reading a research claim: ten questions to ask
A headline result such as 'this rule returned 18% a year' says little until you know how many things were tried, how much data stood behind it and what it cost. Ten questions turn a claim into something you can weigh.
In this lesson
- Ask ten standard questions of any performance or research claim
- Turn a quoted Sharpe ratio and track length into a rough confidence range
- Work out how likely it is that the best of many tries looks good by luck
- Tell a claim that survives scrutiny from one that only reports a result
Read first: Risk and return
A claim such as "this rule returned 18% a year" is a sentence, not a finding. It leaves out how many other rules were tried, how long the record is, what trading cost and what would have changed the author's mind. Most mistakes in quantitative work start by accepting the sentence without these answers. This lesson gives ten questions to ask, in the order they usually matter.
The ten questions
- What exactly is claimed? A rule, a pattern, a model or a portfolio? Over which markets and dates?
- How many things were tried? The best of many tries looks good by chance alone. Ask for the count, including discarded ideas.
- How much data is there? Count independent observations, not rows. Ten years of daily data is not 2,500 independent tests of a monthly idea.
- Was the idea fixed before the test? A hypothesis written down first is stronger than one found by looking. See pre-registration.
- Was any data held back? Results on data used to choose the rule are in-sample. Ask for a holdout that was touched once.
- Are costs and delays included? Spread, commission, slippage and the lag between signal and fill.
- What is the comparison? Against holding the asset, against random entries, against a simple benchmark, all on the same risk.
- Is the effect larger than its noise? A standard error or confidence interval, not a bare average.
- Does a reason exist? A mechanism that explains why the pattern should persist and who pays for it.
- Could it be reproduced? Data, code and definitions clear enough that someone else gets the same number.
Worked example: a quoted Sharpe ratio
A fund note reports a Sharpe ratio of 1.2 over 3 years. The Sharpe ratio is the average excess return divided by its volatility. Its sampling error is the point most notes omit.
For independent returns, the standard error of an annual Sharpe ratio estimated from T years is about sqrt((1 + 0.5 x SR^2) / T) (Lo, 2002). Here that is sqrt((1 + 0.5 x 1.2^2) / 3), which is 0.76.
A rough 95% range is the estimate plus or minus 1.96 standard errors: from -0.28 to 2.68. The range includes zero. The t-statistic is 1.2 / 0.76, which is 1.58, below the usual cut-off of about 2. Three years cannot separate this result from no edge at all.
Worked example: the best of many tries
Test one rule at the 5% level and the chance of a false positive is 5%. Test 5 independent rules and the chance that at least one passes is 1 - 0.95^5, which is 22.6%. Test 20 and it is 1 - 0.95^20, which is 64.2%.
So a paper or a backtest that reports only the winner of 20 variants has a better-than-even chance of showing a winner even when none works. Harvey (2017) and Ioannidis (2005) make the same point for research in general: the more hypotheses are tried, and the smaller the samples, the more published findings are false. This is why question 2 comes before almost everything else.
Accuracy needs a base rate
A model that calls the direction of the next day with 58% accuracy sounds good. If 57% of days are up days, a rule that always says "up" scores 57%. The model adds 1 percentage point. The base rate is what you would get with no skill. Every hit rate, win rate or accuracy figure needs one beside it.
Common mistakes
- Reading a return without a risk measure or a record length.
- Accepting "it works" without a count of what was tried.
- Treating in-sample fit as evidence of future performance.
- Comparing a rule's result with zero instead of with the buy-and-hold result or a random-entry baseline.
- Trusting a chart. Charts are drawn after the result is known.
- Asking only whether the result is statistically significant. Ask whether it is large enough to matter after costs.
In practice
Keep the ten questions as a checklist and write the answers next to every claim you rely on, including your own. A claim that cannot answer questions 2, 3 and 6 is a lead to investigate, not a result. Research on FinUties is where studies built on methods like these are published, with their method and limits stated.
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| quoted annual Sharpe ratio | 1.2 | 1.2 |
| length of the record in years | 3 | 3 years |
| standard error of the Sharpe ratio | sqrt((1 + 0.5 * sr**2) / years) | 0.76 |
| lower end of the rough 95% range | sr - 1.96 * se | 0.28 |
| upper end of the rough 95% range | sr + 1.96 * se | 2.68 |
| t-statistic of the quoted Sharpe ratio | sr / se | 1.58 |
| chance that at least one of 20 tries passes a 5% test by luck | 1 - 0.95**20 | 64.2% |
| same for 5 tries | 1 - 0.95**5 | 22.6% |
| accuracy above the base rate | (0.58 - 0.57) * 100 | 1 percentage point |
Check yourself
A report says a rule had a Sharpe ratio of 1.2 over 3 years. Is that clearly better than zero?
No. The standard error is sqrt((1 + 0.5 x 1.2^2) / 3) = 0.76, so the rough 95% range runs from -0.28 to 2.68 and includes zero.
Twenty variants of a rule are tried and the best is reported. Each variant has a 5% chance of looking significant by luck. How likely is at least one lucky winner?
1 - 0.95^20 = 64.2%. A reported winner among twenty tries is closer to expected than surprising.
A model predicts 'up' days with 58% accuracy. Up days are 57% of all days. Is it useful?
Barely. Always saying 'up' scores 57%, so the model adds one percentage point. Accuracy means nothing until it is set against the base rate.
Which single question is the most often skipped?
How many things were tried. A result without a count of trials cannot be judged, because the best of many tries always looks good.
Key terms
Sources
- Presidential Address: The Scientific Outlook in Financial Economics. The Journal of Finance 72(4), 1399-1440 (Campbell R. Harvey), 2017. Accessed 9 Nov 2026
- Why Most Published Research Findings Are False. PLoS Medicine 2(8), e124 (John P. A. Ioannidis), 2005. Accessed 9 Nov 2026
- The Statistics of Sharpe Ratios. Financial Analysts Journal 58(4), 36-52 (Andrew W. Lo), 2002. Accessed 9 Nov 2026