Quant Analytics: from scratch to pro · Professional practice and failure case studies · Advanced
Pre-registration and trial ledgers
A result is only as strong as the count of things tried to get it. A pre-registered plan and a ledger that records every trial make that count honest, and set the hurdle a winner has to clear.
In this lesson
- Explain what pre-registration fixes and what it does not
- Keep a trial ledger that counts every variant, including failed and abandoned ones
- Estimate how large a best-of-N result is expected to be by luck alone
- Raise the pass mark to account for the number of trials, using Bonferroni or a harder t-statistic
- Protect a final holdout so it is used once
Read first: The research process: question, hypothesis, test, decision, Reading a research claim: ten questions to ask
Every result carries a hidden number: how many things were tried to get it. A rule found after one test and a rule found as the best of 100 can print the same Sharpe ratio and mean very different things. Pre-registration and a trial ledger put that number on the page.
What pre-registration is
Pre-registration means writing the plan down, dated, before the data is analysed: the question, the data, the definitions, the pass mark and what ends the work. It comes from experimental science, where it separates tests of an idea from searches for one (Nosek et al., 2018). It does not stop exploration. It marks which results were predicted and which were found afterwards, and it treats the second kind with more suspicion.
Harvey (2017), in his American Finance Association address, argued for the same discipline in finance: state the hypothesis and the tests before looking, and report every attempt.
The trial ledger
A trial ledger is a log with one row per variant tried. Each row holds the date, a description, the parameters, the data used and the result. The rules:
- A trial is any change that can alter results: a parameter, a filter, a market, a date range, a cost model, a definition.
- Failed and abandoned trials stay in the ledger. Deleting them is the error the ledger exists to prevent.
- The count of trials is reported next to the headline result.
- Trials that are close variants of each other are marked as related, which gives an effective count somewhat below the raw one.
Worked example: the best of 100 rules
Take 100 rules with no edge at all, each tested over 5 years. The standard error of an annual Sharpe ratio with no edge is about 1 / sqrt(years), which is 0.45.
The expected largest of N standard normal draws is approximately (1 - g) x z(1 - 1/N) + g x z(1 - 1/(N x e)), where g is 0.577 and z is the normal quantile (Bailey and López de Prado, 2014). For N = 100 this is 2.53. The best rule is therefore expected to show a Sharpe ratio of 2.53 x 0.45, which is 1.13, from luck alone. The result a reader would call excellent is the average winner of a pointless search.
The chance of at least one false pass at the 5% level is 1 - 0.95^100, which is 99.4%. With 10 trials it is 40.1%. The chart shows how fast it climbs.
Raising the hurdle
Two common corrections:
- Bonferroni: divide the test level by the number of trials. For 100 trials the level is 0.05%, which is a two-sided z-score of about 3.48.
- A higher t-statistic: Harvey, Liu and Zhu (2016) studied hundreds of published return factors and argued that a newly discovered one should clear a t-statistic above 3.0, not 2.0.
Both are conservative when trials are correlated. The point is the direction: the more tries, the higher the bar. Multiple testing covers the less strict false discovery rate approach.
Protecting the holdout
Split off a final stretch of data before any work starts. Use everything else to design and select. Touch the holdout once, for the final candidate, with a pass mark written in advance. If it fails, the idea is rejected, and the holdout is spent for that idea. Looking again after a tweak turns it into training data. See walk-forward and the locked holdout.
Common mistakes
- Keeping a ledger of the successes only.
- Counting only parameter changes and ignoring changes of market, dates or definition.
- Pre-registering loosely enough that any result fits ("the rule will work in some regime").
- Moving the pass mark after the ledger shows many trials, in either direction, without saying so.
- Using the holdout to choose between two finalists.
- Treating a pre-registered plan as proof. It limits bias; it does not create an edge.
In practice
Start the ledger on the first day of a project and make the count part of every summary: "best of 40 variants, one holdout look". Ask for the same from anyone else's claim. Research on FinUties is where studies built on methods like these are published, with their method and limits stated.
Chance of at least one false pass, by number of trials
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| number of trials in the ledger | 100 | 100 |
| record length in years | 5 | 5 years |
| standard error of an annual Sharpe ratio under no edge | 1 / sqrt(years) | 0.45 |
| Euler-Mascheroni constant | 0.5772156649 | 0.577 |
| expected largest of 100 standard normal draws | (1 - gamma) * phi_inv(1 - 1/trials) + gamma * phi_inv(1 - 1/(trials * e)) | 2.53 |
| Sharpe ratio of the best rule by luck alone | emax * se | 1.13 |
| chance at least one of 100 useless rules passes at 5% | 1 - 0.95**trials | 99.4% |
| Bonferroni per-test level for 100 tests | 0.05 / trials | 0.05% |
| two-sided z-score for that level | phi_inv(1 - bonf / 2) | 3.48 |
| chance at least one of 10 useless rules passes at 5% | 1 - 0.95**10 | 40.1% |
Check yourself
Why is the best of 100 random rules expected to look good?
Each rule's estimated Sharpe ratio is noisy. The largest of 100 noisy values sits about 2.5 standard errors above zero. With a 5-year record, the standard error is 0.45, so the best rule shows a Sharpe ratio of about 1.13 with no edge at all.
What is the chance that at least one of 100 useless rules passes a 5% test?
1 - 0.95^100 = 99.4%. It is nearly certain.
What does a Bonferroni correction do for 100 tests?
It divides the 5% level by 100, so each test must pass at 0.05%. For a two-sided test that is a z-score of about 3.48.
A researcher reports a result and says only 3 variants were run, but the code history shows 40. What is wrong?
The ledger undercounts. Judge the result as the best of 40, not the best of 3. Every edit that changes results is a trial.
Why use a holdout only once?
Each look at it and each change that follows turns it into training data. After repeated looks it no longer measures anything out of sample.
Key terms
Checked facts
Every figure below carries how it was checked: Primary checked (read in the cited primary, official, academic or data source), Two sources agree, or Approximate (an order of magnitude only).
- Primary checked
Harvey, Liu and Zhu (2016) argue that a newly discovered factor should clear a t-statistic above 3.0, not the conventional 2.0.
Sources
- The preregistration revolution. Proceedings of the National Academy of Sciences 115(11), 2600-2606 (Brian A. Nosek et al.), 2018. Accessed 9 Nov 2026
- Presidential Address: The Scientific Outlook in Financial Economics. The Journal of Finance 72(4), 1399-1440 (Campbell R. Harvey), 2017. Accessed 9 Nov 2026
- ... and the Cross-Section of Expected Returns. The Review of Financial Studies 29(1), 5-68 (Campbell R. Harvey, Yan Liu, Heqing Zhu), 2016. Accessed 9 Nov 2026
- The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality. The Journal of Portfolio Management 40(5), 94-107 (David H. Bailey, Marcos López de Prado), 2014. Accessed 9 Nov 2026