Skip to main content

Quant Analytics: from scratch to pro · Professional practice and failure case studies · Advanced

Pre-registration and trial ledgers

A result is only as strong as the count of things tried to get it. A pre-registered plan and a ledger that records every trial make that count honest, and set the hurdle a winner has to clear.

4 min read Reviewed Revision 1

In this lesson

  • Explain what pre-registration fixes and what it does not
  • Keep a trial ledger that counts every variant, including failed and abandoned ones
  • Estimate how large a best-of-N result is expected to be by luck alone
  • Raise the pass mark to account for the number of trials, using Bonferroni or a harder t-statistic
  • Protect a final holdout so it is used once

Read first: The research process: question, hypothesis, test, decision, Reading a research claim: ten questions to ask

Chance of at least one false pass, by number of trials

0% 20% 40% 60% 80% 100% 151020305075100 Number of independent trials
Chance of a false pass. Computed from the formula for independent tests at the 5% level. Real trials are often correlated, which lowers the curve, but the direction is the same.

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
number of trials in the ledger100100
record length in years55 years
standard error of an annual Sharpe ratio under no edge1 / sqrt(years)0.45
Euler-Mascheroni constant0.57721566490.577
expected largest of 100 standard normal draws(1 - gamma) * phi_inv(1 - 1/trials) + gamma * phi_inv(1 - 1/(trials * e))2.53
Sharpe ratio of the best rule by luck aloneemax * se1.13
chance at least one of 100 useless rules passes at 5%1 - 0.95**trials99.4%
Bonferroni per-test level for 100 tests0.05 / trials0.05%
two-sided z-score for that levelphi_inv(1 - bonf / 2)3.48
chance at least one of 10 useless rules passes at 5%1 - 0.95**1040.1%

Check yourself

Why is the best of 100 random rules expected to look good?

Each rule's estimated Sharpe ratio is noisy. The largest of 100 noisy values sits about 2.5 standard errors above zero. With a 5-year record, the standard error is 0.45, so the best rule shows a Sharpe ratio of about 1.13 with no edge at all.

What is the chance that at least one of 100 useless rules passes a 5% test?

1 - 0.95^100 = 99.4%. It is nearly certain.

What does a Bonferroni correction do for 100 tests?

It divides the 5% level by 100, so each test must pass at 0.05%. For a two-sided test that is a z-score of about 3.48.

A researcher reports a result and says only 3 variants were run, but the code history shows 40. What is wrong?

The ledger undercounts. Judge the result as the best of 40, not the best of 3. Every edit that changes results is a trial.

Why use a holdout only once?

Each look at it and each change that follows turns it into training data. After repeated looks it no longer measures anything out of sample.

Key terms

Pre-registrationTrial ledgerMultiple testing

Checked facts

Every figure below carries how it was checked: Primary checked (read in the cited primary, official, academic or data source), Two sources agree, or Approximate (an order of magnitude only).

Sources

Informational content. Not investment advice.