Glossary
Holdout set
A final stretch of data set aside before research starts and used once, to judge the finished rule. Looking at it again to make a choice uses it up.
A holdout set is cut off before any research and kept where the research code cannot read it. The pass level is written down first, the data is examined once, and the result is final. If the rule is changed in response, the holdout becomes training data.
Each extra look counts as a trial. A short holdout is also weak evidence: a true annual Sharpe ratio of 1 over a 2-year holdout gives a z-score of about 1.41, under the 1.96 usually needed.
Example: five looks at a 5% level give a 22.6% chance that a rule with no edge passes at least once.
Sources
- Advances in Financial Machine Learning. Wiley (Marcos López de Prado), 2018. Accessed 9 Nov 2026