Glossary
Trial ledger
A log with one row for every variant tried in a research project, including failures. It supplies the count needed to judge a best-of-N result.
A trial ledger records every variant tried: date, description, parameters, data and result. A trial is any change that can alter results. Failed and abandoned trials stay in the log.
The count matters because the best of many tries looks good by chance. Among N independent useless rules tested at the 5% level, the chance that at least one passes is 1 - 0.95^N.
Example: a ledger of 40 variants with one reported winner means the result is judged as the best of 40, and 1 - 0.95^40 is about 87% for a false pass.