Glossary
Multiple testing
Running many statistical tests on the same data, so that some pass by chance. The more tests, the higher the chance that the best looks significant.
If each of N independent tests on noise has a 5% chance of passing, the chance that at least one passes is 1 - 0.95^N. The honest response is to count every test and raise the bar: divide the level by N (Bonferroni) or control the false discovery rate (Benjamini-Hochberg).
Results from a search over many rules or parameters are the maximum of many draws, not one draw.
Example: 20 tests on pure noise give a 64.15% chance that at least one has p below 5%.