Skip to main content

Quant Analytics: from scratch to pro · Machine learning in finance · Advanced

Leakage, purged cross-validation and embargo

Ordinary cross-validation assumes independent samples. Market labels overlap in time, so a random split puts near-copies of the test data into training and inflates the score. Purging and an embargo remove them.

3 min read Reviewed Revision 1

In this lesson

  • Recognise the common forms of data leakage in a model-building process
  • Explain why a random k-fold split overstates performance on time-dependent data
  • Apply purging and an embargo and count the training samples that remain
  • Keep preprocessing inside each training fold

Read first: Features and labels: fixed horizons and the triple barrier, Look-ahead bias and other leaks into a backtest

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
samples10001,000
test fold sizen / 5200
neighbours sharing a label window2 * 918
expected overlapping training samples per test sampleneighbours * 0.814.4
embargo at 1% of the sample0.01 * n10
training samples, middle foldn - fold - 9 - embargo781
training samples, first foldn - fold - embargo790
share of the sample used for training, middle foldtrain_mid / n78%

Check yourself

Labels span 10 observations. A random split puts a test sample next to training samples. How many neighbours share part of its label window, and what fraction of them are in training with 5 folds?

9 neighbours on each side share some of the window, so 18. With 5 folds about 80% are in training: 18 x 0.8 = 14.4 overlapping training samples per test sample, on average.

1,000 samples, 5 folds, labels of length 10, embargo of 1%. How many training samples remain for a middle fold?

Test fold: 200. Purge: the 9 samples before the test fold whose 10-observation labels reach into it. Embargo: 1% of 1,000 = 10 after it. Training = 1,000 - 200 - 9 - 10 = 781.

Why is an embargo needed on top of purging?

Purging removes training samples whose labels overlap the test period. Serial dependence (volatility clustering, slow-moving features) also carries information from the test period into samples that start just after it. The embargo removes that stretch.

A feature is standardised using the mean and standard deviation of the whole data set before splitting. Is that leakage?

Yes. The test period contributes to the scaling used in training. The statistics must be computed on each training fold only and then applied to the test fold.

A score rises when the purge is switched off. What does that tell you?

That part of the score came from overlap between training and test labels, not from skill that carries to new data.

Key terms

Data leakageCross-validationPurged cross-validationTriple-barrier method

Sources

Informational content. Not investment advice.