Skip to main content

Quant Analytics: from scratch to pro · Backtesting done right · Advanced

Walk-forward, out-of-sample and the locked holdout

Fit on one stretch of history, test on the next, then roll forward. Keep a last stretch that is touched once. These designs measure what a method does on data it did not choose its settings from, and show how little a short test can say.

4 min read Reviewed Revision 1

In this lesson

  • Lay out a walk-forward test with train and test windows and count its windows
  • State the rules for a locked holdout and what counts as peeking
  • Estimate how much out-of-sample data is needed to tell a Sharpe ratio from zero
  • Compare in-sample and out-of-sample results and read the gap

Read first: Overfitting, trial counts and the deflated Sharpe ratio, Look-ahead bias and other leaks into a backtest

Standard error of an annual Sharpe ratio against years of data

  • true Sharpe 0
  • true Sharpe 1
0.00 0.25 0.50 0.75 1.00 1.25 1351020 Years of data
Standard error. Computed from the large-sample formula sqrt((1 + SR^2/2) / years) for independent, normal returns (Lo, 2002). Real returns with fat tails and autocorrelation have larger errors.

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
years of data10
number of test windowsyears_total - 37 test windows
standard error of the annual Sharpe ratio over 7 years, zero skill1 / sqrt(7)0.38
standard error over one test window1 / sqrt(1)1.00
out-of-sample Sharpe as a share of in-sample0.6 / 1.540%
out-of-sample Sharpe in standard errors0.6 / se71.6
holdout length, 20% of 10 years0.2 * years_total2 years
z-score of a true Sharpe of 1 over the holdout1 * sqrt(hold_years)1.41
years to separate a Sharpe of 1 from zero at 1.96 standard errors1.96**2 * (1 + 0.5 * 1**2) / 1**25.8 years
chance of at least one pass in 5 looks at 5%1 - 0.95**522.6%

Check yourself

Ten years of data, a 3-year training window and a 1-year test window, rolled one year at a time. How many test windows, and how many years of out-of-sample data?

Windows start at years 0 to 6, so there are 10 - 3 = 7 test windows and 7 years of out-of-sample data. The first 3 years are only ever training data.

A method has an in-sample Sharpe ratio of 1.5 and an out-of-sample Sharpe ratio of 0.6. What share survived?

0.6 / 1.5 = 40%. A large drop is common after a search and is not itself a failure; a drop to zero or below is.

Why is a 2-year holdout weak evidence for a true Sharpe ratio of 1?

The z-score is about 1 x sqrt(2) = 1.41 for a rule with that true Sharpe, under the 1.96 needed. Two years is not long enough to separate it from zero reliably.

A holdout is looked at 5 times while the rule is adjusted between looks. Roughly what is the chance that a rule with no edge passes a 5% test at least once?

1 - 0.95^5 = 22.6%. Each look is a trial, so the holdout stops being untouched after the first.

Key terms

Walk-forward analysisOut-of-sampleHoldout setOverfitting

Sources

Informational content. Not investment advice.