Quant Analytics: from scratch to pro · Backtesting done right · Advanced
Walk-forward, out-of-sample and the locked holdout
Fit on one stretch of history, test on the next, then roll forward. Keep a last stretch that is touched once. These designs measure what a method does on data it did not choose its settings from, and show how little a short test can say.
In this lesson
- Lay out a walk-forward test with train and test windows and count its windows
- State the rules for a locked holdout and what counts as peeking
- Estimate how much out-of-sample data is needed to tell a Sharpe ratio from zero
- Compare in-sample and out-of-sample results and read the gap
Read first: Overfitting, trial counts and the deflated Sharpe ratio, Look-ahead bias and other leaks into a backtest
A rule tuned on the same data it is tested on will look good. The remedy is to test on data the tuning never saw. This lesson covers three designs for that: the simple split, the walk-forward test and the locked holdout. It also shows how little a short test can settle.
In-sample and out-of-sample
In-sample data is used to choose a rule, its settings and its filters. Out-of-sample data is used only to judge the chosen rule. The two must not overlap, and the out-of-sample part must come later in time, because markets cannot be shuffled.
A simple split uses the first part for research and the rest for a test. Its weakness is that there is only one test, and a single stretch of data may be unusual.
Walk-forward analysis
Walk-forward analysis repeats the split along the timeline. Fit on a training window, apply the fitted rule to the next test window without changes, record the result, then move forward and repeat.
- Rolling: the training window keeps a fixed length and drops the oldest data.
- Anchored (expanding): the start is fixed and the window grows.
The out-of-sample results of all windows are stitched into one series. That series was never used to choose the settings in force when each part was traded. Rolling suits markets that change; anchored uses all the history for the fit. Neither removes the problem of the whole procedure being tuned: the window lengths and the search space are also choices.
Worked example
Ten years of data, a 3-year training window and a 1-year test window, rolled one year at a time. The first test year is year 4. There are 7 test windows, and 7 years of out-of-sample data. In-sample, the tuned rule shows an annual Sharpe ratio of 1.5. Stitched out-of-sample, it shows 0.6, which is 40% of the in-sample value. A drop is normal, since the in-sample figure was the best of a search.
How much does 0.6 tell us? For independent returns, the standard error of an annual Sharpe ratio is about sqrt((1 + SR^2/2) / years). With zero skill and 7 years it is 0.38. A single test window of one year has a standard error of 1.00, larger than most Sharpe ratios. So 0.6 over 7 years is about 1.6 standard errors from zero: suggestive, not settled.
To separate a true Sharpe ratio of 1 from zero at 1.96 standard errors, the sample must be about 1.96^2 x (1 + 0.5) / 1, which is 5.8 years. A holdout of 2 years (20% of the ten) gives a z-score of only 1 x sqrt(2), or 1.41. The chart shows how slowly the error shrinks.
The locked holdout
A holdout set is a final stretch of data set aside before any research starts. Its rules:
- Lock it first. Cut it off before looking at results, and store it where the research code cannot read it.
- One decision rule. Write what counts as a pass (a metric and a level) before the look.
- One look. After the look, the result is final. If the rule is changed in response and tested again, the holdout is used up and becomes training data.
- Count the looks. Five looks at a 5% level give a 22.6% chance that a rule with no edge passes at least once.
Peeking is any use of the holdout to make a choice: picking between two versions, stopping early, or adjusting a threshold.
Overlapping labels
If each training example is labelled with a result that spans several days, the last training examples overlap the first test examples. Information then leaks across the boundary. The fix is to purge the overlapping examples and add an embargo gap after the test window. See leakage and purged cross-validation.
Common mistakes
- Tuning on the test windows, then reporting the walk-forward result as if untouched.
- Choosing the window lengths by looking at the walk-forward result. That is another search.
- Reading a short test as proof when it only fails to reject. Report the standard error.
- Splitting at random, which mixes future and past.
- Releasing the holdout to several researchers who all try ideas on it.
In practice
Decide the split, the windows and the pass level before the first run, and write them down. Report in-sample and out-of-sample results together, with the number of trials from overfitting. Research on FinUties is where studies built on methods like these are published, with their method and limits stated.
Standard error of an annual Sharpe ratio against years of data
- true Sharpe 0
- true Sharpe 1
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| years of data | 10 | |
| number of test windows | years_total - 3 | 7 test windows |
| standard error of the annual Sharpe ratio over 7 years, zero skill | 1 / sqrt(7) | 0.38 |
| standard error over one test window | 1 / sqrt(1) | 1.00 |
| out-of-sample Sharpe as a share of in-sample | 0.6 / 1.5 | 40% |
| out-of-sample Sharpe in standard errors | 0.6 / se7 | 1.6 |
| holdout length, 20% of 10 years | 0.2 * years_total | 2 years |
| z-score of a true Sharpe of 1 over the holdout | 1 * sqrt(hold_years) | 1.41 |
| years to separate a Sharpe of 1 from zero at 1.96 standard errors | 1.96**2 * (1 + 0.5 * 1**2) / 1**2 | 5.8 years |
| chance of at least one pass in 5 looks at 5% | 1 - 0.95**5 | 22.6% |
Check yourself
Ten years of data, a 3-year training window and a 1-year test window, rolled one year at a time. How many test windows, and how many years of out-of-sample data?
Windows start at years 0 to 6, so there are 10 - 3 = 7 test windows and 7 years of out-of-sample data. The first 3 years are only ever training data.
A method has an in-sample Sharpe ratio of 1.5 and an out-of-sample Sharpe ratio of 0.6. What share survived?
0.6 / 1.5 = 40%. A large drop is common after a search and is not itself a failure; a drop to zero or below is.
Why is a 2-year holdout weak evidence for a true Sharpe ratio of 1?
The z-score is about 1 x sqrt(2) = 1.41 for a rule with that true Sharpe, under the 1.96 needed. Two years is not long enough to separate it from zero reliably.
A holdout is looked at 5 times while the rule is adjusted between looks. Roughly what is the chance that a rule with no edge passes a 5% test at least once?
1 - 0.95^5 = 22.6%. Each look is a trial, so the holdout stops being untouched after the first.
Key terms
Sources
- Advances in Financial Machine Learning. Wiley (Marcos López de Prado), 2018. Accessed 9 Nov 2026
- The Statistics of Sharpe Ratios. Financial Analysts Journal 58(4), 36-52 (Andrew W. Lo), 2002. Accessed 9 Nov 2026
- The Elements of Statistical Learning (2nd edition). Springer (Trevor Hastie, Robert Tibshirani, Jerome Friedman), 2009. Accessed 9 Nov 2026