Quant Analytics: from scratch to pro · Indicators, honestly measured · Intermediate
Does an indicator carry information? Measuring it honestly
An indicator is useful only if what follows its signals differs from what follows any other day. This lesson sets out an event study: forward returns after a signal, the base rate, overlap, a random-entry comparison, costs and the count of variants tried.
In this lesson
- Set up an event study with forward returns, a base rate and a clear horizon
- Compare a signal's average outcome with the unconditional average and compute its standard error
- Adjust for overlapping windows, costs and the number of variants tried
- Describe what the data-snooping literature on technical rules did and why
Read first: Moving averages and their lag, Oscillators: what RSI and MACD compute, Hypothesis tests and p-values for market data
A chart shows every indicator signal sitting at a sensible place, because the eye looks for them after the move. The honest question is different: after the indicator fires, is the future return different from what it is on any other day, by more than chance would give? An event study answers that.
The set-up
- Define the signal before looking. Exact formula, parameters, price used, timeframe. For an RSI, say which length and which level, and whether the signal is the crossing or the state.
- Mark the events. Every date on which the signal fires.
- Fix a horizon. The forward return over
hperiods after the event, chosen before the test. - Compute the outcome after events. Mean forward return, hit rate, spread.
- Compute the base rate. The same statistics for all days, or for days in the same regime. A rising market has a positive average return on any day.
- Compare the two, with a standard error.
The base rate is the step most often skipped. A signal followed by +0.30% proves nothing when any day is followed by +0.10%. The comparison is the excess.
Worked example
An indicator fires 100 times over ten years. The 5-day return after the signal averages 0.30%. Over all days it averages 0.10%. The 5-day standard deviation is 2.5%.
- Excess: 0.20%.
- Standard error with 100 independent events:
2.5% / sqrt(100)= 0.25%. - t-statistic: 0.8. Two-sided p-value: 42%.
The excess is well inside the noise. The same arithmetic as in standard error applies.
Hit rates need a base rate too
Up days are 53% of 5-day windows. After the signal 58 of 100 are up. The standard error under the base rate is sqrt(0.53 x 0.47 / 100) = 4.99%, so z = 5% / 4.99% = 1.0, a p-value of 31.6%. A 58% hit rate sounds strong. Against 53% it is not distinguishable from luck at this sample size.
Overlap
Signals often cluster: a falling price keeps an oscillator low for several days. If the horizon is 5 days and the signal fires on four consecutive days, the windows overlap and share most of their returns. They are not independent events. If the 100 events come in runs of four, the independent count is about 25, the standard error doubles to 0.50% and the t-statistic falls to 0.4. Count one event per run, or use methods that allow for the dependence, such as a block bootstrap.
A random-entry comparison
A direct way to build the right yardstick is to draw the same number of dates at random, compute the same statistic, and repeat many times. The share of random draws that match or beat the indicator is its p-value, with no assumption about the shape of returns. To keep it fair, draw from the same periods and the same market state as the signals, and keep the horizon and the holding rules the same.
Costs
An average of the forward return is a gross figure. Subtract the round-trip cost (spread, commission, slippage; see spreads and trading costs). In the example, a cost of 0.15% takes the excess from 0.20% to 0.05%, and it does not make the statistic any less noisy. Edges that survive only when costs are ignored have no use.
Variants and the search
How many versions were tried: lengths, thresholds, horizons, markets? If 20 variants were tested and the best is reported, the chance that at least one noise variant passes a 5% test is 64.15%. Report all of them, or adjust as in multiple testing.
What the research literature did
Brock, Lakonishok and LeBaron (1992) tested two simple families of rules, moving averages and trading-range breaks, on Dow Jones data from 1897 to 1986. They reported that returns in buy-signal periods were higher and less volatile than in sell-signal periods, and not consistent with four standard models of returns, using bootstrap tests to judge this.
Sullivan, Timmermann and White (1999) took up the same question with the data-snooping problem in mind. Rules like these are drawn from a large universe, and the best of a universe looks better than a rule chosen in advance. They applied White's (2000) reality check, a bootstrap that tests the best rule against the entire set of rules tried, to a larger universe than the original study on about a century of daily Dow Jones data. The point to take away is the method: judge the best rule against the full search, not against zero.
Common mistakes
- Comparing the signal's outcome with zero instead of the base rate.
- Counting overlapping windows as separate events.
- Choosing the horizon, level or length after seeing the results.
- Ignoring costs, or using costs far below what the instrument charges.
- Reporting the best variant without the number of variants.
In practice
Write the signal, horizon and comparison first. Report the excess over the base rate with its standard error, the number of independent events, the costs and the number of variants tried. A positive result earns a test on data not used to find it. Analytics on FinUties applies measures like these to the markets it covers, so they are at hand when you read a chart.
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| excess forward return over the base rate | 0.0030 - 0.0010 | 0.20% |
| standard error with 100 independent events | 0.025 / sqrt(100) | 0.25% |
| t-statistic | excess / se_ev | 0.8 |
| two-sided p-value | 2 * (1 - phi(t_ev)) | 42% |
| independent events when signals arrive in runs of four | 100 / 4 | 25 |
| standard error with 25 independent events | 0.025 / sqrt(n_eff) | 0.50% |
| t-statistic with 25 independent events | excess / se_eff | 0.4 |
| excess after a round-trip cost of 0.15% | excess - 0.0015 | 0.05% |
| standard error of a hit rate under a 53% base rate | sqrt(0.53 * 0.47 / 100) | 4.99% |
| z-statistic of 58 up out of 100 | (0.58 - 0.53) / se_hit | 1.0 |
| two-sided p-value | 2 * (1 - phi(z_hit)) | 31.6% |
| chance that one of 20 noise variants passes at 5% | 1 - 0.95**20 | 64.15% |
Check yourself
A signal fires 100 times. The next 5-day return averages 0.30%. Any 5-day window averages 0.10%. What is the excess?
0.30% - 0.10% = 0.20%. The comparison is with the base rate, not with zero.
The 5-day return has a standard deviation of 2.5%. How significant is that excess over 100 independent signals?
SE = 2.5% / sqrt(100) = 0.25%, so t = 0.20% / 0.25% = 0.8, a two-sided p-value of about 42%. Not significant.
The signals come in runs of four consecutive days and the horizon is 5 days. How many independent events are there?
About 100 / 4 = 25. The standard error doubles to 0.50% and t falls to 0.4.
Up days are 53% of all 5-day windows. After the signal, 58 of 100 are up. Is that evidence?
SE under the base rate = sqrt(0.53 x 0.47 / 100) = 4.99%. z = 5 / 4.99 = 1.0, two-sided p = 31.6%. No.
Twenty variants of the indicator were tried and the best is reported. What is the chance that one of 20 noise variants passes a 5% test?
1 - 0.95^20 = 64.15%.
Key terms
Base rateMoving averageRelative Strength IndexBollinger Bands
Checked facts
Every figure below carries how it was checked: Primary checked (read in the cited primary, official, academic or data source), Two sources agree, or Approximate (an order of magnitude only).
- Primary checked
Brock, Lakonishok and LeBaron (1992) tested two simple rule families, moving average and trading range break, on the Dow Jones index from 1897 to 1986.
- Primary checked
They reported that returns in buy-signal periods were higher and less volatile than in sell-signal periods, and not consistent with four null models (random walk, AR(1), GARCH-M, EGARCH), using bootstrap tests.
- Primary checked
Sullivan, Timmermann and White (1999) applied White's Reality Check bootstrap to a larger universe of technical rules on about a century of daily Dow Jones data, to adjust for data snooping across the whole universe.
Sources
- Simple Technical Trading Rules and the Stochastic Properties of Stock Returns. The Journal of Finance 47(5), 1731-1764 (William Brock, Josef Lakonishok, Blake LeBaron), 1992. Accessed 9 Nov 2026
- Data-Snooping, Technical Trading Rule Performance, and the Bootstrap. The Journal of Finance 54(5), 1647-1691 (Ryan Sullivan, Allan Timmermann, Halbert White), 1999. Accessed 9 Nov 2026
- A Reality Check for Data Snooping. Econometrica 68(5), 1097-1126 (Halbert White), 2000. Accessed 9 Nov 2026
- The Econometrics of Financial Markets. Princeton University Press (John Y. Campbell, Andrew W. Lo, A. Craig MacKinlay), 1997. Accessed 9 Nov 2026