Skip to main content

Quant Analytics: from scratch to pro · Indicators, honestly measured · Intermediate

Does an indicator carry information? Measuring it honestly

An indicator is useful only if what follows its signals differs from what follows any other day. This lesson sets out an event study: forward returns after a signal, the base rate, overlap, a random-entry comparison, costs and the count of variants tried.

5 min read Reviewed Revision 1

In this lesson

  • Set up an event study with forward returns, a base rate and a clear horizon
  • Compare a signal's average outcome with the unconditional average and compute its standard error
  • Adjust for overlapping windows, costs and the number of variants tried
  • Describe what the data-snooping literature on technical rules did and why

Read first: Moving averages and their lag, Oscillators: what RSI and MACD compute, Hypothesis tests and p-values for market data

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
excess forward return over the base rate0.0030 - 0.00100.20%
standard error with 100 independent events0.025 / sqrt(100)0.25%
t-statisticexcess / se_ev0.8
two-sided p-value2 * (1 - phi(t_ev))42%
independent events when signals arrive in runs of four100 / 425
standard error with 25 independent events0.025 / sqrt(n_eff)0.50%
t-statistic with 25 independent eventsexcess / se_eff0.4
excess after a round-trip cost of 0.15%excess - 0.00150.05%
standard error of a hit rate under a 53% base ratesqrt(0.53 * 0.47 / 100)4.99%
z-statistic of 58 up out of 100(0.58 - 0.53) / se_hit1.0
two-sided p-value2 * (1 - phi(z_hit))31.6%
chance that one of 20 noise variants passes at 5%1 - 0.95**2064.15%

Check yourself

A signal fires 100 times. The next 5-day return averages 0.30%. Any 5-day window averages 0.10%. What is the excess?

0.30% - 0.10% = 0.20%. The comparison is with the base rate, not with zero.

The 5-day return has a standard deviation of 2.5%. How significant is that excess over 100 independent signals?

SE = 2.5% / sqrt(100) = 0.25%, so t = 0.20% / 0.25% = 0.8, a two-sided p-value of about 42%. Not significant.

The signals come in runs of four consecutive days and the horizon is 5 days. How many independent events are there?

About 100 / 4 = 25. The standard error doubles to 0.50% and t falls to 0.4.

Up days are 53% of all 5-day windows. After the signal, 58 of 100 are up. Is that evidence?

SE under the base rate = sqrt(0.53 x 0.47 / 100) = 4.99%. z = 5 / 4.99 = 1.0, two-sided p = 31.6%. No.

Twenty variants of the indicator were tried and the best is reported. What is the chance that one of 20 noise variants passes a 5% test?

1 - 0.95^20 = 64.15%.

Key terms

Base rateMoving averageRelative Strength IndexBollinger Bands

Checked facts

Every figure below carries how it was checked: Primary checked (read in the cited primary, official, academic or data source), Two sources agree, or Approximate (an order of magnitude only).

Sources

Informational content. Not investment advice.