Quant Analytics: from scratch to pro · Factors, volatility, regimes and lead-lag · Advanced
Market regimes without hindsight
Markets seem to switch between calm and turbulent states. A regime model describes those states with transition probabilities, but labels drawn afterwards use information nobody had at the time. This lesson separates regimes you can know in real time from those you only see in hindsight.
In this lesson
- Compute the expected duration and long-run share of each state from transition probabilities
- Update the probability of a state with Bayes' rule after a new return
- Tell a filtered (real-time) regime estimate from a smoothed (hindsight) one
- Check that a regime rule uses only past data
Read first: Volatility models: EWMA and GARCH, Time series basics: random walks, autocorrelation and stationarity
Volatility, correlation and trend seem to differ across stretches of time: a calm bull market, a panic, a slow grind. It is natural to say the market is in a "regime". The danger is that regimes are easy to see on a finished chart and hard to know on the day. This lesson builds the simplest regime model and then asks what could be known when.
A two-state model
Hamilton (1989) modelled a series whose parameters depend on a hidden state that follows a Markov process, and applied it to postwar US real GNP. The state is not observed; it is inferred from the data.
Take two states: calm (daily volatility 0.6%) and turbulent (1.8%). The state follows a Markov chain: tomorrow's state depends only on today's. Suppose:
- calm stays calm with probability 0.98, and turns turbulent with probability 0.02;
- turbulent stays turbulent with probability 0.95, and calms with probability 0.05.
A state that persists with probability p lasts on average 1 / (1 - p) days. Calm lasts 50 days and turbulent lasts 20. In the long run the share of time in calm is (1 - 0.95) / ((1 - 0.98) + (1 - 0.95)), which is 71.4%, and 28.6% is turbulent. The unconditional daily volatility of the mix is 1.088%, between the two states' values.
Worked example: updating the state
Each day, the model combines a prior (from the transition probabilities) with the day's return, using Bayes' rule. Suppose yesterday was calm, so the prior probability of turbulence today is 0.02.
A return of -3% is 5 standard deviations in the calm state and 1.7 in the turbulent one. The relative density values (the constant shared by both states cancels) are 0.00062 in calm and 13.85 in turbulent. The posterior probability of turbulence is:
0.02 x 13.85 / (0.02 x 13.85 + 0.98 x 0.00062), which is 99.78%.
A single extreme day flips the belief. A milder return of -1.2% is plausible in both states (densities 22.56 and 44.5) and the posterior is only 3.87%. The estimate that uses data up to today is the filtered probability. It is available in real time.
Filtered versus smoothed
A smoothed probability uses the whole sample, including days after the one being labelled. A day sitting in the middle of a turbulent stretch gets a high turbulence probability partly because the days that followed were turbulent. Smoothed labels give the clean regime charts in papers, and they are not available on the day. Using them to decide a past trade is a form of look-ahead bias.
The same applies to simple labels. "Bear market begins at the peak" is known only after the fall. A rule such as "turbulent when the 20-day volatility exceeds a threshold" uses only past data, but it reports late: the average age of data in a 20-day window is (20 - 1) / 2, which is 9.5 days. Faster windows react earlier and flip more often.
What can be known in real time
- Estimating the model: fit the parameters on data up to the date and refit on a schedule, not once on the whole sample.
- The state: use the filtered probability, never the smoothed one.
- The number of states: choose it before looking at results and keep it small.
Regime models describe the volatility clustering that GARCH also captures. See Cont (2001) for the clustering facts and Hamilton (1994) for the textbook treatment.
Common mistakes
- Labelling regimes on the full sample and testing a rule on them. The labels contain the answer.
- Reading a model's states as real, separate worlds. They are a convenient description.
- Choosing the number of states by what fits the past best.
- Fitting on a short sample with one or two real episodes. The transition probabilities are then guesses.
- Expecting regime changes to be announced. A switch is recognised after it has started.
In practice
Test any regime rule in a loop that, on each date, sees only earlier data, and report how late the signal is, not only how clean the labels look. Analytics on FinUties applies measures like these to the markets it covers, so they are at hand when you read a chart.
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| expected duration of the calm state, days | 1 / (1 - 0.98) | 50 |
| expected duration of the turbulent state, days | 1 / (1 - 0.95) | 20 |
| long-run share of calm days | (1 - 0.95) / ((1 - 0.98) + (1 - 0.95)) | 71.4% |
| long-run share of turbulent days | 1 - share_calm | 28.6% |
| unconditional daily volatility of the mix | sqrt(share_calm * 0.006**2 + share_turb * 0.018**2) | 1.088% |
| calm density at -3% | exp(-0.5 * (0.03 / 0.006)**2) / 0.006 | 0.00062 |
| turbulent density at -3% | exp(-0.5 * (0.03 / 0.018)**2) / 0.018 | 13.85 |
| probability of turbulent after a -3% day | 0.02 * lt / (0.02 * lt + 0.98 * lc) | 99.78% |
| calm density at -1.2% | exp(-0.5 * (0.012 / 0.006)**2) / 0.006 | 22.56 |
| turbulent density at -1.2% | exp(-0.5 * (0.012 / 0.018)**2) / 0.018 | 44.5 |
| probability of turbulent after a -1.2% day | 0.02 * lt2 / (0.02 * lt2 + 0.98 * lc2) | 3.87% |
| average age of data in a 20-day window | (20 - 1) / 2 | 9.5 |
Check yourself
A state persists from one day to the next with probability 0.95. What is its expected duration?
1 / (1 - 0.95) = 20 days.
Calm persists with probability 0.98 and turbulent with 0.95. What share of days is turbulent in the long run?
The calm-to-turbulent chance is 0.02 and the turbulent-to-calm chance is 0.05. The turbulent share is 0.02 / (0.02 + 0.05) = 28.6%.
A chart labels March 2020 as a crisis regime from the start. Why can that label not be used to trade on 5 March?
The start date was known only after later data confirmed the episode. On 5 March the label did not yet exist. Trading on it uses the future.
A 20-day trailing volatility is used as a regime signal. How old is the information in it on average?
(20 - 1) / 2 = 9.5 days. A trailing rule always reports a regime late.
A model finds three regimes in 10 years of data. Does that prove three regimes exist?
No. The number of states is a modelling choice. A sample with a few real episodes cannot separate three states from two with noise.
Key terms
Market regimeHalf-lifeExponentially weighted moving averageGARCH
Checked facts
Every figure below carries how it was checked: Primary checked (read in the cited primary, official, academic or data source), Two sources agree, or Approximate (an order of magnitude only).
- Primary checked
Hamilton (1989) modelled the parameters of an autoregression as the outcome of a hidden discrete-state Markov process and applied it to postwar US real GNP.
Sources
- A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle. Econometrica 57(2), 357-384 (James D. Hamilton), 1989. Accessed 9 Nov 2026
- Time Series Analysis. Princeton University Press (James D. Hamilton), 1994. Accessed 9 Nov 2026
- Empirical properties of asset returns: stylized facts and statistical issues. Quantitative Finance 1(2), 223-236 (Rama Cont), 2001. Accessed 9 Nov 2026