Skip to main content

Quant Analytics: from scratch to pro · Professional practice and failure case studies · Advanced

Model risk and live monitoring

Every model can be wrong, or right but used wrongly. Model risk management adds independent challenge, validation and ongoing monitoring, including a simple live-versus-expected band that says when results have drifted too far from what was assumed.

4 min read Reviewed Revision 1

In this lesson

  • Define model risk and name its two main sources
  • Describe the parts of model risk management in the US guidance (SR 11-7, replaced in 2026 by SR 26-2): development, validation and governance
  • Build a live-versus-expected band from backtest assumptions
  • Compute a z-score for a live result and decide what it does and does not show
  • Account for the false alarms that repeated monitoring produces

Read first: Pre-registration and trial ledgers, Volatility explained

Chance of at least one false alarm by number of checks

0% 20% 40% 60% 80% 100% 12510203050 Independent checks
Chance of a false alarm. Computed from the two-sided false-alarm chance of a 2-standard-deviation band, assuming independent checks and that the model is correct.

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
expected daily mean return from the backtest0.00050.05%
daily standard deviation assumed0.011%
days live6060 days
expected cumulative returnmu * days3%
standard deviation of the cumulative returnsd_day * sqrt(days)7.75%
lower edge of a 2-standard-deviation bandexp_cum - 2 * sd_cum-12.49%
upper edge of the bandexp_cum + 2 * sd_cum18.49%
live cumulative return-0.09-9%
z-score of the live result(live - exp_cum) / sd_cum-1.55
chance of a result this low or lower if assumptions holdphi(z)6.1%
z-score if the live result were -14%(-0.14 - exp_cum) / sd_cum-2.19
two-sided false-alarm chance of a 2-standard-deviation band2 * phi(-2)4.55%
chance of at least one false alarm in 20 independent checks1 - (1 - alarm)**2060.6%

Check yourself

Name the two main sources of model risk.

The model may have errors or be built on poor assumptions, so its outputs are inaccurate for its purpose. Or it may be used incorrectly, outside the purpose or conditions it was designed for.

Expected daily mean is 0.05%, daily volatility is 1%. After 60 days, what are the expected cumulative return and its standard deviation?

Expected: 60 x 0.05% = 3%. Standard deviation: 1% x sqrt(60) = 7.75%.

The live result after 60 days is -9%. What is the z-score and what does it mean?

z = (-9% - 3%) / 7.75% = -1.55. A result this low or lower happens about 6.1% of the time if the assumptions hold, so it is a warning, not proof of failure.

Why does a 2-standard-deviation alarm trigger often if checked every day for a month?

Each check has a small false-alarm chance (4.55% two-sided). Over 20 independent checks the chance of at least one alarm is 60.6%. Use a wider band for frequent checks, and predefine the action.

What is effective challenge?

Critical analysis of a model by objective, informed people who can find its limits and assumptions and get changes made. It needs competence, independence and the standing to be heard.

Key terms

Model riskEffective challengeStandard deviation

Sources

  • SR 11-7: Supervisory Guidance on Model Risk Management. Board of Governors of the Federal Reserve System, with the OCC, 2011. Accessed 9 Nov 2026
  • SR 26-2: Revised Guidance on Model Risk Management. Board of Governors of the Federal Reserve System, with the FDIC and the OCC, 2026. Accessed 9 Nov 2026
  • The Statistics of Sharpe Ratios. Financial Analysts Journal 58(4), 36-52 (Andrew W. Lo), 2002. Accessed 9 Nov 2026

Informational content. Not investment advice.