Quant Analytics: from scratch to pro · Professional practice and failure case studies · Advanced
Model risk and live monitoring
Every model can be wrong, or right but used wrongly. Model risk management adds independent challenge, validation and ongoing monitoring, including a simple live-versus-expected band that says when results have drifted too far from what was assumed.
In this lesson
- Define model risk and name its two main sources
- Describe the parts of model risk management in the US guidance (SR 11-7, replaced in 2026 by SR 26-2): development, validation and governance
- Build a live-versus-expected band from backtest assumptions
- Compute a z-score for a live result and decide what it does and does not show
- Account for the false alarms that repeated monitoring produces
Read first: Pre-registration and trial ledgers, Volatility explained
A model is a simplified description of the world used to make a decision. Because it is simplified, it can be wrong. Because people reuse it, it can also be used where it does not fit. The cost of either is model risk, and it applies to a spreadsheet of position sizes as much as to a bank's pricing system.
What the US guidance says
The US Federal Reserve's supervisory guidance SR 11-7 (2011) became the common reference. In April 2026 the Fed, the FDIC and the OCC replaced it with SR 26-2, Revised Guidance on Model Risk Management. The 2011 text describes model risk as arising mainly from two sources: the model may have fundamental errors and give inaccurate outputs for its purpose, or it may be used incorrectly or outside what it was built for. Its framework has three parts:
- Development and implementation: clear purpose, sound data and theory, testing, documentation.
- Validation: an independent evaluation of conceptual soundness, ongoing monitoring and outcomes analysis, which compares the model's forecasts with what happened. Validation continues after the model goes into use.
- Governance: roles, policies and an inventory of models, with limits on use.
The guiding idea is effective challenge: critical analysis by objective, informed people who can identify the model's limits and assumptions and get changes made. Independence and standing matter as much as skill.
The guidance was written for banks. The same logic fits any individual or team running a rule-based approach: someone other than the builder reviews it, and its behaviour is checked after launch.
Sources of model risk in trading models
- Fitting noise: the live edge is smaller than the backtest, because of overfitting.
- Changed conditions: volatility, costs or correlations differ from the sample.
- Wrong inputs: stale data, bad prices, unit or time-zone errors.
- Implementation faults: the code does not do what the document says. The Knight Capital case is an extreme example.
- Misuse: a model built for one market or size is applied to another.
The live-versus-expected band
Before launch, write down what the backtest implies about live results: an expected mean and a volatility, and a band around the cumulative result. After launch, compare.
Assume a backtest mean of 0.05% a day and a daily standard deviation of 1%. After 60 days:
- expected cumulative return:
0.05% x 60, which is 3%; - standard deviation of the cumulative return:
1% x sqrt(60), which is 7.75% (returns scale with time, volatility with its square root); - a 2-standard-deviation band: from -12.49% to 18.49%.
Worked example: a z-score
The live result after 60 days is -9%. That is inside the band. The z-score is (-9% - 3%) / 7.75%, which is -1.55. If the assumptions hold, a result this low or lower occurs about 6.1% of the time. A loss of 9% looks bad and is still within ordinary variation for this model. Had the live result been -14%, the z-score would be -2.19, which is outside the band and calls for review.
Three cautions:
- A z-score tests the assumptions as a bundle. A breach says something is off: edge, volatility, costs or implementation. It does not say which.
- Short windows have little power. Telling an edge from zero can take years of data (see the research process), so a band catches large failures quickly and small ones slowly.
- Repeated checking produces false alarms. A 2-standard-deviation band has a two-sided false-alarm chance of 4.55% per check. Over 20 independent checks the chance of at least one alarm is 60.6%.
What to do on a breach
Decide the action before launch: reduce size, pause, or review. A breach starts a review of data, costs and code first, since faults are more common than edges that vanish. Do not retune the model on the live period without recording it as a new trial.
Common mistakes
- Setting the band after seeing live results.
- Using the backtest's maximum drawdown as the band instead of a distribution.
- Ignoring volatility drift. If realised volatility doubles, the band is wrong before the edge is.
- Reviewing only the builder's own work.
- Treating "inside the band" as proof the model is sound.
- Monitoring only returns. Fill prices, slippage, data lateness and rejected orders reveal faults earlier.
In practice
Keep a model inventory, a named reviewer who is not the builder, and a one-page monitoring plan with the band, the checks and the actions. Research on FinUties is where studies built on methods like these are published, with their method and limits stated.
Chance of at least one false alarm by number of checks
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| expected daily mean return from the backtest | 0.0005 | 0.05% |
| daily standard deviation assumed | 0.01 | 1% |
| days live | 60 | 60 days |
| expected cumulative return | mu * days | 3% |
| standard deviation of the cumulative return | sd_day * sqrt(days) | 7.75% |
| lower edge of a 2-standard-deviation band | exp_cum - 2 * sd_cum | -12.49% |
| upper edge of the band | exp_cum + 2 * sd_cum | 18.49% |
| live cumulative return | -0.09 | -9% |
| z-score of the live result | (live - exp_cum) / sd_cum | -1.55 |
| chance of a result this low or lower if assumptions hold | phi(z) | 6.1% |
| z-score if the live result were -14% | (-0.14 - exp_cum) / sd_cum | -2.19 |
| two-sided false-alarm chance of a 2-standard-deviation band | 2 * phi(-2) | 4.55% |
| chance of at least one false alarm in 20 independent checks | 1 - (1 - alarm)**20 | 60.6% |
Check yourself
Name the two main sources of model risk.
The model may have errors or be built on poor assumptions, so its outputs are inaccurate for its purpose. Or it may be used incorrectly, outside the purpose or conditions it was designed for.
Expected daily mean is 0.05%, daily volatility is 1%. After 60 days, what are the expected cumulative return and its standard deviation?
Expected: 60 x 0.05% = 3%. Standard deviation: 1% x sqrt(60) = 7.75%.
The live result after 60 days is -9%. What is the z-score and what does it mean?
z = (-9% - 3%) / 7.75% = -1.55. A result this low or lower happens about 6.1% of the time if the assumptions hold, so it is a warning, not proof of failure.
Why does a 2-standard-deviation alarm trigger often if checked every day for a month?
Each check has a small false-alarm chance (4.55% two-sided). Over 20 independent checks the chance of at least one alarm is 60.6%. Use a wider band for frequent checks, and predefine the action.
What is effective challenge?
Critical analysis of a model by objective, informed people who can find its limits and assumptions and get changes made. It needs competence, independence and the standing to be heard.
Key terms
Sources
- SR 11-7: Supervisory Guidance on Model Risk Management. Board of Governors of the Federal Reserve System, with the OCC, 2011. Accessed 9 Nov 2026
- SR 26-2: Revised Guidance on Model Risk Management. Board of Governors of the Federal Reserve System, with the FDIC and the OCC, 2026. Accessed 9 Nov 2026
- The Statistics of Sharpe Ratios. Financial Analysts Journal 58(4), 36-52 (Andrew W. Lo), 2002. Accessed 9 Nov 2026