Quant Analytics: from scratch to pro · Machine learning in finance · Pro
Calibration, Brier score and drift monitoring
A model that outputs probabilities must be right about how often it is right. Calibration checks that, the Brier score summarises probability errors, and the population stability index flags when live inputs no longer resemble the training data.
In this lesson
- Compute a Brier score and compare it with the base-rate forecast
- Build a reliability table and compute a calibration error
- Compute the population stability index for a feature
- Set up monitoring that separates drift in inputs from a loss of accuracy
Read first: Machine learning as a filter: meta-labelling, How sure is an average? Standard error and confidence intervals
A classifier used for trading seldom needs a yes or no. It needs a probability: how likely is this signal to succeed? That number may set a threshold, a size or a ranking. If the probabilities are wrong, everything built on them is wrong. Two checks matter: are the probabilities honest, and do they stay honest over time?
The Brier score
Brier (1950) introduced a score for probability forecasts: the mean squared difference between the forecast p and the outcome o, which is 1 if the event occurred and 0 if not.
Brier = mean((p - o)^2)
It runs from 0 (perfect) to 1. It punishes confident mistakes and rewards forecasts that match true frequencies. Its expected value is lowest when the forecast equals the true probability, as the chart shows. See Brier score.
Worked example: Brier
Five forecasts are 0.8, 0.6, 0.3, 0.7 and 0.2. Outcomes are 1, 1, 0, 0, 0. The squared errors are 0.04, 0.16, 0.09, 0.49 and 0.04, and their mean is 0.164.
A score alone means little. Compare it with the forecast that always says the base rate. If the event happens 40% of the time, that forecast scores 0.4 x 0.6^2 + 0.6 x 0.4^2, which is 0.24. The Brier skill score is 1 - 0.164 / 0.24, which is 31.7%. With five points this is an illustration; real evaluation needs hundreds of outcomes.
Calibration
A forecaster is calibrated when, among cases forecast at 70%, about 70% occur. Check it with a reliability table: sort forecasts into bins, then compare the average forecast with the observed frequency in each.
| Bin | Cases | Mean forecast | Observed | Gap |
|---|---|---|---|---|
| Low | 500 | 0.30 | 0.28 | 0.02 |
| Middle | 300 | 0.50 | 0.42 | 0.08 |
| High | 200 | 0.70 | 0.55 | 0.15 |
This model is overconfident, most of all where it is most sure. The weighted average gap, the expected calibration error, is (500 x 0.02 + 300 x 0.08 + 200 x 0.15) / 1,000, which is 0.064. A model can rank cases well and be badly calibrated; ranking is not enough when probabilities set the size of a bet. Overconfidence is common in flexible models fitted on noisy data. A calibration map fitted on training folds only can correct it.
Drift
Markets change, and a model fitted on the past may stop describing the present. Outcomes arrive late, so the first warning comes from the inputs. The population stability index (PSI) compares the share of cases in each bin of a feature now (actual) with the training period (expected):
PSI = sum over bins of (actual - expected) x ln(actual / expected)
Worked example: PSI
A feature was split in five bins of 20% each in training. Now the bins hold 10%, 15%, 20%, 25% and 30%. The terms are 0.0693, 0.0144, 0, 0.0112 and 0.0405, and PSI is 0.135. A common rule of thumb calls values below 0.1 stable, 0.1 to 0.25 a moderate shift and above 0.25 a major shift. These cut-offs are convention, not statistical tests.
A monitoring routine
- Track PSI for each important feature, and for the model's output.
- Track Brier score and calibration on recent outcomes, with enough cases to be meaningful.
- Compare live trade count and precision with the validation values; see meta-labelling.
- Fix in advance what triggers a review, a refit or a stop. A written threshold removes the temptation to explain away drift.
- Keep the model's version, training window and validation results, as the US supervisory guidance on model risk expected (SR 11-7, 2011).
Common mistakes
- Reporting a Brier score without the base-rate score.
- Calibrating on the data used to fit the model.
- Treating PSI cut-offs as laws.
- Waiting for outcomes to confirm a shift the inputs already show.
- Refitting at every wobble, which turns monitoring into overfitting.
In practice
A probability is a claim that can be checked. Keep a record of forecasts and outcomes from the first day, and review calibration and input drift on a schedule. Analytics on FinUties applies measures like these to the markets it covers, so they are at hand when you read a chart. The wider process is in model risk and live monitoring.
Expected Brier score against the forecast, event probability 0.4
Worked numbers
Each result is recomputed from the formula shown.
| What | Formula | Result |
|---|---|---|
| squared error 1 | (0.8 - 1) ** 2 | 0.04 |
| squared error 2 | (0.6 - 1) ** 2 | 0.16 |
| squared error 3 | (0.3 - 0) ** 2 | 0.09 |
| squared error 4 | (0.7 - 0) ** 2 | 0.49 |
| squared error 5 | (0.2 - 0) ** 2 | 0.04 |
| Brier score | (e1 + e2 + e3 + e4 + e5) / 5 | 0.164 |
| Brier score of always forecasting the base rate 0.4 | 0.4 * 0.6 ** 2 + 0.6 * 0.4 ** 2 | 0.24 |
| Brier skill score | 1 - brier / brier_base | 31.7% |
| calibration gap, low bin | 0.30 - 0.28 | 0.02 |
| calibration gap, high bin | 0.70 - 0.55 | 0.15 |
| expected calibration error | (500 * 0.02 + 300 * 0.08 + 200 * 0.15) / 1000 | 0.064 |
| PSI term, bin 1 | (0.10 - 0.20) * ln(0.10 / 0.20) | 0.0693 |
| PSI term, bin 2 | (0.15 - 0.20) * ln(0.15 / 0.20) | 0.0144 |
| PSI term, bin 4 | (0.25 - 0.20) * ln(0.25 / 0.20) | 0.0112 |
| PSI term, bin 5 | (0.30 - 0.20) * ln(0.30 / 0.20) | 0.0405 |
| population stability index | psi1 + psi2 + psi4 + psi5 | 0.135 |
Check yourself
Forecasts are 0.8, 0.6, 0.3, 0.7, 0.2 and outcomes 1, 1, 0, 0, 0. What is the Brier score?
Squared errors: 0.04, 0.16, 0.09, 0.49, 0.04. The sum is 0.82 and the mean is 0.164.
The event happens 40% of the time. What Brier score does always forecasting 0.4 earn, and how much better is the model above?
0.4 x 0.6^2 + 0.6 x 0.4^2 = 0.144 + 0.096 = 0.24. Brier skill = 1 - 0.164 / 0.24 = 31.7%.
A bin of 200 forecasts averages 0.70 but 55% of the cases occur. What is wrong?
The model is overconfident in that bin by 15 percentage points. Used for sizing or thresholds, it overstates the odds.
A feature's five equal bins now hold 10%, 15%, 20%, 25% and 30% of cases against 20% each in training. What is the PSI, and how is it read?
PSI = sum of (actual - expected) x ln(actual / expected) = 0.135. A common rule of thumb reads below 0.1 as stable, 0.1 to 0.25 as a moderate shift, above 0.25 as a major one. It is a flag, not a test.
PSI is high but live accuracy is unchanged. Should you ignore it?
Not entirely. Inputs have moved, which may or may not affect the target. Check calibration and accuracy on recent outcomes; outcomes arrive late, so input drift is the early warning.
Key terms
Sources
- Verification of forecasts expressed in terms of probability. Monthly Weather Review 78(1), 1-3 (Glenn W. Brier), 1950. Accessed 9 Nov 2026
- Advances in Financial Machine Learning. Wiley (Marcos López de Prado), 2018. Accessed 9 Nov 2026
- The Elements of Statistical Learning (2nd edition). Springer (Trevor Hastie, Robert Tibshirani, Jerome Friedman), 2009. Accessed 9 Nov 2026
- SR 11-7: Supervisory Guidance on Model Risk Management. Board of Governors of the Federal Reserve System, with the OCC, 2011. Accessed 9 Nov 2026