Skip to main content

Quant Analytics: from scratch to pro · Machine learning in finance · Pro

Calibration, Brier score and drift monitoring

A model that outputs probabilities must be right about how often it is right. Calibration checks that, the Brier score summarises probability errors, and the population stability index flags when live inputs no longer resemble the training data.

4 min read Reviewed Revision 1

In this lesson

  • Compute a Brier score and compare it with the base-rate forecast
  • Build a reliability table and compute a calibration error
  • Compute the population stability index for a feature
  • Set up monitoring that separates drift in inputs from a loss of accuracy

Read first: Machine learning as a filter: meta-labelling, How sure is an average? Standard error and confidence intervals

Expected Brier score against the forecast, event probability 0.4

0.2 0.3 0.4 0.5 0.6 00.20.40.60.81 Forecast probability
Expected Brier score. Computed from the formula, not from data. The expected score is lowest when the forecast equals the true probability, which is why the Brier score rewards honest probabilities.

Worked numbers

Each result is recomputed from the formula shown.

WhatFormulaResult
squared error 1(0.8 - 1) ** 20.04
squared error 2(0.6 - 1) ** 20.16
squared error 3(0.3 - 0) ** 20.09
squared error 4(0.7 - 0) ** 20.49
squared error 5(0.2 - 0) ** 20.04
Brier score(e1 + e2 + e3 + e4 + e5) / 50.164
Brier score of always forecasting the base rate 0.40.4 * 0.6 ** 2 + 0.6 * 0.4 ** 20.24
Brier skill score1 - brier / brier_base31.7%
calibration gap, low bin0.30 - 0.280.02
calibration gap, high bin0.70 - 0.550.15
expected calibration error(500 * 0.02 + 300 * 0.08 + 200 * 0.15) / 10000.064
PSI term, bin 1(0.10 - 0.20) * ln(0.10 / 0.20)0.0693
PSI term, bin 2(0.15 - 0.20) * ln(0.15 / 0.20)0.0144
PSI term, bin 4(0.25 - 0.20) * ln(0.25 / 0.20)0.0112
PSI term, bin 5(0.30 - 0.20) * ln(0.30 / 0.20)0.0405
population stability indexpsi1 + psi2 + psi4 + psi50.135

Check yourself

Forecasts are 0.8, 0.6, 0.3, 0.7, 0.2 and outcomes 1, 1, 0, 0, 0. What is the Brier score?

Squared errors: 0.04, 0.16, 0.09, 0.49, 0.04. The sum is 0.82 and the mean is 0.164.

The event happens 40% of the time. What Brier score does always forecasting 0.4 earn, and how much better is the model above?

0.4 x 0.6^2 + 0.6 x 0.4^2 = 0.144 + 0.096 = 0.24. Brier skill = 1 - 0.164 / 0.24 = 31.7%.

A bin of 200 forecasts averages 0.70 but 55% of the cases occur. What is wrong?

The model is overconfident in that bin by 15 percentage points. Used for sizing or thresholds, it overstates the odds.

A feature's five equal bins now hold 10%, 15%, 20%, 25% and 30% of cases against 20% each in training. What is the PSI, and how is it read?

PSI = sum of (actual - expected) x ln(actual / expected) = 0.135. A common rule of thumb reads below 0.1 as stable, 0.1 to 0.25 as a moderate shift, above 0.25 as a major one. It is a flag, not a test.

PSI is high but live accuracy is unchanged. Should you ignore it?

Not entirely. Inputs have moved, which may or may not affect the target. Check calibration and accuracy on recent outcomes; outcomes arrive late, so input drift is the early warning.

Key terms

CalibrationBrier scoreConcept driftMeta-labelling

Sources

  • Verification of forecasts expressed in terms of probability. Monthly Weather Review 78(1), 1-3 (Glenn W. Brier), 1950. Accessed 9 Nov 2026
  • Advances in Financial Machine Learning. Wiley (Marcos López de Prado), 2018. Accessed 9 Nov 2026
  • The Elements of Statistical Learning (2nd edition). Springer (Trevor Hastie, Robert Tibshirani, Jerome Friedman), 2009. Accessed 9 Nov 2026
  • SR 11-7: Supervisory Guidance on Model Risk Management. Board of Governors of the Federal Reserve System, with the OCC, 2011. Accessed 9 Nov 2026

Informational content. Not investment advice.