Glossary
Calibration
Agreement between predicted probabilities and observed frequencies: among cases forecast at 70%, about 70% should occur.
A forecaster is calibrated when its stated probabilities match long-run frequencies. It is checked with a reliability table: group forecasts into bins and compare the mean forecast with the observed rate in each bin.
A model can rank cases well and still be badly calibrated, which matters when probabilities are used for sizing or for thresholds.
Example: 200 forecasts averaging 0.70 of which only 110 occur (55%) are overconfident by 15 percentage points.