P36Model EvaluationAdvancedDiagnose16 min
Diagnose Poor Probability Calibration
Decide whether predicted probabilities can be trusted for risk prioritization.
#calibration#probabilities#reliability-curve#thresholds
Scenario
A risk model assigns 0.80 probability to a set of accounts, but only 45% actually default. Ranking is good, but teams use probabilities to allocate reserves.
Problem statement
Explain the calibration problem and outline a safe validation and correction approach.
Your Task
- Separate ranking quality from probability accuracy.
- Use a calibration curve and held-out data.
- Name calibration methods and their deployment caveat.
Related Concepts
Reliability curvesPlatt scalingDistribution shift
Ready to count this problem?
Mark it complete when you have reasoned through the solution in your own words.