P85Model EvaluationAdvancedDiagnose16 min
Diagnose High AUROC but Poor Alert Precision
Diagnose why strong ranking quality can coexist with poor precision at an operational threshold.
#diagnose#auroc#precision
Scenario
A fraud model has AUROC 0.93 on data with 1% fraud. At the deployed threshold it creates 1,000 alerts: 70 are fraud and 930 are false positives.
Problem statement
Explain the apparent conflict and identify checks or changes before declaring the model useful.
Your Task
- Separate ranking quality from threshold behavior.
- Use prevalence and false positives in the explanation.
- Recommend threshold- and workload-aware checks.
Related Concepts
AUROCPrecision-recallPrevalence
Ready to count this problem?
Mark it complete when you have reasoned through the solution in your own words.