ROC-AUC and Threshold Trade-offs

Concept

Changing a threshold changes which probabilities become positive labels. Lower thresholds create more predicted positives; higher thresholds create fewer. That changes true positives, false positives, true negatives, and false negatives.

ROC and AUC

Each threshold produces a true-positive-rate and false-positive-rate pair. A ROC curve summarizes those pairs across thresholds. AUC summarizes how well scores rank positives above negatives across thresholds. It is not the probability that an individual prediction is correct, and it does not choose an operating threshold.

A Threshold Sweep

Consider ordered fraud scores 0.91, 0.72, 0.51, 0.38, and 0.12. At 0.80, only the first case is positive. At 0.50, the first three are positive. At 0.30, four are positive. As the threshold moves downward, some actual fraud cases become true positives while some legitimate cases can become false positives. Each setting is one operating point on the ROC curve.

True-positive rate asks how much of the actual positive class is detected; false-positive rate asks how often actual negatives are incorrectly labeled positive. AUC summarizes separation across all such settings. It can be strong while the threshold a team can afford is still unhelpful, so AUC complements rather than replaces operating decisions.

Case	Actual class	Score	Positive at 0.70	Positive at 0.50	Positive at 0.30
A	Fraud	0.91	Yes	Yes	Yes
B	Legitimate	0.72	Yes	Yes	Yes
C	Fraud	0.51	No	Yes	Yes
D	Legitimate	0.38	No	No	Yes
E	Legitimate	0.12	No	No	No

At 0.70 the model finds one fraud case but also flags one legitimate case. At 0.50 it finds the second fraud case, without adding another false positive in this toy example. At 0.30 it flags an additional legitimate transaction. Real scores produce many more points, but the reasoning is the same: every threshold selects a different operational balance.

Use ROC Carefully

roc_curve returns arrays of false-positive rates, true-positive rates, and associated thresholds. Each array position is an operating point, not a recommendation. Under severe imbalance, a precision-recall perspective can sometimes describe operational burden more directly, but it still does not replace understanding the action and error costs. Select thresholds during development or validation, then keep the final test set protected.

from sklearn.metrics import roc_auc_score, roc_curve

auc = roc_auc_score(y_test, positive_scores)
fpr, tpr, thresholds = roc_curve(y_test, positive_scores)

Practical Interpretation

A strong AUC can still be operationally weak at the threshold a fraud team can afford. Threshold choice must reflect costs, review capacity, and validation evidence. In imbalanced settings, also inspect class-focused behavior; the existing Accuracy lesson introduces precision-recall measures.

Capacity and Cost Can Change the Evaluation Target

Suppose a retention team can contact only 2,000 of 100,000 customers each week. The decision may not be "which threshold maximizes F1?" It may be "which 2,000 customers should receive outreach?" In that case, ranking quality and precision or recall at the top 2,000 can be more relevant than one global threshold metric. This is capacity-constrained or top-k evaluation: it connects model output to the actual amount of action the team can take.

Costs can justify a lower threshold even when precision declines. If a missed fraudulent transaction has an expected $500 loss and an unnecessary review costs $8, accepting more reviews may be rational. Those values are business assumptions, not fixed facts; estimate them, test sensitivity, and revisit them as operations change.

ROC-AUC, Precision-Recall, and the Operating Point

ROC-AUC measures discrimination: how well a model tends to rank positive cases above negative cases across many thresholds. It does not choose a production threshold, guarantee calibrated probabilities, or state the operational cost at one threshold. A model with higher ROC-AUC can still be less useful at the workload a team can support.

Precision-recall curves, and PR-AUC when used as a summary, can be especially informative when positives are rare and the quality of positive predictions matters. They do not automatically replace ROC-AUC; compare metrics that answer the deployment question and inspect behavior at the intended operating point.

Calibration: Are Scores Meaningful Probabilities?

Discrimination asks whether higher-risk cases tend to rank above lower-risk cases. Calibration asks whether predicted probabilities correspond reasonably to observed frequencies. Among cases predicted near 0.80, a well-calibrated model would have roughly 80% positive outcomes over a suitable comparable group.

Two models can rank customers nearly identically while carrying different probability meaning. If Model A's customers scored near 0.80 churn about 80% of the time, but Model B's similar-scored customers churn only 55% of the time, Model B may still rank well but its probabilities are poorly calibrated. Thresholding asks where to act; calibration asks whether the score itself is interpretable as risk. A model can have useful ranking or a useful threshold while still needing calibration work.

Calibration matters most when probability magnitude feeds expected financial loss, medical-risk communication, insurance pricing, inventory decisions, or resource allocation. If a team uses scores only to rank a fixed top-k list, ranking may matter more, but calibration can still affect how confidently the team interprets risk.

Practice Questions

1. What happens to predicted positives as threshold falls?
2. Does AUC set the best threshold? Why not?
3. Why keep final test data out of threshold selection?
4. What does ROC summarize?
5. Why can good AUC still be operationally insufficient?

Next Lesson

Next, compare models without trusting one fortunate validation split.