Regression Metrics for Real Decisions

Concept

Residuals explain individual misses; regression metrics summarize many held-out predictions. No metric is universally best because error costs differ by decision.

MAE, MSE, and RMSE

MAE averages absolute error and stays in target units. If delivery-time MAE is 1.5 hours, stakeholders can interpret it directly. MSE squares errors, so a miss of 20 contributes four times as much as a miss of 10. RMSE is the square root of MSE, returning to target units while still emphasizing large misses.

For errors 2, 2, and 10, MAE is 14 / 3, while RMSE is larger because the 10-unit miss receives extra weight. Choose that emphasis deliberately: severe late deliveries may justify stronger attention to large misses.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

mae = mean_absolute_error(y_test, predictions)
mse = mean_squared_error(y_test, predictions)
rmse = mse ** 0.5
r_squared = r2_score(y_test, predictions)

R-Squared Carefully

R-squared compares a model with a simple target-mean reference. Near 1 can indicate a strong fit; near 0 means little improvement over that reference; it can be negative on held-out data when the model is worse. R-squared is not "percent accuracy" and does not say whether errors are acceptable in dollars or hours.

Use One Error Set to Compare

For errors 2, 2, and 10, MAE is (2 + 2 + 10) / 3 = 4.67. MSE is (4 + 4 + 100) / 3 = 36, and RMSE is the square root of that value, about 6. The large miss changes RMSE more than MAE. For house-price estimates, MAE may be easiest to discuss in dollars. For emergency delivery estimates, a rare large delay may justify paying more attention to RMSE. Metric choice expresses what the decision treats as costly.

Actual hours	Predicted hours	Error	Absolute error	Squared error
10	8	2	2	4
12	14	-2	2	4
20	10	10	10	100

The table shows why squared metrics react differently: the 10-hour miss dominates MSE and RMSE, while MAE treats it as five times a two-hour miss. RMSE returns to hours after the square root; MSE remains in squared hours, so it is less intuitive to explain directly. For demand forecasting, large stockouts may justify stronger large-error sensitivity. For house prices, absolute dollar error may be clearer to stakeholders. There is no metric that is correct without a decision context.

Read Metric Code With Context

The scikit-learn functions summarize y_test and aligned predictions; they do not decide whether the error is acceptable. Always compare against a baseline, inspect important segments, and use held-out data. R-squared near zero means little improvement over predicting the target mean, while a negative held-out value means the model performed worse than that reference.

Failure Signals

Common Mistakes

Treating the lowest metric as automatically best for every decision.
Calling R-squared an accuracy percentage.
Comparing errors across targets with different units without context.
Selecting metrics from training data alone.

Interview Perspective

Question: When might RMSE be preferable to MAE? Answer: when large misses deserve disproportionately more attention. What the interviewer is testing: decision-aware metric choice. Follow-up: why is R-squared not accuracy?

Practice Questions

Which metric is easiest to communicate for house-price error?
Why does RMSE react more to a large miss?
What does negative held-out R-squared suggest?
Pick a metric emphasis for a severe delivery-delay problem.
Calculate MAE for errors of 1, 3, and 8.

Quick Quiz

Which metrics share the target's unit? Answer: MAE and RMSE.
Does R-squared equal percent accuracy? Answer: no.
What does MSE do to large errors? Answer: squares them.

Key Takeaway

Key Takeaways

Metrics encode priorities. Use held-out MAE, RMSE, and R-squared as complementary evidence, then connect the choice to the decision.

Next Lesson

Next, inspect classification outcomes through a confusion matrix.