Predictions, Residuals, and Regression Loss

Concept

A regression model produces a prediction, written y_hat, while the dataset contains the actual target, y. Their difference is a residual:

residual = actual - predicted

Residuals show where a model is high or low for individual observations. A model also needs a summary of those misses to decide which fitted relationship is better. That summary is a metric; the quantity an algorithm seeks to reduce while fitting is often called loss.

Why It Matters

Predictions rarely match real outcomes exactly. A demand forecast can miss because of a promotion, weather, or noise in the data. Looking only at a single score can hide systematic problems. Residuals help ask whether errors are random, unusually large for one segment, or consistently biased in one direction.

A Small Example

Actual sales	Predicted sales	Residual: actual - predicted
120	110	10
90	105	-15
150	145	5

A positive residual means the prediction was too low. A negative residual means the prediction was too high. The signs matter for diagnosis: if a demand model has mostly positive residuals for weekends, it may systematically underestimate weekend demand.

Residual, Metric, and Loss

A residual belongs to one observation. A metric summarizes prediction quality across many observations, such as mean absolute error (MAE). A loss is the mathematical quantity the fitting procedure may minimize to select model parameters. These ideas overlap but are not identical: a business team may report MAE while linear regression commonly fits coefficients using squared residuals.

MAE averages abs(actual - predicted). It treats a miss of 10 as twice as bad as a miss of 5. Squared error squares each miss first, so a miss of 10 contributes four times as much as a miss of 5. This stronger penalty makes large misses matter more during fitting.

Least-Squares Intuition

In simple linear regression, many candidate lines are possible. Each line makes different predictions, producing different residuals. Least squares chooses coefficient values that make the total squared residuals as small as possible. You do not need the closed-form calculation to understand the goal: select the line that is collectively closest to the observed targets under a squared-error objective.

Squared loss is not a universal business choice. It is useful here because it gives linear regression a clear fitting objective and strongly reacts to large errors. Module 4 returns to selecting and interpreting regression metrics for real decisions.

Code: Inspecting Residuals

results = pd.DataFrame({
    "actual": y_test,
    "predicted": predictions,
})
results["residual"] = results["actual"] - results["predicted"]
results["absolute_error"] = results["residual"].abs()

print(results)

This code assumes the y_test and predictions from the earlier workflow remain aligned. Inspecting the table helps turn a single MAE number into questions: which cases are far off, are errors mostly positive or negative, and do difficult cases share a pattern?

MAE and RMSE: Similar Averages Can Hide Different Risks

MAE treats error magnitude approximately linearly. RMSE gives larger misses extra weight because errors are squared before averaging. If two models have similar MAE but one has a noticeably worse RMSE, that often suggests a smaller number of much larger mistakes.

For example, Model A has MAE = 8.1 and RMSE = 11.3; Model B has MAE = 7.8 and RMSE = 14.8. Model B is slightly better on typical error, but its larger RMSE suggests rare but more severe misses. Neither metric is universally better. Ask whether those large misses occur in costly, safety-critical, or important customer cases before choosing a model.

Applied Assumptions: Diagnose Consequences

For linear regression to be a useful predictor, the chosen features need to represent enough of the relationship for held-out errors to be acceptable. A straight or additive form may miss curvature or interactions. Related observations over time or within one customer group can make validation look more independent than it really is. Changing error spread and influential observations can hide unequal reliability. Correlated predictors can also make individual coefficients unstable to interpret.

These concerns do not mean every imperfect assumption requires rejecting the model. They are evidence to inspect and communicate. Stronger assumptions are needed when making formal statistical-inference claims about coefficient uncertainty; predictive usefulness is primarily judged through appropriate held-out evaluation, diagnostics, and decision context.

Best Practices

Check residuals on held-out data, examine both typical and large errors, and preserve their link to original rows. Describe error in target units when communicating with stakeholders. A model can have a reasonable average error while failing badly for an important customer segment.

Interview Perspective

Question: What is a residual? Answer: the actual target minus the model prediction for one observation. What the interviewer is testing: whether you can connect a prediction equation to model diagnosis. Follow-up: why might squared error emphasize large misses?

Practice Questions

1. Calculate the residual when actual demand is 80 and predicted demand is 95.
2. What does a consistently positive residual for a product category suggest?
3. Why is MAE easier to explain than a squared-error value?
4. Which contributes more squared error: a miss of 20 or two misses of 10? Explain.
5. Why should residuals be inspected on held-out rows?

Quick Quiz

1. What does a negative residual mean under this convention? Answer: the model predicted too high.
2. Is MAE an individual residual or a summary metric? Answer: a summary metric.
3. What does least squares seek to reduce? Answer: the total squared residuals.

Next Lesson

Next, extend the line from one feature to several and interpret coefficients carefully.