Extending and Controlling Linear Models

Concept

A simple straight line can underfit a curved relationship. We can extend a linear model with transformed features, such as x_squared, while still learning coefficients in a linear equation. We can also control a model when it becomes too flexible or unstable through regularization.

Linear Does Not Always Mean Straight

Suppose advertising spend and sales rise quickly at first, then level off. A straight line may miss that curve. Add a squared feature and the model can represent a bend:

prediction = b0 + b1 * x + b2 * x_squared

The relationship with x can now be curved, but the model remains linear in its learned coefficients b0, b1, and b2. This distinction explains why polynomial regression belongs with linear models.

Polynomial Features in Code

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

sales = pd.DataFrame({
    "ad_spend": [1, 2, 3, 4, 5],
    "revenue": [12, 25, 39, 50, 57],
})

X = sales[["ad_spend"]]
y = sales["revenue"]

polynomial = PolynomialFeatures(degree=2, include_bias=False)
X_polynomial = polynomial.fit_transform(X)

model = LinearRegression()
model.fit(X_polynomial, y)

PolynomialFeatures creates columns for x and x_squared. The code demonstrates the idea, not a complete production workflow. In a real project, transformations must be fit consistently on training data and applied to validation/test data; the later preprocessing module teaches the safer workflow.

Flexibility and Generalization

Adding terms gives a model more ways to match training data. That can help when a real pattern is curved, but a high-degree polynomial can bend around random noise. Module 1's diagnostic still applies: excellent training performance with much worse validation performance is a warning about overfitting. More features are not automatically better.

Use the simplest model that captures a useful held-out pattern. Compare models with appropriate validation evidence rather than choosing the most complicated curve because it looks best on training data.

Target Transformations Need a Diagnostic Reason

Targets such as revenue, home price, and claim amount are often right-skewed: a small number of large values sit far above most observations. If residual spread grows sharply with prediction scale, modeling log(target) can sometimes reduce the influence of extreme scale, stabilize variance, and make a multiplicative relationship easier to represent.

This is not an automatic fix. A model trained on a transformed target learns and measures errors on that transformed scale, so predictions must be converted back and interpreted carefully in the original units. First identify the residual pattern, then compare a transformation against the original model with the same held-out evaluation and business decision in mind.

Let Diagnostics Suggest, Not Dictate, Alternatives

Residual evidence should generate a hypothesis, not an automatic upgrade. A smooth curve can justify testing a transformed feature or low-degree polynomial. A subgroup-specific shift can suggest a missing feature or interaction. Unequal spread can motivate examining the target scale, segmenting the problem, or using an uncertainty-aware approach. A tree-based model may capture nonlinear structure, but it should earn its complexity through fair validation and an improvement in the failure pattern that matters.

Regularization Intuition

Regularization adds a preference for smaller, less extreme coefficients while fitting a model. It does not remove the need for good data or validation; it is one tool for reducing unnecessary sensitivity to a training sample.

Ridge regression shrinks coefficients toward zero. It usually keeps all features in the model, but reduces the influence of unusually large coefficients. Lasso regression also shrinks coefficients and can set some exactly to zero, effectively excluding them from the fitted equation. Both trade a little training fit for a model that may generalize better.

The regularization strength controls this trade-off. Very little regularization behaves more like ordinary linear regression. Too much regularization can underfit by shrinking useful relationships too strongly. Selecting that strength carefully belongs later with validation and hyperparameter tuning.

Why Feature Scale Matters

Regularization reacts to coefficient magnitude. If one feature is measured in dollars and another in thousands, their coefficient sizes are not directly comparable. Features are commonly scaled before regularized models, but scaling and leakage-safe preprocessing are separate Module 5 topics. The important idea now is to notice the dependency rather than applying regularization blindly.

When to Use These Ideas

Use polynomial features when a simple plot and domain reasoning suggest a smooth curve that a straight line cannot capture. Consider Ridge or Lasso when a linear model has many features, unstable coefficients, or signs of overfitting. Neither technique proves causation or replaces examining residuals and validation performance.

Best Practices

Start with a simple model, inspect error patterns, and add flexibility only when held-out evidence supports it. Keep the business range in mind: polynomial extrapolation outside observed x values can behave implausibly. Record transformations and validate model choices without repeatedly using the final test set.

Interview Perspective

Question: What is the difference between Ridge and Lasso? Answer: both shrink coefficients, while Lasso can shrink some exactly to zero and Ridge usually retains all features. What the interviewer is testing: whether you understand regularization as a generalization trade-off rather than a magic feature-selection method. Follow-up: why does scaling matter for regularization?

Practice Questions

1. Why can a model with x and x_squared produce a curved prediction?
2. A fifth-degree polynomial has excellent training error and poor validation error. What concern does this raise?
3. Describe one situation where Ridge may be useful.
4. What does it mean when Lasso sets a coefficient to zero?
5. Why should a polynomial model not be trusted far outside its observed feature range?

Quick Quiz

1. Is polynomial regression linear in its coefficients? Answer: yes.
2. What does regularization discourage? Answer: excessively large or unnecessarily flexible coefficient patterns.
3. Does a stronger regularization setting always help? Answer: no; too much can underfit.

Regression Module Synthesis

You can now frame a regression problem, interpret a linear prediction, inspect residuals and loss, combine multiple features, read coefficients carefully, model some curved patterns, and use regularization intuition to protect generalization. Regression predicts numeric values; the next module, Classification, predicts classes and probabilities.

Next Lesson

Next, begin Module 3 and learn how classification turns features into class predictions and probabilities.