Train, Validation, Test Sets, and Generalization

Concept

The goal of supervised ML is generalization: useful performance on new examples from the kind of world the model will face. A model can remember training examples and still fail on new ones, so training performance alone is not an honest answer.

Roles of the Splits

The train set teaches the model. The validation set helps compare experimental choices, such as candidate models, without touching the final test data. The test set is a final unbiased check after major choices are made. Repeatedly changing a model after looking at test results turns the test set into another validation set and weakens its independence.

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.40, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, random_state=42
)

This creates training, validation, and test portions. The proportions are not universal: dataset size, time order, and grouped observations matter. The important principle is that evaluation data should represent cases the model did not learn from.

Why Protect the Test Set?

Imagine a team evaluates model A on its test set, sees disappointing results, changes features, evaluates model B on the same test set, tunes settings, and repeats. It never calls fit() on the test rows, but the test results still influenced each design choice. The team has adapted to that particular test set, so its final score is no longer an independent estimate of future performance.

Validation data exists to support development decisions. The test set stays protected until major choices are complete. In small projects the boundary can be imperfect, but the principle remains: do not repeatedly optimize against the result you plan to present as unbiased.

When Random Splits Need Caution

For fraud detection, a random split can put transactions from the same period on both sides while fraud patterns change over time. For customer churn, multiple rows from the same customer can make held-out performance look too optimistic if related records appear in training. Medical records may need patients, rather than individual visits, kept together. These examples clarify what "new" should mean; specialized split strategies come later.

Early Leakage Warning

Split before fitting a scaler or selecting features from outcomes. A scaler fit on every row has already learned about validation and test distributions. Likewise, repeatedly inspecting test performance to choose features leaks test information into decisions. Later lessons cover leakage-safe pipelines in detail.

Failure Signals

Common Mistakes

Evaluating only on training data.
Tuning repeatedly against the test set.
Assuming random splitting always fits time-based or grouped records.
Using future information to predict an earlier event.
Treating a high test score as a guarantee of real-world success.

Best Practices

Define what “new” means for the problem, keep the test set protected, and record which decisions used validation results. Real-world data can change after evaluation, so generalization is evidence, not a permanent guarantee.

Interview Perspective

Question: Why not use the test set while tuning? Answer: repeated tuning adapts choices to that test data, making its final estimate optimistic. Follow-up: when might a random split be inappropriate?

Practice Questions

Explain the role of validation data in one sentence.
Why is fitting a scaler before splitting risky?
A model scores well on training but poorly on test data. What concern does that raise?

Quick Quiz

Which split learns model parameters? Answer: train.
Which split is the final check? Answer: test.
Is 60/20/20 universal? Answer: no.

Key Takeaway

Key Takeaways

Generalization matters more than memorization. Train learns, validation guides choices, and test provides a protected final check on unseen data.

Next Lesson

Next, learn why models can be too simple or too adapted to training examples.