Cross-Validation and Comparing Models

Concept

One validation split can favor a model by chance. K-fold cross-validation divides development data into K folds, trains on K-1 folds, validates on the remaining fold, rotates, and summarizes the scores. The protected final test set remains separate.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X_train, y_train, cv=5, scoring="neg_mean_absolute_error")

Compare models on the same data, target, scoring rule, and folds. Inspect the average and variation: a slightly better average with wildly unstable folds deserves caution. Classification folds often preserve class proportions through stratification.

Boundaries

Ordinary K-fold may be wrong for time-ordered data or repeated customer records. The principle is to make validation resemble genuinely new cases. Never cross-validate over the final test set.

Why One Split Can Mislead

Model A can beat Model B on one validation split simply because that split contains easier cases. Across five folds, Model A might score consistently near its average while Model B swings widely depending on which observations it sees. Cross-validation therefore gives both an average estimate and evidence of variation. Compare candidates with the same folds, metric, target, and preprocessing boundaries; otherwise the comparison is not fair.

For classification, stratification keeps class proportions similar across folds when feasible. For time-ordered data, random folds can let the future inform the past. For repeated customer rows, related records can leak across folds. These cases require a scheme matching the data, while the final test set remains outside every development choice.

Read Fold Scores, Not Only the Average

Suppose Model A produces 0.81, 0.79, 0.82, 0.80, 0.81. Model B produces 0.84, 0.73, 0.86, 0.75, 0.83. Model B may have a similar or slightly higher average, but its variation is much larger. That can indicate sensitivity to which examples land in training and validation. It does not automatically disqualify Model B, but it raises a useful question about stability and data representativeness.

K-fold works step by step: divide development data into K folds; train on K-1; validate on the remaining fold; rotate the held-out fold; collect all scores; then summarize them. The final test set is not one of the K folds. It remains untouched until the development choices are complete.

Code Interpretation

cross_val_score fits a fresh copy of the estimator for each fold. The returned scores are development evidence, not final production performance. Use the same metric and fold scheme for Model A and Model B so a difference reflects models rather than a changed experiment.

Deep Dive

Deep Dive: Choose the Split That Resembles Deployment

Cross-validation is not only a way to split rows several times. Its central question is: what information would genuinely be unavailable when this prediction is made? A validation fold should represent that future or new-use situation as closely as practical.

Data structure	Main risk from a random row split	Better validation idea
Approximately independent rows from one stable population	No obvious relationship crosses folds.	Ordinary shuffled K-fold can be reasonable.
Imbalanced classification target	A fold may contain an unrepresentative class proportion.	Stratified folds preserve a useful class balance.
Repeated customers, patients, devices, or households	The model can see the same entity in training and validation.	Keep each entity together with grouped splitting.
Time-dependent observations	Training can accidentally use future information.	Use a chronological holdout or rolling / expanding-window validation.
Repeated entities over time	Both entity dependence and future information can cross folds.	Reason about group and temporal boundaries together.

Ordinary shuffled K-fold is reasonable when observations are approximately independent, no meaningful time order must be preserved, the same entity cannot appear across folds in a leakage-causing way, and deployment resembles new samples from the same population. It is not a universal default.

Grouped and Repeated Observations

Consider a healthcare dataset with many visits per patient. If visits from one patient appear in both training and validation folds, a model can learn patient-specific patterns and look stronger than it will be for an unseen patient. Grouped cross-validation keeps all rows for one logical entity in the same fold. The entity might be a customer, patient, device, store, household, account, or machine.

Rows can also be technically different but highly related: multiple images of one person, repeated measurements from one machine, transactions from one customer, or augmented versions of one sample. Choose the unit of independence before choosing the split. The correct boundary depends on deployment: predicting another visit for a known patient is a different task from predicting for a new patient.

Time Must Flow Forward

For next-month demand prediction, training on December and validating on October creates an unrealistic information flow. A chronological holdout trains on earlier data and evaluates on a later period. Rolling or expanding-window validation repeats that idea across several later windows: train on the past, then evaluate on what comes next.

Time-aware validation does not mean time-series rows can never be shuffled in every context. It means the split must match the prediction time and the information actually available then. Calendar features, target definitions, feature freshness, and deployment cadence all matter.

Stratification Solves One Narrow Problem

For an imbalanced classification target, stratification helps each fold keep a useful approximation of the overall class proportion. That makes fold scores more comparable when positives are rare. Stratification does not fix grouped dependence, future leakage, feature/preprocessing leakage, or production distribution shift. It addresses class balance, not the whole validation design.

Feature or preprocessing leakage happens when validation or test information influences learned transformations or features. Split-design leakage happens when the fold structure lets related or future information cross the boundary. Both can create overly optimistic validation results.

Decision Lab

Decision Lab: Churn Snapshots Across Customers and Time

A subscription company has 400,000 monthly customer snapshots. Each customer appears in many months, and churn behavior changes over time. The team considers: A) random row-level 5-fold CV, B) grouped CV by customer, or C) time-based validation using earlier months for training and later months for validation.

Random row-level CV can be too optimistic because the same customer can appear in both training and validation, and later information may be mixed with earlier rows. Grouped CV addresses customer overlap, so it is useful when deployment targets entirely new customers. Time-based validation addresses future-to-past leakage and is often closer to a system that scores the current customer population for a future period.

Both customer grouping and time can matter. If deployment predicts future churn for existing customers, a time-respecting split with carefully defined customer history may be the most faithful design. If it predicts churn for newly acquired customers, the design must also prevent customer overlap. State the intended deployment unit before declaring one option best.

Stability, Tuning, and the Final Test

Do not automatically choose the highest mean CV score when the margin is small and one model varies sharply by fold. Large fold-to-fold variation, a changing model ranking, or one unusually difficult fold can reveal sensitivity to data composition. Inspect the fold scores, the split design, and the cost of instability alongside the mean.

Cross-validation results influence choices when they select a model family, features, or hyperparameters. Reusing those results repeatedly can overfit the development process, so keep the final test set untouched until major choices are complete. More rigorous workflows can use nested CV, but the practical rule here is simpler: tune on development evidence and use the protected test set for a final independent estimate.

Cross-Validation Cannot Guarantee Production Performance

CV estimates performance under the validation distribution. It cannot rescue a dataset that does not represent deployment conditions. A change in geography, customer population, acquisition channel, market conditions, or policy can create distribution shift after launch. Monitor the real data and revisit assumptions when the deployment setting changes.

Failure Signals

Validation Warning Signs

Treat implausibly high CV performance, the same entity appearing across folds, future information in training, large fold-to-fold variation, changing model rankings, a collapse on chronological holdout, or final-test performance far below CV as reasons to investigate. Also ask whether the folds resemble the population and decision the model will face after deployment.

Check Your Reasoning

Question: A failure-prediction model uses thousands of sensor readings from each machine. Random 5-fold CV is excellent, but performance collapses on newly installed machines. What validation-design problem might explain this, and what split would you try next?

Reasoning: Readings from the same machine may have crossed folds, allowing the model to learn machine-specific patterns instead of generalizing. Try grouped validation by machine when deployment targets new machines. If the model will predict future readings for known machines, also inspect whether a time-respecting split better matches the real prediction moment.

Practice and Build

Practice and Build Connection

Use Detect Data Leakage Before Deployment to practice prediction-time reasoning. Demand Forecasting for Retail reinforces chronological validation, while End-to-End Customer Retention Modeling requires choosing and justifying random or chronological splits before model comparison.

Practice Questions

Why can one split mislead?
What data stays outside cross-validation?
Why compare models using the same folds?
When might ordinary K-fold be inappropriate?
Why should two candidate models use the same folds?

Key Takeaway

Key Takeaways

Cross-validation gives model comparison more than one view of development data while preserving an independent final test.

Next Lesson

Next, tune hyperparameters without adapting choices to the test set.