Hyperparameter Tuning Without Overfitting

Concept

Parameters are learned by fit(), such as linear coefficients. Hyperparameters are choices made before fitting, such as logistic-regression regularization strength C. Different settings can change underfitting, overfitting, and generalization.

Honest Tuning

Bad workflow: try a setting, inspect the test score, change it, and repeat. The test set then influences choices and no longer provides an independent final check.

Better workflow: use training/development data and cross-validation to compare settings, select one configuration, retrain appropriately, then evaluate once on the protected test set.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

search = GridSearchCV(
    LogisticRegression(max_iter=1000),
    param_grid={"C": [0.1, 1, 10]},
    cv=5,
)
search.fit(X_train, y_train)

Tuning optimizes the scoring rule supplied, so the wrong score can select the wrong model. Repeatedly trying many features, model families, and search spaces can also overfit the development process. The final test remains the safeguard.

Reading the Search

param_grid lists candidate values. cv=5 asks GridSearchCV to compare each configuration across five development folds. After fitting, search.best_params_ reports the selected configuration and search.best_score_ reports its cross-validation score under the chosen scoring rule. Neither value is the final production claim: retrain the selected configuration appropriately, then evaluate once on the protected test set.

The safe flow is development data -> cross-validation -> select settings -> final test. It prevents the common loop of checking test performance, changing C, and checking the same test again. Even CV can be overfit when enough experiments are tried, so document choices and keep the final check independent.

Parameters Versus Hyperparameters

Logistic regression learns its coefficients from the training examples during fit(). Those are parameters. C, the inverse regularization strength, is a hyperparameter: the team selects candidate values around training, then validation evidence helps choose one. Different values can make a model too constrained or too sensitive, connecting tuning to underfitting, overfitting, and generalization.

What GridSearchCV Is Doing

param_grid lists candidate C values. cv=5 evaluates each candidate across five development folds. scoring defines what "best" means. After fitting, best_params_ identifies the chosen setting and best_score_ is its cross-validation score under that rule. If the scoring rule ignores the business cost that matters, GridSearchCV can faithfully select the wrong model. The final test result is the independent check, not another tuning signal.

A Bad Workflow to Avoid

Configuration A -> inspect test score; configuration B -> inspect test score; configuration C -> inspect test score; choose the winner. This leaks test information through model selection even though no test row was used in fit(). Use development cross-validation instead, then evaluate the selected configuration once on the protected test set.

Practice Questions

1. What is the difference between a parameter and hyperparameter?
2. Why is tuning against the test score unsafe?
3. What does GridSearchCV combine?
4. Why must scoring match the real decision?
5. What do best_params_ and best_score_ represent?

Module 4 Synthesis

Evaluation is a connected process: baseline, metric, error types, accuracy limits, threshold behavior, cross-validation, tuning, and a protected test check. Module 5 will make this trustworthy through leakage-safe data preparation.