Decision Trees: Splits, Predictions, and Overfitting

Concept

A decision tree learns human-readable questions that divide rows into useful groups. A churn tree might ask whether a contract is monthly, then whether tenure is below six months. The first question is the root, later questions are internal nodes, paths are branches, and leaves are final prediction regions.

For classification, a leaf may predict the dominant class or class proportions. For regression, it predicts a numeric value based on target values in that leaf. The model tests candidate splits and favors child groups with more similar classes or numeric targets. Gini impurity and entropy are names for classification split-quality measures; the key intuition is cleaner child groups.

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(max_depth=3, random_state=42)
model.fit(X_train, y_train)
predictions = model.predict(X_test)

Trees naturally represent thresholds, nonlinear patterns, and interactions such as monthly contract and short tenure. Deep trees can memorize noise, outliers, and tiny groups: excellent training performance with weaker validation performance signals possible overfitting. max_depth, min_samples_split, and min_samples_leaf limit this growth.

Trees usually do not need scaling because a split such as age < 35 depends on ordering; scaling changes the threshold number but not row order. Preparation and valid features still matter.

Walk Through a Tree

Imagine a root split: contract_type == monthly. The yes branch asks tenure_months < 6; its yes branch asks monthly_charge > 80 and ends in a leaf predicting high churn risk. A monthly customer with two months of tenure and a $95 charge follows those branches to that leaf. A yearly-contract customer follows the no branch and may reach a low-risk leaf. The tree learns these questions from training data; it is not a hand-written causal policy.

Shallow trees create broad leaves and may miss useful structure. Deep trees create increasingly specific leaves, eventually modeling tiny groups or noise. max_depth caps question layers, min_samples_split prevents splitting very small groups, and min_samples_leaf requires each final region to contain enough training rows. Validate these choices: a perfect training tree can still generalize poorly.

Monotonic rescaling preserves feature ordering, so the same candidate rows remain on either side of a threshold even if 35 becomes a standardized value. Trees therefore usually need no scaling for split mechanics, but missing values, categories, leakage, and inconsistent inference inputs still require preparation.

Gini = 1 - (p0^2 + p1^2)

Greedy Choices, Instability, and Validation

Trees usually grow greedily: at each node, they choose a strong split for the current training data rather than exhaustively searching every possible future tree. A locally attractive first split is not proof of a globally optimal decision rule.

This makes an individual tree high variance. With one training sample, age <= 42 might be the first split; after a few rows or noisy observations change, income <= 58k might become first instead. Both trees can perform similarly while having very different downstream structures. Interpret one fitted tree as a learned summary of one sample, not as a fixed truth about the domain.

Watch for near-perfect training performance with much weaker validation performance, tiny leaves, important splits supported by few rows, large structure changes across resamples, or validation performance that changes sharply with depth. max_depth, min_samples_split, and min_samples_leaf control complexity, but no parameter setting is magic. Choose them with representative validation evidence and the cost of mistakes in mind.

Practice Questions

1. Identify root, branch, and leaf in a churn rule.
2. Why can a deep tree overfit?
3. Why is a readable rule not causal proof?
4. Why is scaling usually less important for trees?

Next Lesson

Next, stabilize trees by averaging many varied versions.