Random Forests and Bagging

Concept

One deep tree can change substantially when training data changes. Bagging trains many models on bootstrap samples: rows sampled with replacement, so some rows repeat and some are absent in each sample. Classification combines votes or probabilities; regression averages predictions. This primarily reduces variance, not every source of error.

Random Forest adds feature randomness: each split considers only a random feature subset. If every tree always uses the same dominant feature, trees are too similar and averaging helps less. Less-correlated trees create a more stable ensemble.

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(X_train, y_train)

n_estimators is the number of trees. Random Forest is often a strong tabular baseline, handles nonlinearities and interactions, and usually needs little scaling. It is less interpretable, larger, and can still overfit.

Why Diversity Stabilizes Trees

A single customer near a candidate split can change which feature becomes a tree's first question. Bootstrap sample one may include that row twice, while sample two may omit it. Sampling with replacement means each sample draws from the original rows repeatedly, creating varied training views. Each tree learns independently; classification combines votes or averaged probabilities, and regression averages numeric predictions.

Random Forest adds random feature subsets at each split. If income dominates every split, identical trees all repeat its decisions and averaging contributes little. Feature randomness decorrelates trees, so their individual mistakes are less likely to occur together. Averaging varied high-variance trees therefore produces a more stable result, though it does not remove bias or guarantee good validation performance.

Use a shallow tree when a small transparent rules system matters. Use a forest when a stronger nonlinear tabular baseline is more valuable than inspecting one complete tree. max_depth still limits each tree; max_features controls candidate-feature diversity.

Practice Questions

1. What does sampling with replacement mean?
2. Why does tree diversity matter?
3. How do classification and regression forests combine predictions?
4. Is Random Forest boosting? Why not?

Next Lesson

Next, contrast independent averaging with sequential correction.