Boosting and Gradient Boosting Intuition

Concept

Bagging trains many learners largely independently and averages them. Boosting trains learners sequentially: each new learner improves what the current ensemble still gets wrong. Random Forest is not boosting.

For regression, a first shallow tree captures a broad pattern. Its residuals show remaining structure. The next tree learns part of that remaining error, and the ensemble updates its prediction. Gradient boosting generalizes this sequential correction using the model loss; no calculus is needed to understand the workflow.

from sklearn.ensemble import GradientBoostingClassifier

model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.05, random_state=42)
model.fit(X_train, y_train)

learning_rate controls how strongly each new tree contributes. Smaller steps commonly need more trees. Boosting can model nonlinearities and interactions strongly, but is more tuning-sensitive, sequential, and capable of overfitting.

A Staged Residual Example

For three homes, an initial model predicts 200, 200, 200; actual prices are 180, 230, 260. Residuals are -20, +30, +60. A small next tree can learn part of that remaining pattern, perhaps reducing predictions for the first kind of home and increasing them for larger homes. The ensemble adds the correction instead of discarding the first model. "Gradient" refers intuitively to moving predictions in a direction that reduces the chosen loss.

Bagging	Boosting
Learners train largely independently	Learners train sequentially
Diversity plus averaging/voting	Incremental error correction
Primarily reduces instability	Can reduce remaining structured error

Boosting commonly uses shallow trees as weak learners. Smaller learning_rate values make each correction gentler and often require more n_estimators. Too many corrections or overly flexible learners can overfit, so use cross-validation rather than assuming boosting is always superior. XGBoost, LightGBM, and CatBoost are related industry implementations, not dependencies in this course.

Practice Questions

1. How does boosting differ from bagging?
2. What does a later boosted tree focus on?
3. What happens when learning rate is smaller?
4. Why can boosting overfit?

Next Lesson

Next, choose among tree approaches and interpret importance carefully.