Comparing Ensembles and Feature Importance Carefully

Model Choices

A shallow tree is simple and inspectable but unstable. Random Forest is a robust nonlinear baseline with less variance. Gradient boosting can achieve strong structured-data performance but needs more careful tuning. No model is universally best: compare with consistent metrics, cross-validation, and the final protected test set.

Tree ensembles expose feature_importances_, reflecting how strongly splits using each feature reduced impurity or error. High importance means this fitted model relied on a feature for prediction; it does not prove changing that feature causes the target to change.

Correlated inputs can share or steal importance. For example, annual and monthly income overlap, so a model may favor one arbitrarily. Impurity-based measures can also favor features with many possible split points. Permutation importance shuffles one feature and measures held-out performance change, but correlated features and compute cost still limit interpretation.

from sklearn.inspection import permutation_importance

importance = permutation_importance(model, X_test, y_test, random_state=42)

Choose a tree for a small understandable rule system, Random Forest for a reliable low-tuning benchmark, or boosting when validated predictive gains justify added complexity. Importance guides investigation, not causal claims.

Compare Models by the Decision

Model	Strength	Trade-off
Shallow tree	Transparent rules	Unstable and often weaker
Random Forest	Stable nonlinear baseline	More compute, less inspectable
Gradient boosting	Strong flexible performance	More tuning-sensitive and sequential

Feature importance from feature_importances_ summarizes how much splits using a feature reduced impurity or error in that fitted model. If annual_income and monthly_income encode the same signal, one may receive most importance while the other appears unimportant. Features with many possible split points can also be favored. High importance is model-specific reliance, never evidence that changing the feature causes the target.

Permutation importance starts with a held-out baseline score, shuffles one feature, then measures performance change. A large drop suggests the model depended on that column. It offers a different perspective, but correlated features can mask each other, repeated shuffling has randomness and compute cost, and it remains non-causal. Select models with the same validation scheme, not importance rankings alone.

Decision Lab

Decision Lab: Stable Predictions, Unstable Explanations

A churn team compares two models:

Model	Training accuracy	Validation accuracy	Importance finding
Decision Tree	99%	78%	`support_calls` dominates one fitted tree.
Random Forest	94%	84%	Across folds, `support_calls`, `tenure`, and `recent_usage` trade importance.

The large training-validation gap suggests the single tree may be fitting training-specific detail. Small changes in the training rows can change an early split and the rest of a high-variance tree, so its apparently dominant feature may be unstable. The forest can generalize better because bootstrap sampling, feature subsampling, and averaging reduce the impact of any one unstable tree when their errors are not perfectly correlated.

Do not conclude that support_calls causes churn. Inspect whether the feature is available at prediction time, whether its importance is stable across validation folds, whether correlated predictors redistribute importance, and whether removing or grouping related features changes held-out behavior. A feature ranking is evidence about this fitted model's reliance under this dataset and validation design, not a business explanation by itself.

Failure Signals

Importance Warning Signs

Investigate when one feature dominates unexpectedly, importance changes substantially across folds or runs, a high-cardinality or identifier-like field appears important, or a split is supported by very few observations. Impurity-based importance can favor variables with many possible split points, such as continuous values or identifier-like fields. Permutation importance offers a different question - how much held-out performance changes when one feature is shuffled - but correlated features can mask each other and it remains non-causal.

Check Your Reasoning

Question: A fraud Random Forest gives transaction_id the highest feature importance and achieves excellent validation performance. What should you investigate before trusting the model?

Reasoning: Check whether the identifier carries hidden ordering, leakage, duplicated entities, or encoding artifacts; whether the validation split keeps related transactions appropriately separate; whether importance is stable; and whether performance remains after removing or sanitizing the ID. High importance does not establish causality or validity.

Practice and Build

Practice and Build Connection

Use Diagnose an Overfitting Model to investigate train-validation gaps and Choose a Tree, Random Forest, or Boosting Model to reason through stability, tuning, and interpretability trade-offs. End-to-End Customer Retention Modeling provides the broader pipeline, model-comparison, and error-analysis setting.

Practice Questions

When is a shallow tree useful?
Why can correlated features distort importance?
Does high importance prove causation?
What does permutation importance test?
Why compare ensembles with the same CV scheme?

Module Synthesis

Single trees learn splits; bagging and Random Forest average variation; boosting adds sequential correction; importance requires careful interpretation. Next, Unsupervised Learning explores patterns without a known target.

Key Takeaway

Key Takeaways

Predictive strength and interpretable importance are not causal evidence. Choose ensembles from validated trade-offs.