Decision Trees

If you have ever played the game "20 Questions" — where you narrow down possibilities by asking yes/no questions — you already understand the intuition behind decision trees. A decision tree classifier works by asking a series of questions about the input features, with each question splitting the data into smaller, more homogeneous groups until it reaches a prediction. Among all machine learning algorithms, decision trees are arguably the most human-readable and interpretable, making them a favorite in industries where you need to explain why a model made a certain prediction.

The Intuition: Thinking Like a Doctor

Imagine a doctor diagnosing whether a patient has the flu. The doctor does not run every possible test simultaneously. Instead, they ask sequential questions: "Do you have a fever?" If yes, "Is it above 101°F?" If yes, "Do you have body aches?" Each answer narrows down the diagnosis. A decision tree does exactly this — it learns which questions to ask and in what order by analyzing training data.

The beauty of this approach is that the resulting model is a flowchart that anyone can follow. No linear algebra, no probability theory needed to understand the prediction — just follow the branches from root to leaf.

How Decision Trees Split Data

The core challenge in building a decision tree is deciding which feature to split on at each node. The algorithm needs a way to measure which split creates the most "pure" child nodes — groups where most data points belong to the same class.

Entropy and Information Gain

Entropy measures the disorder or impurity in a group. A group with all same-class examples has zero entropy (perfectly pure). A group with 50-50 split has maximum entropy (maximum uncertainty).

The mathematical formula for entropy is:

Information Gain measures how much entropy decreases after splitting on a particular feature. The feature with the highest information gain is chosen for the split:

The intuition behind this is simple: we want each split to reduce uncertainty as much as possible. If splitting on "age > 50" separates your data into one group that is 95 percent positive and another that is 90 percent negative, that is an excellent split because both children are nearly pure.

Gini Impurity — The Practical Alternative

While entropy uses logarithms, Gini impurity achieves similar results with simpler computation:

Scikit-learn uses Gini by default because it is computationally cheaper and produces similar trees to entropy in practice. The key insight is that both metrics agree on what constitutes a good split — they just measure purity slightly differently.

The CART Algorithm: Building Trees Step by Step

The Classification and Regression Trees (CART) algorithm builds decision trees using a greedy, top-down approach:

1. Start with all training data at the root node
2. For each feature, find the best split threshold (the value that maximizes information gain or minimizes Gini impurity)
3. Select the feature and threshold that produces the best split overall
4. Create two child nodes — one for data points satisfying the condition, one for the rest
5. Recursively repeat for each child until a stopping criterion is met
6. Assign the majority class of training examples at each leaf as the prediction

The Overfitting Problem and Pruning

Here is the critical weakness of decision trees: without constraints, they will keep splitting until every leaf contains a single training example, perfectly memorizing the training data but failing miserably on new data. This is severe overfitting.

Pre-Pruning (Limiting Growth)

Stop the tree from growing too deep in the first place:

• max_depth: Maximum number of levels from root to leaf
• min_samples_split: Minimum examples needed before a node can split
• min_samples_leaf: Minimum examples required in each leaf
• max_features: Only consider a random subset of features at each split

Post-Pruning (Cost-Complexity Pruning)

Grow the full tree first, then trim branches that do not improve validation performance. Scikit-learn implements this via the ccp_alpha parameter — higher alpha means more aggressive pruning.

Advantages and Limitations

Strengths: Decision trees require no feature scaling, handle both numerical and categorical data naturally, provide built-in feature importance rankings, produce interpretable models that non-technical stakeholders can understand, and serve as building blocks for powerful ensemble methods like Random Forest and Gradient Boosting.

Weaknesses: Single decision trees are prone to overfitting, unstable (small data changes can produce very different trees), biased toward features with many levels, and typically less accurate than ensemble methods. They create axis-aligned splits only, struggling with diagonal decision boundaries.

Real-World Applications

Decision trees shine in scenarios requiring interpretability: credit approval decisions (banks must explain why), medical triage protocols, fraud detection rule systems, and customer segmentation. They are also the foundation of Random Forest and Gradient Boosting — the most successful algorithms in structured data competitions.

Key Takeaways

Decision trees learn by asking optimal yes/no questions that maximize information gain at each split. They are wonderfully interpretable but prone to overfitting without proper pruning. Use max_depth, min_samples_leaf, and cross-validated ccp_alpha to control complexity. While a single tree rarely wins competitions, understanding trees deeply is essential because ensemble methods (Random Forest, XGBoost) are simply collections of decision trees working together. Master the single tree, and you have the foundation for the most powerful algorithms in tabular data machine learning.