Decision Trees
A decision tree splits data on feature thresholds into a flowchart of rules, readable and nonlinear but prone to overfitting.
Learning by splitting
A decision tree predicts by asking a sequence of yes/no questions about features, following branches to a leaf that holds the answer. Training grows the tree greedily: at each node it picks the feature and threshold whose split best separates the target, then recurses on each side.
Split criteria
- Classification: minimize Gini impurity or entropy; both reward pure child nodes.
- Regression: minimize the variance (squared error) within each child.
- The best split maximizes the information gain, the drop in impurity from parent to children.
Entropy measures disorder in a node's class mix; information gain is how much a split reduces it. See entropy and information gain for the arithmetic.
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(max_depth=4, min_samples_leaf=20)
clf.fit(X_train, y_train)Strengths
Trees need no feature scaling, handle mixed numeric and categorical data, capture nonlinear interactions, and are easy to read as rules. Feature importances fall out naturally from how much each feature reduces impurity.
The overfitting problem
Left to grow fully, a tree memorizes the training set: deep trees have high variance and generalize poorly. Control this by limiting depth, requiring a minimum number of samples per leaf, or pruning back after growth. Even so, a single tree is unstable; small data changes reshape it. Ensembles fix this: random forests average many trees, and gradient boosting stacks them, both far outperforming a lone tree.