Decision Trees: Classification and Regression Trees for Actuaries

How Decision Trees Work

A decision tree recursively partitions the predictor space into non-overlapping regions and assigns a prediction to each region. The algorithm works by considering all possible binary splits on all predictors at each step, selecting the split that best improves a chosen criterion. The process repeats within each resulting region until a stopping condition is met (e.g., minimum number of observations in a node, maximum depth, or no further improvement).

For regression trees, the prediction in each leaf (terminal node) is the mean of the response variable Y for observations in that region. For classification trees, the prediction is the most common class (majority vote). The result is a tree structure where internal nodes represent split conditions (e.g., "Driver Age > 35" or "Territory in {Urban, Suburban}") and leaf nodes represent predictions.

Trees are popular in insurance because they are intuitive and easy to interpret. A claims analyst can follow the tree from root to leaf and understand exactly why a particular prediction was made. This transparency is valuable for regulatory purposes and for communicating with non-technical stakeholders.

Splitting Criteria for Regression Trees

For regression trees, the splitting criterion is the reduction in the residual sum of squares (RSS). At each node, consider splitting predictor X_j at threshold s into two regions: R_1 = {X | X_j <= s} and R_2 = {X | X_j > s}. The predictions are the means y_bar_1 and y_bar_2 in each region. The RSS for this split is sum_{i in R_1} (y_i - y_bar_1)^2 + sum_{i in R_2} (y_i - y_bar_2)^2.

The algorithm searches over all predictors j and all possible split points s to find the split that minimizes the total RSS. This greedy approach does not guarantee the globally optimal tree but is computationally feasible and works well in practice.

For a worked example: consider a node with 6 observations of claim severity: {200, 250, 300, 800, 900, 1000}. The mean is 408.3, and the total RSS is 556,667. If we split at 500 (values <= 500 go left, > 500 go right), the left node has mean 250 with RSS 5,000, and the right node has mean 900 with RSS 20,000. The total RSS after splitting is 25,000, a reduction of 531,667. This large reduction indicates a useful split.

Impurity Measures for Classification Trees

For classification trees with K classes, the proportion of class k in a node is p_k. Two primary impurity measures are used for growing the tree.

The Gini index: G = sum_{k=1}^{K} p_k * (1 - p_k) = 1 - sum_{k=1}^{K} p_k^2. For a binary classification: G = 2 * p * (1 - p). The Gini index equals zero when the node is pure (all observations belong to one class) and is maximized at 0.5 (for binary) when classes are equally represented. A split is evaluated by the weighted average of Gini indices in the child nodes, and the split that produces the largest decrease in Gini is chosen.

Cross-entropy (deviance): D = -sum_{k=1}^{K} p_k * ln(p_k). Like the Gini index, cross-entropy is zero for pure nodes and maximized for equal class proportions. In practice, Gini and cross-entropy produce very similar trees.

The misclassification rate: E = 1 - max_k(p_k). While intuitive, the misclassification rate is not recommended for tree growing because it is less sensitive to changes in class probabilities. Consider a node with p = 0.5 that splits into children with p = 0.4 and p = 0.6. The misclassification rate is 0.5 in the parent and 0.4 in both children (weighted average 0.4), a decrease of 0.1. But the Gini index decreases from 0.50 to 0.48, and cross-entropy from 0.693 to 0.673. The Gini and cross-entropy are smoother, making them better objective functions for optimization.

Pruning and Overfitting

A fully grown tree, where splitting continues until each leaf contains very few observations (or even one observation), perfectly fits the training data but generalizes poorly to new data. This is because the tree memorizes noise in the training set rather than learning the true underlying pattern. This is the overfitting problem.

Cost-complexity pruning is the standard approach to finding the right tree size. Define the cost-complexity criterion as C_alpha(T) = sum_{m=1}^{|T|} RSS_m + alpha * |T|, where |T| is the number of terminal nodes and alpha >= 0 is the complexity parameter. For alpha = 0, the fully grown tree minimizes C_alpha. As alpha increases, simpler trees are preferred because each additional terminal node must earn a reduction in RSS that exceeds alpha.

For each value of alpha, there exists a unique smallest subtree T(alpha) that minimizes C_alpha. The sequence of subtrees (from the full tree down to the root) forms a nested set. Cross-validation selects the optimal alpha: for each candidate alpha, compute the cross-validated error, and choose the alpha (and corresponding subtree) with the lowest CV error. The "one standard error rule" selects the simplest tree with CV error within one standard error of the minimum.

Advantages and Limitations

Decision trees have several advantages. They require no assumptions about the functional form of the relationship between predictors and response. They handle interactions naturally (a split on X2 within a branch defined by X1 is an interaction). They accommodate both numeric and categorical predictors without preprocessing. And the results are easily visualized and interpreted.

The main limitation is high variance. Small changes in the training data can produce very different tree structures, because early splits affect the entire downstream structure. This instability is the primary motivation for ensemble methods (bagging, random forests, and boosting), which combine many trees to reduce variance.

Other limitations include the greedy splitting algorithm (which may miss globally optimal trees), the tendency to create axis-aligned splits (which may not capture oblique boundaries efficiently), and the inability to extrapolate beyond the range of the training data (predictions for extreme values are the same as for the nearest training region).

Actuarial Applications

In insurance, decision trees are used for risk classification, claim triage (routing claims to the appropriate handler based on characteristics), fraud detection (identifying suspicious patterns), and underwriting (approve/decline decisions). Trees are particularly useful in exploratory analysis for discovering interactions between rating factors that might be missed by additive models like GLMs.

A common actuarial workflow uses trees for variable selection and interaction discovery, then builds a GLM incorporating those findings. This combines the tree's ability to find complex patterns with the GLM's smooth, well-calibrated predictions.

Study Tips for Exam SRM

For Exam SRM, focus on these key topics. First, understand the splitting criteria (RSS for regression, Gini and cross-entropy for classification) and be able to compute the impurity reduction for a proposed split. Second, know the cost-complexity pruning procedure and the role of cross-validation in selecting alpha. Third, understand the bias-variance perspective: unpruned trees have low bias but high variance, while pruned trees trade some bias for lower variance. Fourth, be able to compare trees with other methods (GLMs, random forests) in terms of interpretability, accuracy, and assumptions. Exam questions commonly present a small dataset and ask you to determine the best split, or present a pruned and unpruned tree and ask about the tradeoff.