Random Forests and Ensemble Methods for Exam SRM

The Motivation for Ensemble Methods

A single decision tree is easy to interpret but suffers from high variance: small changes in the training data can produce dramatically different tree structures. Ensemble methods address this by combining many trees, trading interpretability for improved prediction accuracy and stability. The two main ensemble strategies are bagging (which reduces variance) and boosting (which reduces bias). This post covers bagging and its extension, random forests.

The mathematical foundation for why averaging works is straightforward. If you have B independent random variables, each with variance sigma^2, the variance of their average is sigma^2 / B. By averaging B trees, you reduce the prediction variance by a factor of B. In practice, the trees are not fully independent (they are trained on overlapping bootstrap samples), but the variance reduction is still substantial.

Bootstrap Aggregating (Bagging)

Bagging works as follows: (1) draw B bootstrap samples from the training data (each sample is the same size as the original, drawn with replacement), (2) fit a full, unpruned decision tree to each bootstrap sample, and (3) combine the predictions by averaging (regression) or majority vote (classification).

Each bootstrap sample contains approximately 63.2% of the unique original observations (since the probability of not being selected in n draws is (1 - 1/n)^n, which approaches 1/e = 0.368 for large n). The remaining 36.8% are called out-of-bag (OOB) observations for that particular bootstrap sample.

OOB observations provide a built-in estimate of test error without requiring a separate validation set. For each training observation i, compute its prediction using only the trees for which observation i was OOB. The resulting OOB error is an approximately unbiased estimate of the test error. This is a significant practical advantage of bagging, especially when the dataset is too small to afford a dedicated test set.

Bagging is most effective for high-variance, low-bias base learners. Deep, unpruned trees are ideal because they have low bias (they can capture complex patterns) but high variance (they are sensitive to the training data). Bagging dramatically reduces the variance while preserving the low bias. For low-variance models like linear regression, bagging provides little benefit.

From Bagging to Random Forests

Bagging has a limitation: when one predictor is very strong, most trees will use it for the first split, making the trees highly correlated. Correlated trees provide less variance reduction when averaged (the variance of the average of B correlated variables with pairwise correlation rho is rho * sigma^2 + (1 - rho) * sigma^2 / B, which approaches rho * sigma^2 rather than zero as B grows).

Random forests solve this by decorrelating the trees. At each split in each tree, only a random subset of m predictors (out of p total) is considered as candidates. This forces each tree to use different predictors for its early splits, making the trees less correlated and the ensemble average more effective.

The typical choices for m are: m = sqrt(p) for classification and m = p/3 for regression. These defaults work well in practice, though m can be tuned by cross-validation or OOB error. Setting m = p recovers bagging (all predictors available at every split). Smaller m produces more decorrelation but also increases the bias of individual trees, so the optimal m balances these effects.

Random Forest Hyperparameters

The key hyperparameters for random forests are: the number of trees B (more is generally better, with diminishing returns; typically 500 to 2000), the number of predictors considered at each split m (sqrt(p) or p/3 as defaults), the minimum node size (controls tree depth; smaller values allow deeper trees), and the maximum number of terminal nodes.

Random forests are remarkably robust to hyperparameter choices. Increasing B never hurts (it only adds more trees to the average), so the main concern is computational cost. The most important parameter to tune is m, which controls the bias-variance balance. Cross-validation or OOB error can guide this choice.

A practical advantage of random forests is that they perform well "out of the box" with default settings, requiring far less tuning than gradient boosting or neural networks.

Variable Importance Measures

Since the ensemble of hundreds of trees is not directly interpretable, random forests provide variable importance measures to identify which predictors are most influential.

Impurity-based importance measures the total decrease in node impurity (Gini index for classification, RSS for regression) attributed to splits on each variable, averaged across all trees. Variables that frequently appear in splits and produce large impurity reductions are ranked as important. However, this measure is biased toward variables with many possible split points (continuous variables or categorical variables with many levels).

Permutation importance provides a more reliable measure. For each variable X_j: (1) compute the OOB error of the forest, (2) randomly permute the values of X_j in the OOB data (breaking its relationship with Y), and (3) recompute the OOB error. The increase in error is the permutation importance of X_j. A large increase means the model relies heavily on X_j. This measure is unbiased with respect to the number of split points.

In actuarial applications, variable importance helps identify which rating factors most strongly predict claim frequency or severity, guiding feature selection and pricing decisions.

Actuarial Applications and Comparison with GLMs

Random forests are widely used in insurance for claim frequency and severity prediction, fraud detection, customer segmentation, and lapse modeling. They often outperform GLMs in prediction accuracy because they can capture nonlinear effects and complex interactions without explicit specification.

However, GLMs remain the standard in many actuarial applications for several reasons: GLMs produce smooth, monotonic relativities that are easier to explain to regulators; GLMs can be updated easily with new data; and GLM predictions can be decomposed into individual factor contributions. Random forests complement GLMs by identifying important interactions and nonlinear effects that can then be incorporated into a GLM.

Study Tips for Exam SRM

For Exam SRM, focus on these concepts. First, understand the bagging procedure (bootstrap sampling, fitting trees, averaging) and why it reduces variance. Second, know how random forests improve on bagging through the random predictor subset at each split. Third, understand the OOB error estimate and why it approximates test error. Fourth, be able to distinguish between impurity-based and permutation importance. Fifth, understand the tradeoff: random forests gain accuracy at the cost of interpretability compared to single trees. Common exam questions ask about the effect of changing m or B on model performance, the interpretation of variable importance plots, or the comparison between random forests and other methods.