Machine Learning for Insurance Pricing: Gradient Boosting
Apply gradient boosting methods to insurance pricing and classification for Exam MAS-II.
Gradient Boosting Fundamentals
Gradient boosting builds an ensemble of weak learners (typically shallow decision trees) sequentially. Unlike bagging methods such as random forests that build trees independently, gradient boosting is inherently sequential: each new tree is specifically designed to correct the errors of the current ensemble. Starting from an initial prediction F_0(x) = argmin sum of L(y_i, gamma), each iteration m adds a tree h_m(x) multiplied by a learning rate nu: F_m(x) = F_{m-1}(x) + nu * h_m(x).
The "gradient" in gradient boosting refers to the fact that each new tree fits the negative gradient of the loss function with respect to the current predictions. For squared error loss, the negative gradient is simply the residual y_i minus F_{m-1}(x_i). For other loss functions, such as Poisson deviance (used in insurance frequency modeling), the pseudo-residuals take a different form that involves the link function. This generality allows gradient boosting to optimize virtually any differentiable loss function, making it adaptable to a wide range of actuarial modeling tasks.
How the Algorithm Works Step by Step
The gradient boosting algorithm proceeds as follows. First, initialize the model with a constant prediction, typically the mean of the response (for squared error) or the log of the mean (for Poisson). Second, for each iteration m from 1 to M: compute the pseudo-residuals r_im = negative partial derivative of L(y_i, F(x_i)) with respect to F(x_i), evaluated at F = F_{m-1}. Third, fit a regression tree with J terminal nodes to the pseudo-residuals. Fourth, for each terminal node j, compute the optimal constant prediction gamma_jm = argmin sum over i in node j of L(y_i, F_{m-1}(x_i) + gamma). Fifth, update the model: F_m(x) = F_{m-1}(x) + nu * sum over j of gamma_jm * I(x in node j).
The learning rate nu (typically 0.01 to 0.1) shrinks each tree's contribution, requiring more iterations but generally improving generalization. This shrinkage acts as a form of regularization that prevents the model from fitting noise in the training data. The optimal number of iterations M is determined by cross-validation or monitoring performance on a holdout set.
Key Hyperparameters
Tree depth (or maximum number of leaves) controls the complexity of individual trees and the order of interactions captured. A tree of depth 1 (a stump) captures only main effects. A tree of depth 2 captures two-way interactions. Deeper trees capture higher-order interactions but are more prone to overfitting. For insurance pricing, depths of 3 to 6 are common, as the most important rating factors typically interact in ways that can be captured by moderate-depth trees.
The subsampling fraction (also called the bag fraction) specifies what proportion of the training data is used to fit each tree. Values between 0.5 and 0.8 are typical. Subsampling introduces stochastic variation that acts as additional regularization and can improve generalization, similar to how dropout works in neural networks. The minimum number of observations per leaf prevents the model from creating terminal nodes based on very few data points, which would produce noisy predictions.
Interpretability Tools
Variable importance measures rank predictors by their contribution to the model. The most common measure sums the improvement in the loss function across all splits involving each variable across all trees. This provides a global ranking of which features matter most. However, importance scores do not indicate the direction or shape of each variable's effect.
Partial dependence plots (PDPs) reveal the marginal effect of one or two variables on the prediction, averaging over the values of all other variables. For a single variable x_j, the partial dependence function is f_j(x_j) = (1/n) * sum over i of F(x_j, x_{i,-j}), where x_{i,-j} denotes the values of all other variables for observation i. PDPs can reveal nonlinear effects (such as a U-shaped relationship between driver age and claim frequency) and interaction effects (visualized as 2D heatmaps for pairs of variables). Individual Conditional Expectation (ICE) plots extend PDPs by showing the effect for each observation separately, revealing heterogeneity that the average PDP may hide.
Comparison with GLMs
Compared to GLMs, gradient boosting machines offer several advantages. They automatically capture nonlinear effects and interactions without requiring the modeler to specify them in advance. They handle missing values and mixed data types (continuous and categorical) naturally. They can model complex relationships that GLMs miss, potentially improving predictive accuracy.
However, GLMs have advantages that matter in actuarial practice. GLMs produce interpretable coefficients with clear multiplicative effects on the response. They connect naturally to actuarial theory (the Poisson-Gamma frequency-severity framework is a GLM). Regulatory environments often require understandable models, which favors GLMs. In practice, many actuarial teams use GBMs as an exploratory tool to discover important interactions and nonlinear effects, then incorporate those findings into a GLM for production use.
Validation and Actuarial Metrics
Model validation for insurance pricing uses techniques tailored to the actuarial context. Cross-validation or a holdout test set estimates out-of-sample performance. The Gini coefficient (related to the area under the Lorenz curve) measures how well the model ranks risks from low to high. A higher Gini indicates better risk discrimination. Lift charts compare predicted and actual loss ratios across deciles of predicted risk, showing whether the model correctly identifies the most and least risky segments.
Double lift charts compare two models (for example, GBM versus GLM) by sorting observations by the ratio of their predictions and examining actual loss ratios in each group. If the GBM identifies pockets of risk that the GLM misses, the double lift chart will show monotonically increasing actual loss ratios across the groups. Calibration checks verify that predicted and actual losses agree in aggregate across different segments, which is essential for ensuring the model produces adequate premiums.
Common Pitfalls
A common mistake is using too many iterations without adequate regularization, leading to overfitting. Always monitor validation set performance and stop training when validation loss stops improving (early stopping). Another pitfall is interpreting variable importance without examining partial dependence plots. A variable can be important (it improves predictions) without having the intuitive directional effect you expect.
For insurance applications, beware of data leakage: including variables that are consequences of the outcome (such as claim count in a severity model) or variables that proxy for protected characteristics. Also remember that GBM predictions may need adjustment for balance (ensuring that total predicted losses match total actual losses on the training data) before they can be used for pricing.