Predictive Modeling in Insurance: Beyond GLMs
How machine learning techniques complement and extend traditional GLM-based insurance pricing.
Limitations of GLMs
While generalized linear models remain the industry standard for insurance pricing, they have well-known limitations. GLMs assume a specific link function and distribution family, require the actuary to pre-specify interactions and non-linear effects, and may miss complex patterns in high-dimensional data. The multiplicative structure of a GLM with a log link assumes that the effect of each rating variable is independent of the values of other variables, which may not hold in practice. For example, the effect of vehicle age on claim frequency may depend on the driver's age, but capturing this interaction requires the actuary to explicitly specify and include it in the model.
These limitations have motivated the adoption of machine learning methods that can discover patterns in data without requiring the actuary to pre-specify the model structure. Tree-based methods (decision trees, random forests, gradient boosted machines) and neural networks can capture non-linear relationships and high-order interactions automatically, potentially improving predictive accuracy. These methods have proven particularly effective for identifying risk segments that GLMs miss, especially in data-rich environments with many potential predictors.
Decision Trees and Random Forests
Decision trees recursively partition the data into subgroups based on the predictor variable and split point that best separates the outcomes. The result is a tree-shaped model where each leaf node contains a prediction (the average response for observations falling into that leaf). Trees naturally capture non-linear effects (different effects at different levels of a variable) and interactions (the effect of one variable depends on the value of another) without requiring explicit specification.
Single decision trees tend to overfit the training data and produce unstable predictions (small changes in the data can produce very different tree structures). Random forests address this by growing many trees, each trained on a bootstrap sample of the data with a random subset of predictor variables considered at each split. The final prediction averages across all trees, reducing variance and producing more stable, accurate predictions. Random forests are relatively easy to tune (the main parameters are the number of trees and the number of variables considered at each split) and provide built-in measures of variable importance.
Gradient Boosted Machines
Gradient boosted machines (GBMs) are the most popular alternative to GLMs in insurance pricing due to their consistently strong predictive performance. GBMs build an ensemble of small decision trees sequentially, with each new tree fitted to the residuals (errors) of the previous ensemble. The learning rate controls how aggressively each new tree corrects the errors, and regularization parameters (tree depth, minimum observations per leaf, subsampling rate) prevent overfitting.
Popular GBM implementations include XGBoost, LightGBM, and CatBoost. Each offers optimizations for speed and memory efficiency. LightGBM uses histogram-based splitting and leaf-wise tree growth for faster training on large datasets. CatBoost handles categorical variables natively using ordered target encoding, which is particularly useful for insurance data with many categorical rating variables (territory, vehicle make, occupation).
GBMs provide reasonable interpretability through several techniques. Partial dependence plots show the marginal effect of a single variable on the predicted outcome, averaging over the values of all other variables. SHAP (SHapley Additive exPlanations) values decompose each individual prediction into the contribution of each feature, providing both global and local interpretability. These tools help actuaries understand what the model has learned and communicate its behavior to regulators and business stakeholders.
Neural Networks and Deep Learning
Neural networks consist of layers of interconnected nodes that transform input features through non-linear activation functions. The flexibility of neural networks allows them to approximate arbitrarily complex functions, making them powerful predictive tools. In insurance applications, neural networks have shown promise for image-based claims assessment (estimating vehicle damage from photographs), natural language processing of claim descriptions, and pricing in complex multi-peril environments.
However, neural networks present significant challenges for insurance applications. They require large amounts of training data to perform well and are prone to overfitting on the relatively small datasets common in specialty lines. Their "black box" nature makes them difficult to interpret and explain to regulators, which limits their use in rate filings. Training neural networks requires expertise in architecture design, hyperparameter tuning, and regularization that goes beyond traditional actuarial training. For these reasons, neural networks are less commonly used in insurance pricing than tree-based methods, though they find applications in claims processing, fraud detection, and other areas where regulatory interpretability requirements are less stringent.
Practical Integration with GLMs
Most insurers do not replace GLMs entirely with machine learning models. Instead, they use a complementary approach where machine learning models identify important features, interactions, and non-linear effects, which are then incorporated into the GLM framework for regulatory filings. This "model-assisted" approach combines the predictive power of machine learning with the interpretability and regulatory acceptance of GLMs.
A common workflow involves fitting a GBM to the full dataset with all available predictors, using variable importance and partial dependence plots to identify the most important predictors and their functional forms, engineering new features based on these discoveries (such as interaction terms, binned variables, or non-linear transformations), and incorporating these features into a GLM. The resulting GLM often performs significantly better than a naive GLM without the machine-learning-informed feature engineering, while retaining the transparency and regulatory defensibility of the GLM framework.
Model Validation and Fairness
Regardless of the modeling technique used, rigorous validation is essential. This includes out-of-sample testing (using data not involved in model fitting to assess predictive accuracy), out-of-time testing (validating on data from a later time period to assess temporal stability), and lift analysis (comparing predicted versus actual outcomes across deciles or other groupings). Comparing the predictive performance of machine learning models against the current GLM benchmark quantifies the practical benefit of adopting the more complex approach.
Fairness testing has become increasingly important as models incorporate more variables and capture more complex patterns. Actuaries must evaluate whether model predictions produce unfairly discriminatory outcomes for protected classes, even when the model does not use protected class variables directly. Proxy discrimination can occur when variables that are correlated with race, ethnicity, or other protected characteristics serve as indirect proxies. Techniques for assessing and mitigating unfair discrimination include disparate impact analysis, counterfactual fairness testing, and constrained optimization that imposes fairness constraints during model fitting.
Study Relevance
Predictive modeling beyond GLMs is tested on SOA Exam SRM and the Predictive Analytics certificate, as well as CAS exams covering ratemaking and data science. Candidates should understand the conceptual basis of tree-based methods and ensemble techniques, the advantages and limitations of each approach relative to GLMs, and the practical considerations involved in model validation and regulatory compliance. Familiarity with at least one implementation (R or Python) and the ability to interpret partial dependence plots and variable importance measures are valuable skills for both exams and practice.