Ensemble Methods: Stacking and Blending for Actuaries
Advanced ensemble techniques including stacking, blending, and model averaging for actuarial applications on MAS-II.
Beyond Simple Ensembles
While bagging and boosting are well-known ensemble methods, stacking and blending take model combination further. Stacking trains a meta-learner on the predictions of base models, allowing it to learn optimal ways to combine them. Blending uses a holdout set instead of cross-validation for the meta-learner training. Both methods can capture strengths of diverse model types.
To understand why stacking works, recall the bias-variance decomposition of prediction error. Different model types have different bias-variance profiles. A GLM has relatively high bias but low variance. A complex gradient boosting model has low bias but higher variance. By combining models with complementary strengths, the ensemble can achieve lower total error than any individual model. Stacking learns the optimal combination weights from data rather than using equal weighting or subjective choices.
How Stacking Works
The stacking procedure has two levels. At Level 0, you train multiple base models on the training data. At Level 1, you train a meta-learner that takes the predictions of the base models as inputs and produces a final prediction. The key challenge is avoiding data leakage: if the base models are trained on the same data used to generate their predictions for the meta-learner, those predictions will be overfit and the meta-learner will learn to trust overly optimistic base model outputs.
Cross-validated stacking solves this problem. Divide the training data into k folds. For each fold, train each base model on the other k-1 folds and generate predictions for the held-out fold. After cycling through all folds, you have out-of-fold predictions from each base model for every observation in the training set. These out-of-fold predictions become the input features for the meta-learner. This process ensures that the meta-learner trains on predictions that the base models made without seeing the corresponding true outcomes.
Here is the step-by-step algorithm. Step 1: Split training data into k folds. Step 2: For each base model m and each fold j, train model m on all folds except j and predict on fold j. Step 3: Collect all out-of-fold predictions to create a new training matrix where each column is one base model's predictions. Step 4: Train the meta-learner on this matrix with the original target variable. Step 5: To make predictions on new data, run each base model (now trained on the full training set) on the new data, then feed those predictions through the meta-learner.
Blending as an Alternative
Blending simplifies the stacking procedure by using a single holdout set instead of cross-validation. You split the training data into a training portion (typically 70 to 80 percent) and a blending portion (the remainder). Train all base models on the training portion. Generate predictions on the blending portion. Train the meta-learner on those predictions. The advantage is simplicity and speed. The disadvantage is that you use less data for both base model training and meta-learner training, which can reduce performance, particularly with smaller datasets.
Blending is preferred over stacking when computational resources are limited (since stacking requires training each base model k times), when the dataset is large enough that the holdout set provides sufficient data for reliable meta-learner training, or when you need a simpler procedure for production deployment and maintenance.
Actuarial Applications
In insurance pricing, a stack might combine a GLM (for interpretability and regulatory compliance), a gradient boosting machine (for predictive power), and a neural network (for capturing complex interactions). The meta-learner, often a simple linear model, assigns weights based on each model's contribution. For MAS-II, understand how to set up cross-validated stacking to avoid data leakage, when blending is preferred over stacking, and the bias-variance trade-off in ensemble construction. Be prepared to discuss practical concerns like model governance and explaining ensemble predictions to regulators.
Choosing Base Models
The effectiveness of stacking depends on diversity among the base models. Combining five gradient boosting models with slightly different hyperparameters provides less benefit than combining a GLM, a random forest, a gradient boosting model, and a support vector machine. Diversity means that the models make errors on different observations. When one model gets an observation wrong, the others should tend to get it right, allowing the meta-learner to aggregate them into a more accurate prediction.
In actuarial practice, a typical stack might include a Poisson GLM with traditional rating variables as a baseline, a gradient boosting machine trained on the same features for better predictive performance, a regularized GLM (LASSO or Elastic Net) for automatic feature selection, and possibly a model using alternative data sources like telematics or credit information. Each model brings a different perspective on the risk.
The Meta-Learner
The meta-learner is usually kept simple to avoid overfitting at the second level. A linear regression or logistic regression is the most common choice. A constrained linear model that requires non-negative weights summing to one is equivalent to model averaging and is particularly easy to interpret. More complex meta-learners like another gradient boosting model can be used but require careful regularization to prevent overfitting.
For insurance applications, a non-negative linear combination is often preferred because it maintains the interpretability needed for regulatory filings. The meta-learner weights directly tell you how much each base model contributes to the final prediction. If the GLM receives a weight of 0.3 and the GBM receives 0.7, you know the ensemble relies primarily on the GBM's predictive power while still incorporating the GLM's stable structure.
Model Averaging
Simple model averaging, where you take the unweighted mean of base model predictions, is the simplest form of ensembling. Despite its simplicity, averaging often performs surprisingly well. It works because averaging reduces variance without increasing bias (assuming the base models are approximately unbiased). For exam purposes, understand that model averaging is a special case of stacking where the meta-learner assigns equal weights to all base models.
Common Pitfalls
The most critical pitfall in stacking is data leakage. If you train base models on data that they also use to generate meta-learner inputs, the stacked model will appear to perform much better during development than it does in production. Always use cross-validated or holdout-based predictions for the meta-learner. Another pitfall is using too many base models, which increases complexity without proportional gains and can make the system difficult to maintain. Start with three to five diverse base models and add more only if they demonstrably improve performance.
Students also sometimes confuse stacking with boosting. In boosting, models are trained sequentially, with each model focusing on the errors of the previous one. In stacking, base models are trained independently, and the meta-learner combines them after the fact. These are fundamentally different strategies, even though both produce ensemble predictions.
Study Tips
For MAS-II, practice describing the cross-validated stacking algorithm step by step. Be able to explain why out-of-fold predictions are necessary for the meta-learner. Know the difference between stacking and blending and when each is appropriate. Understand how to choose diverse base models and why diversity matters. Be prepared to discuss governance challenges, including how to explain ensemble predictions to regulators who expect transparent models, and consider whether partial dependence plots or SHAP values can provide the needed interpretability.