Bias-Variance Tradeoff in Actuarial Predictive Models

Decomposition of Prediction Error

Understanding the bias-variance tradeoff begins with a precise decomposition of prediction error. For a given input x, the expected prediction error (mean squared error) of an estimator f_hat can be decomposed as: E[(Y - f_hat(x))^2] = [Bias(f_hat(x))]^2 + Var(f_hat(x)) + Var(epsilon).

Each term has a clear interpretation. The bias squared, [E[f_hat(x)] - f(x)]^2, measures how far the average prediction (averaged over all possible training sets) is from the true function value. High bias means the model systematically misses the target, regardless of the training data. The variance, Var(f_hat(x)) = E[(f_hat(x) - E[f_hat(x)])^2], measures how much the predictions fluctuate across different training sets. High variance means the model is sensitive to the specific training data used. The irreducible error, Var(epsilon), is the noise in the data that no model can eliminate. It sets a lower bound on the achievable prediction error.

The total expected error equals bias squared plus variance plus irreducible error. Since we cannot control the irreducible error, the goal of model selection is to minimize the sum of bias squared and variance.

How Complexity Affects Bias and Variance

Model complexity is the key factor connecting bias and variance. Simple models (few parameters, strong assumptions) have high bias and low variance. They impose a rigid structure on the data, so they miss complex patterns (high bias) but produce stable predictions across different training samples (low variance). Examples include linear regression with one or two predictors, and decision stumps (trees with one split).

Complex models (many parameters, flexible structure) have low bias and high variance. They can capture intricate patterns (low bias) but are sensitive to the specific training data (high variance). Examples include unpruned decision trees, high-degree polynomial regression, and k-nearest neighbors with small k.

Consider polynomial regression as a concrete example. A degree-1 polynomial (linear fit) may have substantial bias if the true relationship is curved, but its predictions are stable. A degree-15 polynomial can fit almost any pattern (low bias) but its predictions swing wildly with different training samples (high variance). Somewhere in between lies the optimal degree that minimizes total error.

The U-Shaped Test Error Curve

As model complexity increases from very simple to very complex, training error decreases monotonically (a more complex model always fits the training data better). Test error, however, follows a U-shape. Initially, as complexity increases, test error decreases because the reduction in bias outweighs the increase in variance. At some point, the test error reaches a minimum. Beyond this point, further complexity increases variance faster than it reduces bias, and test error begins to rise.

The gap between training error and test error is a useful diagnostic. A small gap with high values for both indicates underfitting (the model is too simple). A large gap with very low training error but high test error indicates overfitting (the model is too complex). The optimal model has a moderate gap and low test error.

For a worked example: suppose a dataset has an irreducible error of 5.0. A linear model has bias^2 = 8.0 and variance = 1.0, giving test error = 14.0. A moderate model has bias^2 = 2.0 and variance = 3.0, giving test error = 10.0. A very complex model has bias^2 = 0.5 and variance = 10.0, giving test error = 15.5. The moderate model achieves the best balance.

Overfitting and Underfitting

Overfitting occurs when the model fits the noise in the training data rather than the underlying signal. Signs include: training error much lower than test error, predictions that are highly sensitive to the training sample, and poor performance on new data despite excellent performance on training data. Common causes are too many predictors, too few observations, insufficient regularization, and growing trees too deep.

Underfitting occurs when the model fails to capture important patterns in the data. Signs include: both training and test error are high, and the model's predictions are far from the observed values even on the training set. Common causes are too few predictors, overly strong assumptions (e.g., assuming linearity when the relationship is nonlinear), and excessive regularization.

In actuarial pricing, overfitting might produce a model that appears highly accurate on historical data but produces unreliable premiums for new policies. Underfitting might produce a model that ignores important risk factors, leading to inadequate rate differentiation and adverse selection.

Methods for Managing the Tradeoff

Several techniques help find the optimal complexity. Cross-validation directly estimates the test error for each candidate model, allowing you to choose the complexity that minimizes prediction error. This is the most general and widely applicable approach.

Regularization (ridge, LASSO, elastic net) adds a penalty on model complexity to the training objective. This explicitly trades a small increase in bias for a larger decrease in variance. The regularization parameter lambda controls this tradeoff and is tuned by cross-validation.

Ensemble methods address the tradeoff in different ways. Bagging and random forests reduce variance without substantially increasing bias, by averaging many high-variance models. Boosting reduces bias by sequentially correcting prediction errors, with variance controlled through the learning rate and early stopping.

Early stopping in iterative algorithms (gradient boosting, neural network training) monitors the validation error during training and stops when it begins to increase. This prevents the model from continuing to fit noise after it has learned the signal.

Information criteria (AIC, BIC) provide analytical estimates of the bias-variance balance. AIC approximates leave-one-out CV error; BIC more heavily penalizes complexity. Both aim to select models that balance goodness of fit with parsimony.

Bias-Variance for Specific Methods

Understanding where each method falls on the bias-variance spectrum helps guide model selection. Linear regression: low variance, potentially high bias (if the true relationship is nonlinear). K-nearest neighbors with small k: low bias, high variance; with large k: higher bias, lower variance. Single decision tree: low bias when fully grown, high variance; pruning increases bias and decreases variance. Random forest: lower variance than a single tree, similar bias. Gradient boosting: can achieve very low bias, but variance increases with too many iterations.

For Exam SRM, be able to explain where a given method falls on the bias-variance spectrum and how its tuning parameters affect the tradeoff. For instance, increasing the regularization parameter in ridge regression increases bias and decreases variance. Increasing the number of trees in a random forest decreases variance without affecting bias.

Study Tips for Exam SRM

The bias-variance tradeoff is the single most important conceptual framework for Exam SRM's model selection topics. Every method for choosing among models (cross-validation, regularization, pruning, early stopping, information criteria) is an attempt to find the sweet spot that minimizes total prediction error. Practice articulating how each method manages the tradeoff. For exam questions: if asked to explain why a model performs well on training data but poorly on test data, the answer involves overfitting (high variance). If both errors are high, the answer involves underfitting (high bias). Be able to draw the U-shaped test error curve and label its components.