Model Comparison and Information Criteria for MAS-I

The Model Selection Problem

In actuarial modeling, you often consider multiple candidate models for the same data. For pricing auto insurance, you might compare a Poisson GLM with 5 rating factors versus one with 12 factors, or a Poisson model versus a negative binomial. More complex models fit the training data better but may overfit, capturing noise rather than genuine patterns. Model selection methods help identify the model that best balances fit and complexity, producing the best predictions for new data.

Two broad categories of model selection methods exist. Information criteria (AIC, BIC) provide quick, formula-based comparisons. Cross-validation methods estimate out-of-sample prediction error directly by repeatedly splitting the data into training and test sets. Both approaches are tested on Exam MAS-I.

Akaike Information Criterion (AIC)

The AIC is defined as AIC = -2 * ln(L) + 2k, where L is the maximized likelihood and k is the number of estimated parameters. The first term measures fit (smaller is better, corresponding to higher likelihood), and the second term penalizes complexity. The model with the smallest AIC is preferred. AIC can be used to compare any models fitted to the same data, whether nested or not.

For example, suppose Model A has 5 parameters and log-likelihood -1,200, while Model B has 8 parameters and log-likelihood -1,190. Then AIC_A = 2*1200 + 2*5 = 2,410 and AIC_B = 2*1190 + 2*8 = 2,396. Model B has the lower AIC, so it is preferred despite its greater complexity. The improvement in fit (20 units of log-likelihood) more than compensates for the additional 3 parameters (penalty of 6).

AIC has a deep theoretical justification: it is asymptotically equivalent to leave-one-out cross-validation. This means that minimizing AIC approximately minimizes the expected out-of-sample prediction error, which is the quantity you ultimately care about. However, for small samples, AIC tends to select overly complex models.

Corrected AIC and BIC

The corrected AIC, AICc = AIC + 2k(k+1)/(n-k-1), adjusts for the finite-sample bias of AIC. The correction term is negligible for large n but substantial when n is not much larger than k. As a rule of thumb, use AICc instead of AIC whenever n/k < 40. AICc always selects the same model or a simpler one compared to AIC.

The Bayesian Information Criterion (BIC) uses a heavier complexity penalty: BIC = -2 * ln(L) + k * ln(n). For n > 8 (since ln(8) > 2), BIC penalizes each additional parameter more heavily than AIC. BIC is consistent, meaning it selects the true model with probability approaching 1 as n grows (assuming the true model is among the candidates). However, BIC can select overly simple models in finite samples. In practice, AIC tends to favor more complex models than BIC.

When to Use AIC vs. BIC

The choice between AIC and BIC depends on your modeling goal. If the goal is prediction (finding the model that best predicts new data), AIC is generally preferred because it minimizes prediction error. If the goal is identifying the true data-generating process (selecting the "correct" model from the candidates), BIC is preferred because of its consistency property.

In actuarial pricing, the goal is usually prediction: you want the model that most accurately predicts future claims. This favors AIC. In scientific investigations where you want to understand which variables truly affect the outcome, BIC's stronger penalty for complexity is more appropriate. Many practitioners report both and check whether they agree.

Likelihood Ratio Tests

The likelihood ratio test (LRT) compares two nested models. The test statistic is Lambda = -2 * (ln(L_reduced) - ln(L_full)), where L_reduced is the likelihood of the simpler (reduced) model and L_full is the likelihood of the more complex (full) model. Under the null hypothesis that the reduced model is correct, Lambda follows approximately a chi-squared distribution with degrees of freedom equal to the difference in the number of parameters.

For example, to test whether three interaction terms should be included in a GLM, fit the model with and without the interactions. If the log-likelihoods are -1,200 (without) and -1,190 (with), the LRT statistic is -2 * (-1200 - (-1190)) = 20 with 3 degrees of freedom. Since chi-squared(3) at the 5% level is 7.815, the test strongly rejects the reduced model, supporting the inclusion of the interactions.

Cross-Validation

K-fold cross-validation splits the data into K roughly equal subsets (folds). For each fold, train the model on the remaining K-1 folds and compute the prediction error on the held-out fold. The cross-validation estimate of prediction error is the average across all K folds. Common choices are K = 5 or K = 10.

Leave-one-out (LOO) cross-validation is the special case K = n, where each observation is held out once. LOO has the advantage of using nearly all the data for training each time, reducing bias, but it has higher variance and is computationally expensive for large data sets. For GLMs, LOO can be approximated analytically using the hat matrix, avoiding the need to refit the model n times.

Vuong's Test for Non-Nested Models

The likelihood ratio test requires nested models (one is a special case of the other). For non-nested models (such as Poisson versus negative binomial, or lognormal versus gamma for severity), Vuong's test provides a formal comparison. The test statistic is based on the individual log-likelihood differences: compute d_i = ln(f_1(y_i)) - ln(f_2(y_i)) for each observation, then test whether the mean of d_i is significantly different from zero using a t-test. If d_bar is significantly positive, Model 1 is preferred; if negative, Model 2 is preferred.

Common Pitfalls

A frequent mistake is comparing AIC or BIC values across models fitted to different data. These criteria are only valid for comparing models fitted to the same observations. If one model drops missing values and another does not, the comparison is invalid. Another error is using the likelihood ratio test for non-nested models, which violates the chi-squared approximation.

Students also sometimes confuse smaller AIC (better) with larger AIC (worse). Remember: AIC measures the "information loss" of using the model instead of the true model, so smaller is better. The same applies to BIC.

Study Tips

Exam MAS-I tests the application of these methods to select among competing actuarial models. Memorize the formulas for AIC, AICc, and BIC, and know when to use each. Practice computing AIC values and selecting the best model from a set of candidates. Understand the likelihood ratio test setup: which model is null, which is alternative, and how to compute degrees of freedom. Know that AIC is better for prediction while BIC is better for model identification. Be prepared to explain cross-validation conceptually and to compare it with information criteria approaches.