Model Assessment: AIC, BIC, and Adjusted R-Squared

The Model Selection Problem

When building a predictive model, the analyst must choose among competing models that differ in complexity (number of parameters, choice of variables, or model family). Using training error to make this choice is unreliable because training error always decreases with increasing complexity, eventually reaching zero for sufficiently flexible models. We need criteria that account for overfitting by penalizing complexity.

Three widely used criteria for model selection are AIC, BIC, and adjusted R-squared. Each balances model fit against complexity in a different way, and understanding their properties helps you choose the right tool for a given situation. All three can be computed from the fitted model without additional data splitting (unlike cross-validation), making them computationally convenient.

AIC (Akaike Information Criterion)

AIC = -2 * ln(L_hat) + 2k, where L_hat is the maximized likelihood of the model and k is the number of estimated parameters (including the intercept and any variance parameters). The first term rewards goodness of fit (smaller deviance means better fit), while the second term penalizes complexity. Lower AIC values indicate a better balance between fit and parsimony.

AIC has a deep theoretical connection to prediction: it is asymptotically equivalent to leave-one-out cross-validation for linear models. This means that AIC estimates the expected out-of-sample prediction error, making it a prediction-oriented criterion. It tends to select models that predict well on new data.

For linear regression with normally distributed errors, AIC can be expressed as: AIC = n * ln(SSE/n) + 2k + constant. The constant does not depend on the model and cancels when comparing models, so it is often omitted.

A worked example: Model A has 3 parameters and a log-likelihood of -150. AIC_A = -2(-150) + 2(3) = 306. Model B has 5 parameters and a log-likelihood of -145. AIC_B = -2(-145) + 2(5) = 300. Model B has a lower AIC and is preferred, even though it has more parameters, because the improvement in fit (log-likelihood increase from -150 to -145) outweighs the complexity penalty (from 6 to 10).

An important property of AIC is that it is not consistent. As the sample size n grows, AIC tends to select models that are slightly too complex (it overfits asymptotically). In practice, this means AIC may include one or two unnecessary predictors but generally produces good predictions.

BIC (Bayesian Information Criterion)

BIC = -2 * ln(L_hat) + k * ln(n), where n is the sample size. The only difference from AIC is the penalty: BIC uses k * ln(n) instead of 2k. Since ln(n) > 2 whenever n >= 8 (which is virtually always), BIC penalizes complexity more heavily than AIC.

This stronger penalty means BIC tends to select simpler models than AIC. BIC has a key theoretical advantage: it is consistent, meaning that as n approaches infinity, BIC selects the true model (the data-generating model) with probability approaching 1, assuming the true model is among the candidates. AIC lacks this property.

Continuing the worked example: BIC_A = -2(-150) + 3 * ln(100) = 300 + 13.82 = 313.82. BIC_B = -2(-145) + 5 * ln(100) = 290 + 23.03 = 313.03. BIC also selects Model B, but the margin is much smaller (0.79 vs. 6 for AIC). With a slightly different dataset, BIC might prefer the simpler Model A, illustrating its stronger preference for parsimony.

The choice between AIC and BIC depends on your goal. If the goal is prediction (minimizing forecast error), AIC is generally preferred because it is calibrated to prediction performance. If the goal is model identification (finding the "true" model or the most interpretable one), BIC is preferred because of its consistency and tendency toward simpler models.

Corrected AIC (AICc)

For small samples, the standard AIC can over-select complex models because the penalty 2k underestimates the bias. The corrected AIC, AICc = AIC + 2k(k+1) / (n - k - 1), adds a correction that increases the effective penalty when n is small relative to k. As n grows large, the correction becomes negligible and AICc converges to AIC.

A common rule of thumb: use AICc instead of AIC when n/k < 40. In actuarial modeling, where datasets can sometimes be modest, AICc is a safer default choice.

Adjusted R-Squared

The coefficient of determination R^2 = 1 - SSE/SST measures the proportion of variance explained by the model. However, R^2 always increases (or stays the same) when a predictor is added, even if the predictor is uninformative. This makes R^2 unreliable for model selection.

Adjusted R^2 = 1 - (SSE/(n - p - 1)) / (SST/(n - 1)) corrects for the number of predictors p by using the mean squared error (SSE/(n-p-1)) instead of the total sum of squares. The denominator SSE/(n-p-1) is the unbiased estimate of the error variance. Unlike R^2, adjusted R^2 can decrease when an uninformative predictor is added, because the loss of one degree of freedom may outweigh any trivial reduction in SSE.

Adjusted R^2 is easy to interpret (it is on the same scale as R^2) and simple to compute. However, it has important limitations. It is only applicable to linear regression models with normally distributed errors and the same response variable. It cannot be used to compare models with different transformations of Y (e.g., ln(Y) vs. Y), models from different families (e.g., Poisson vs. normal), or non-nested models in general.

Comparing the Three Criteria

Understanding when to use each criterion is essential for Exam SRM. AIC is the most versatile: it applies to any model estimated by maximum likelihood (linear regression, GLMs, logistic regression, time series models), it focuses on prediction, and it approximates LOOCV. BIC shares AIC's broad applicability but focuses on model identification and selects simpler models. Adjusted R^2 is limited to linear regression but is intuitive and quick to compute.

All three criteria are relative measures, meaning they are used to compare candidate models, not to assess a model's absolute quality. A model with the best AIC might still fit poorly in absolute terms. Residual diagnostics and goodness-of-fit tests should complement any criterion-based model selection.

When AIC and BIC disagree (which happens when AIC selects a more complex model), consider the modeling goal. For insurance pricing (where prediction accuracy matters most), follow AIC. For parsimonious rate filing (where simplicity and interpretability are valued), follow BIC.

Study Tips for Exam SRM

For Exam SRM, know the formulas for AIC, BIC, and adjusted R^2, and be able to compute them from given summary statistics. Understand the key difference: AIC favors prediction, BIC favors identification of the true model. Remember that lower values of AIC and BIC are better, while higher values of adjusted R^2 are better. Know that AIC approximately equals LOOCV, and that BIC is consistent. Practice comparing models using these criteria and interpreting the results. A common exam format presents output from several fitted models and asks which model is preferred under AIC vs. BIC.