Multiple Linear Regression: Model Building for Actuaries
Learn multiple linear regression, interpretation, and model building for Exam SRM.
The Multiple Regression Model
Multiple linear regression extends the simple model to p predictors: Y = beta_0 + beta_1*X1 + beta_2*X2 + ... + beta_p*Xp + epsilon. In matrix notation, Y = X*beta + epsilon, and the OLS solution is beta_hat = (X'X)^(-1)*X'Y. The model has p+1 parameters (p slopes and one intercept) and requires n > p+1 observations to estimate them.
The interpretation of each coefficient changes from simple regression: beta_j represents the expected change in Y for a one-unit increase in Xj, holding all other predictors constant. This "holding constant" interpretation is crucial and distinguishes multiple regression from running separate simple regressions. In actuarial modeling, this means the effect of age on claim cost is estimated while controlling for gender, territory, vehicle type, and other rating factors.
Coefficients can change sign or magnitude when additional predictors are added. A variable that appears positively associated with Y in simple regression might have a negative coefficient in multiple regression after controlling for confounders. This phenomenon, related to Simpson's paradox, highlights why multivariate analysis is essential.
R-Squared and Adjusted R-Squared
R^2 = 1 - SSE/SST = SSR/SST measures the proportion of variability in Y explained by all predictors together. Unlike simple regression, R^2 in multiple regression is not the square of a single correlation. R^2 always increases (or stays the same) when predictors are added, even if they are irrelevant. Adding a random noise variable to the model will almost surely increase R^2 slightly, making R^2 unreliable for comparing models with different numbers of predictors.
Adjusted R^2 = 1 - (SSE/(n-p-1))/(SST/(n-1)) = 1 - (n-1)/(n-p-1) * (1 - R^2) penalizes for the number of predictors and can decrease when an unhelpful predictor is added. Use adjusted R^2 for comparing models with different numbers of predictors. A model with higher adjusted R^2 provides a better balance of fit and parsimony.
Example: A model with 3 predictors and n = 50 gives R^2 = 0.65 and adjusted R^2 = 0.627. Adding a fourth predictor increases R^2 to 0.652 but adjusted R^2 drops to 0.621. The slight improvement in R^2 does not justify the added complexity, as indicated by the decrease in adjusted R^2.
The Overall F-Test
The overall F-test tests H0: beta_1 = beta_2 = ... = beta_p = 0, meaning no predictor is useful. The test statistic is F = (SSR/p) / (SSE/(n-p-1)) = MSR/MSE, which follows an F-distribution with (p, n-p-1) degrees of freedom under H0. A large F-statistic (small p-value) indicates that at least one predictor is significantly related to Y.
Rejecting H0 in the overall F-test does not tell you which predictors are significant; individual t-tests are needed for that. It is possible for the overall F-test to be significant while no individual t-test is significant, particularly when predictors are highly correlated (multicollinearity).
Partial F-Tests for Nested Models
A partial F-test compares two nested models: a reduced model with p1 predictors and a full model with p2 predictors (p2 > p1). The test statistic is F = ((SSE_reduced - SSE_full) / q) / MSE_full, where q = p2 - p1 is the number of additional parameters in the full model. This follows an F-distribution with (q, n-p2-1) degrees of freedom under H0.
The partial F-test determines whether the additional q predictors provide a statistically significant improvement in fit beyond what the reduced model achieves. This is more informative than comparing individual t-tests because it assesses the joint contribution of the added predictors.
Example: A reduced model with 3 predictors gives SSE = 450, and a full model with 5 predictors gives SSE = 380 with n = 60. F = ((450 - 380)/2) / (380/54) = 35/7.037 = 4.97 on (2, 54) degrees of freedom. Comparing to the critical value determines whether the two additional predictors significantly improve the model.
Interpreting Regression Output
Exam SRM frequently presents regression output tables and asks for interpretation. A typical output includes: coefficients (beta_hat values), standard errors, t-statistics, p-values, R^2, adjusted R^2, the F-statistic, and sometimes confidence intervals. Key skills include identifying which predictors are significant at a given alpha level, interpreting coefficient magnitudes, and assessing overall model quality.
When interpreting coefficients with interaction terms (e.g., X1*X2), the effect of X1 on Y depends on the value of X2. The coefficient of the interaction term measures how the slope of X1 changes per unit change in X2. Similarly, polynomial terms (X^2) indicate non-linear relationships: the effect of X on Y changes with the level of X.
Categorical predictors enter the model through dummy variables. A categorical variable with k levels requires k-1 dummy variables, with the omitted level serving as the reference category. Each dummy variable's coefficient represents the difference in expected Y between that category and the reference category, holding other predictors constant.
Model Building Strategies
Building a good multiple regression model involves selecting the right predictors, checking assumptions, and avoiding overfitting. Start with substantive knowledge: include predictors that theory or domain expertise suggests are relevant. In actuarial applications, this means using established rating factors and known risk drivers.
After fitting an initial model, examine residual plots for violations of assumptions (nonlinearity, heteroscedasticity, non-normality). Consider transformations (log, square root) if the assumptions are violated. Check for influential observations and outliers. Use adjusted R^2, AIC, BIC, or cross-validation to compare competing models.
Avoid the temptation to include too many predictors, especially with limited data. As a rule of thumb, you need at least 10 to 15 observations per predictor to obtain reliable estimates. Overfitting produces a model that fits the training data well but generalizes poorly to new data.
Study Tips and Common Pitfalls
The most common mistake is interpreting coefficients without the "holding constant" qualifier. Saying "beta_1 = 3 means Y increases by 3 for each unit increase in X1" is incomplete; the correct statement adds "holding X2, X3, ... constant." Another frequent error is comparing R^2 values across models with different numbers of predictors instead of using adjusted R^2.
On Exam SRM, practice reading and interpreting regression output tables quickly. Be able to compute predicted values Y_hat = beta_0_hat + beta_1_hat*X1 + ... from given coefficients. Know how to set up and interpret partial F-tests. Work through at least 6 problems involving model comparison, coefficient interpretation, and output analysis to build confidence.