Variable Selection Methods: Forward, Backward, Stepwise for Exam SRM

Why Variable Selection Matters

In multiple regression, including too many predictors leads to overfitting (the model captures noise rather than signal), while including too few leads to underfitting (the model misses important relationships). Variable selection methods seek to identify the subset of predictors that provides the best balance of model fit and parsimony.

In actuarial applications, a pricing model might have dozens of candidate rating factors (age, gender, territory, vehicle type, credit score, driving history, and more). Not all of these will be statistically significant or improve predictions. Variable selection helps identify which factors genuinely contribute to explaining claim frequency or severity.

The ideal approach would be to evaluate all possible subsets of predictors (best subset selection), but this is computationally prohibitive when the number of candidate predictors is large. With p predictors, there are 2^p possible models. Automated sequential methods provide a practical alternative.

Forward Selection

Forward selection starts with no predictors (just the intercept) and adds them one at a time. At each step, it fits a model with each candidate predictor added individually and selects the one with the smallest p-value (equivalently, the largest partial F-statistic), provided that p-value is below an entry threshold (e.g., alpha_enter = 0.05 or 0.15). The process stops when no remaining predictor meets the entry criterion.

Example: With 5 candidate predictors X1 through X5, step 1 fits 5 simple regressions and finds X3 has the smallest p-value (p = 0.001). X3 enters. Step 2 fits 4 models, each adding one of the remaining predictors to the model already containing X3. X1 has the smallest p-value (p = 0.02). X1 enters. Step 3 finds that no remaining predictor has p < 0.05. The process stops with the model Y ~ X3 + X1.

Advantages: computationally efficient (fits at most p*(p+1)/2 models), works well when the true model is sparse. Disadvantage: once a variable enters, it stays, even if it becomes insignificant after other variables are added. This greedy approach can miss the best combination of predictors.

Backward Elimination

Backward elimination starts with all p predictors in the model and removes them one at a time. At each step, it identifies the predictor with the largest p-value (least significant) and removes it if the p-value exceeds an exit threshold (e.g., alpha_exit = 0.05 or 0.10). The process stops when all remaining predictors are significant (all p-values are below the threshold).

Advantages: considers all predictors simultaneously from the start, which can detect suppressor effects (a variable that is not significant alone but becomes significant in the presence of other variables). Backward elimination often finds better models than forward selection because it begins with the full picture of predictor relationships.

Disadvantage: requires n > p+1 (more observations than parameters), which is not always satisfied in high-dimensional actuarial datasets. It is also computationally intensive when p is large, though this is rarely a practical concern with modern computing.

Stepwise Selection

Stepwise selection combines forward and backward: after adding a variable (using the forward entry criterion), it checks whether any previously entered variable should now be removed (using the backward exit criterion). This addresses the main weakness of pure forward selection, where a variable that was significant early on might become redundant once other variables enter.

Typically, alpha_enter < alpha_exit (e.g., enter at 0.05, exit at 0.10) to prevent a variable from entering and immediately being removed, which would create an infinite loop. Stepwise selection usually finds a model similar to the backward elimination result, but at lower computational cost because it does not require fitting the full model first.

Information Criteria: AIC and BIC

An alternative to p-value-based selection is to use information criteria that explicitly balance fit and complexity. AIC = -2*ln(L) + 2*k, where L is the maximized likelihood and k is the number of parameters. BIC = -2*ln(L) + ln(n)*k, where n is the sample size. Lower values indicate better models.

BIC penalizes model complexity more heavily than AIC (when n >= 8), so BIC tends to select smaller models. AIC is more appropriate when the goal is prediction (minimizing expected prediction error), while BIC is better for identifying the true model (consistent model selection). In actuarial practice, AIC is more commonly used for pricing models, while BIC may be preferred for explanatory models.

These criteria can be used within any selection framework: forward, backward, stepwise, or best subset. At each step, instead of comparing p-values to a threshold, compare AIC or BIC values of competing models.

Limitations of Automated Selection

All automated variable selection methods share important limitations that Exam SRM frequently tests. They do not guarantee finding the globally best subset of predictors. The selected model depends on arbitrary thresholds (p-value cutoffs for sequential methods, or the choice of AIC vs. BIC). Multiple testing inflates the Type I error rate: when many predictors are tested, some will appear significant by chance.

Standard errors, p-values, and confidence intervals reported after variable selection are biased because the selection process is not accounted for. The reported standard errors are too small and p-values are too low, making the selected predictors appear more significant than they truly are. This post-selection inference problem is a major concern in modern statistics.

Domain knowledge should always guide variable selection. If actuarial theory strongly supports a predictor, it should remain in the model even if its p-value is slightly above the threshold. Conversely, a variable that is statistically significant but has no theoretical justification may be capturing noise.

Study Tips and Common Pitfalls

On Exam SRM, understand the mechanics of each method (what enters or exits at each step) and be able to trace through the process given a table of p-values. Know the advantages and disadvantages of each approach and when each is appropriate.

A common exam question presents the results of a selection procedure and asks you to determine which model would be chosen. Another common format asks you to identify limitations or compare methods. Practice at least 4 problems tracing through selection procedures and 3 problems comparing AIC, BIC, and adjusted R^2 for model comparison.