Cross-Validation Techniques: k-Fold and Leave-One-Out
Master cross-validation for model selection and assessment on Exam SRM.
Why Cross-Validation?
The fundamental challenge in model selection is estimating how well a model will perform on new, unseen data. Training error (the error on the data used to fit the model) is a poor estimate of prediction performance because it decreases as model complexity increases. A sufficiently complex model can achieve zero training error by memorizing the data, yet perform terribly on new observations.
The ideal approach is to evaluate models on a separate test set that was not used during training. However, holding out data for testing reduces the amount available for training, which can hurt model quality, especially with small datasets. Cross-validation provides a clever compromise: it systematically rotates which portion of the data serves as the test set, using all observations for both training and evaluation.
Cross-validation serves three primary purposes in actuarial modeling: selecting among competing models (e.g., choosing between a GLM and a random forest), tuning hyperparameters (e.g., finding the optimal regularization strength for ridge regression), and providing an honest estimate of prediction accuracy for the final model.
k-Fold Cross-Validation
In k-fold CV, the data is randomly divided into k roughly equal groups (folds). The procedure then repeats k times: in iteration i, fold i is held out as the validation set and the model is trained on the remaining k-1 folds. The prediction error is computed on the held-out fold. The CV estimate of test error is the average of the k fold-specific errors: CV(k) = (1/k) * sum_{i=1}^{k} MSE_i, where MSE_i is the mean squared error (or other metric) on fold i.
Common choices are k = 5 or k = 10. With 5-fold CV, each fold contains 20% of the data, and each model is trained on 80% of the observations. With 10-fold CV, each fold contains 10%, and training uses 90%. Larger k uses more data for training in each iteration (reducing bias) but produces fold estimates that are more correlated (increasing variance).
A worked example: suppose you have 100 observations and use 5-fold CV with folds of 20 observations each. In iteration 1, train on observations from folds 2-5 (80 observations) and compute MSE_1 on fold 1 (20 observations). Repeat for folds 2 through 5. If the fold MSEs are 12.3, 14.1, 11.8, 13.5, and 12.9, then CV(5) = (12.3 + 14.1 + 11.8 + 13.5 + 12.9) / 5 = 12.92.
The One Standard Error Rule
When using CV for model selection, the simplest approach is to choose the model (or hyperparameter value) with the lowest CV error. However, the "one standard error rule" offers a more conservative alternative: select the simplest model whose CV error is within one standard error of the minimum CV error.
The standard error of the CV estimate is SE = sqrt(Var(MSE_1, ..., MSE_k) / k). If the minimum CV error across all candidate models is 12.92 with SE = 1.5, then any model with CV error below 12.92 + 1.5 = 14.42 is considered "statistically equivalent" to the best. Among these, choose the simplest (fewest parameters, smallest tree, or strongest regularization).
The rationale is that simpler models are more interpretable, more stable, and less likely to overfit. Since the CV error estimates have uncertainty, picking the absolute minimum often selects a model that is unnecessarily complex. This rule is particularly common in tree pruning and regularization parameter selection.
Leave-One-Out Cross-Validation (LOOCV)
LOOCV is the extreme case of k-fold CV with k = n (the sample size). Each observation is held out exactly once, and the model is trained on the remaining n-1 observations. The CV error is CV(n) = (1/n) * sum_{i=1}^{n} (y_i - y_hat_{-i})^2, where y_hat_{-i} is the prediction for observation i from the model trained without it.
LOOCV has very low bias because each training set uses n-1 observations, nearly all the data. However, it has high variance because the n training sets are almost identical (differing by only one observation), making the n fold estimates highly correlated.
For linear regression models, LOOCV can be computed without actually refitting the model n times. The formula CV = (1/n) * sum_{i=1}^{n} (e_i / (1 - h_ii))^2 uses the ordinary residuals e_i and the leverage values h_ii from the hat matrix H = X(X'X)^(-1)X'. This shortcut makes LOOCV computationally trivial for linear models.
For other models (trees, neural networks, etc.), LOOCV requires n separate model fits, which can be computationally prohibitive for large datasets. In these cases, 5-fold or 10-fold CV is preferred.
Bias-Variance Tradeoff in Choosing k
The choice of k involves a bias-variance tradeoff for the CV error estimate itself (not the model). With small k (e.g., k = 2), each training set uses only half the data, so the model is trained on less data than the full training set. This means the CV error is a biased (pessimistic) estimate of the true test error for a model trained on all the data. However, the fold estimates are less correlated, giving lower variance.
With large k (approaching LOOCV), the training sets are nearly the full dataset, so the bias is low. But the fold estimates are highly correlated (they share almost all the same training data), increasing the variance of the average.
In practice, k = 5 and k = 10 offer a good balance. Empirical studies (by Breiman, Tibshirani, and others) suggest that 10-fold CV provides a good tradeoff between bias and variance. Five-fold CV is slightly more biased but is faster and may be preferable for very large datasets or computationally expensive models.
Stratified and Repeated Cross-Validation
Stratified CV ensures that each fold has approximately the same distribution of the response variable as the full dataset. For classification, each fold should contain roughly the same proportion of each class. For regression, the response values should be spread across folds. Stratification is particularly important when classes are imbalanced (e.g., rare claims in insurance data), as a random split might produce folds with few or no positive cases.
Repeated CV runs the entire k-fold procedure multiple times with different random splits and averages the results. For example, "5 times repeated 10-fold CV" creates 50 different train-test splits, reducing the variance of the error estimate at the cost of more computation. This is useful when the dataset is small and a single run of CV produces noisy estimates.
Study Tips for Exam SRM
For Exam SRM, focus on these key concepts. First, understand why training error underestimates test error and why cross-validation provides a better estimate. Second, know the k-fold CV procedure and be able to describe it step by step. Third, understand the bias-variance tradeoff in choosing k (LOOCV has low bias but high variance; small k has higher bias but lower variance). Fourth, know the LOOCV shortcut formula for linear models using leverage values. Fifth, understand the one standard error rule and when it is applied. Common exam questions present CV results for several models and ask which model to select, or ask about the effect of changing k on the bias and variance of the CV estimate.