Regularization: Ridge, Lasso, and Elastic Net for Exam SRM

Why Regularization?

Regularization modifies the model fitting process by adding a penalty on the magnitude of the coefficients. This trades a small increase in bias for a potentially large decrease in variance, often resulting in better prediction on new data. Regularization is particularly valuable when the number of predictors is large relative to the sample size, when predictors are highly correlated (multicollinearity), or when the goal is to identify the most important variables among many candidates.

Without regularization, OLS estimates minimize only the training error (sum of squared residuals). Adding a penalty term shifts the objective toward solutions that fit the data reasonably well while keeping coefficients small. This constrains the model's complexity, reducing its tendency to overfit the training data.

All regularization methods require standardizing the predictors before fitting (subtracting the mean and dividing by the standard deviation). Without standardization, the penalty treats predictors on different scales unequally, penalizing variables with large scales more heavily regardless of their importance. The intercept is typically not penalized.

Ridge Regression (L2 Penalty)

Ridge regression modifies the OLS objective by adding an L2 penalty: minimize sum_{i=1}^{n} (y_i - x_i' * beta)^2 + lambda * sum_{j=1}^{p} beta_j^2. The penalty term lambda * sum(beta_j^2) penalizes large coefficient values, pushing them toward zero without setting them exactly to zero.

The ridge solution has a closed form: beta_ridge = (X'X + lambda * I)^(-1) * X'Y. Compared to the OLS solution (X'X)^(-1) * X'Y, the addition of lambda * I to X'X makes the matrix invertible even when X'X is singular or near-singular. This is why ridge regression is so effective for multicollinearity: it directly addresses the numerical instability that causes the problem.

The tuning parameter lambda controls the strength of regularization. When lambda = 0, the solution equals OLS. As lambda increases, coefficients shrink toward zero, reducing variance but increasing bias. In the limit as lambda approaches infinity, all coefficients equal zero. The coefficient paths (plotting each beta_j against lambda) show smooth curves approaching zero, providing insight into which variables are most important (those that remain large for larger lambda values resist shrinkage).

A worked example: with two correlated predictors (correlation 0.95), OLS might give beta_1 = 15.3 and beta_2 = -12.8, with standard errors of 8.5 and 8.1. Ridge regression with lambda = 5 might give beta_1 = 3.2 and beta_2 = -0.8, with standard errors of 1.2 and 1.1. The ridge estimates are much more stable, though slightly biased.

LASSO Regression (L1 Penalty)

The LASSO (Least Absolute Shrinkage and Selection Operator) uses an L1 penalty: minimize sum_{i=1}^{n} (y_i - x_i' * beta)^2 + lambda * sum_{j=1}^{p} |beta_j|. The crucial difference from ridge is the absolute value penalty instead of the squared penalty.

The L1 penalty has a remarkable property: it can shrink coefficients exactly to zero, performing automatic variable selection. As lambda increases, more and more coefficients are set to exactly zero, producing increasingly sparse models. This makes the LASSO valuable for identifying the most important predictors among many candidates.

The geometric explanation for this sparsity property is informative. The L1 constraint region (the set of beta values with sum(|beta_j|) <= t for some t) has corners at the coordinate axes. The OLS solution, when projected onto this constraint region, tends to hit a corner, where one or more coordinates are exactly zero. The L2 constraint region (a sphere) has no corners, so the projected solution rarely has exact zeros.

Unlike ridge, the LASSO has no closed-form solution. It is typically solved using coordinate descent, which iteratively optimizes each coefficient while holding the others fixed. The computation is fast and efficient even with many predictors.

A limitation of the LASSO arises with groups of highly correlated predictors. The LASSO tends to select one predictor from the group and set the others to zero, and the selection can be unstable (a different member of the group might be selected with slightly different data). Ridge regression handles correlated groups more gracefully by shrinking all of them together.

Elastic Net

The elastic net combines the L1 and L2 penalties to get the best of both worlds: minimize sum_{i=1}^{n} (y_i - x_i' * beta)^2 + lambda * [alpha * sum(|beta_j|) + (1 - alpha) * sum(beta_j^2) / 2]. The mixing parameter alpha (between 0 and 1) controls the blend: alpha = 1 gives pure LASSO, alpha = 0 gives pure ridge.

The elastic net retains the variable selection property of the LASSO (it can set coefficients to exactly zero) while inheriting the stability of ridge regression for groups of correlated predictors. When correlated predictors are present, the elastic net tends to select or exclude the entire group together, rather than arbitrarily picking one as the LASSO does.

Two parameters must be tuned: lambda (overall regularization strength) and alpha (the L1/L2 mixing ratio). Both are typically selected by cross-validation, often using a grid search over (alpha, lambda) pairs. In practice, alpha values of 0.5, 0.7, and 0.9 are common starting points.

Choosing the Tuning Parameter

The regularization parameter lambda is almost always chosen by cross-validation. For each candidate value of lambda, compute the k-fold CV error, and select the lambda that minimizes the CV error. Many implementations provide a "lambda.min" (the lambda that minimizes CV error) and a "lambda.1se" (the largest lambda within one standard error of the minimum, producing a simpler model).

The coefficient path plot (regularization path) is a useful diagnostic. It shows how each coefficient changes as lambda varies from 0 (OLS) to large values (all zeros). For LASSO and elastic net, you can see when each variable enters or leaves the model. Variables that enter early (at large lambda) are the most important predictors.

Comparison Summary

• Ridge: shrinks all coefficients, never zeros them out. Best for prediction with many correlated predictors. Closed-form solution.
• LASSO: shrinks and can zero out coefficients. Best for variable selection when few predictors are truly important. No closed-form; solved by coordinate descent.
• Elastic Net: combines shrinkage and selection. Best when there are groups of correlated predictors and variable selection is desired. Two tuning parameters.

All three methods require standardizing predictors and selecting tuning parameters by cross-validation.

Study Tips for Exam SRM

For Exam SRM, prioritize understanding the conceptual differences between ridge and LASSO. Know that ridge shrinks but never zeros out coefficients, while LASSO can perform variable selection. Understand the role of the tuning parameter lambda and how cross-validation is used to select it. Be able to interpret a coefficient path plot. Know that elastic net combines the benefits of both methods and is particularly useful with correlated predictors. Common exam questions ask you to choose between ridge and LASSO for a given scenario, interpret the effect of increasing lambda, or explain why standardization is necessary before applying regularization.