Principal Component Analysis: Dimensionality Reduction for Exam SRM

What Is PCA?

Principal Component Analysis (PCA) is an unsupervised technique that transforms p correlated variables into p uncorrelated linear combinations called principal components (PCs). The PCs are ordered so that the first captures the maximum possible variance, the second captures the maximum remaining variance (subject to being orthogonal to the first), and so on. PCA is used for dimensionality reduction, data visualization, noise reduction, and preprocessing before regression.

The key insight is that when variables are correlated, the information they carry is partially redundant. PCA exploits this redundancy by finding directions in the data space along which variation is greatest. By keeping only the first few PCs (which capture most of the total variance), you can represent the data in fewer dimensions with minimal information loss.

Mathematical Foundation

Mathematically, the principal components are the eigenvectors of the covariance matrix (or correlation matrix) of the data. If Sigma is the p x p covariance matrix, the eigendecomposition gives Sigma = V * Lambda * V', where V is the matrix of eigenvectors (the PC directions) and Lambda is the diagonal matrix of eigenvalues. The k-th eigenvalue lambda_k equals the variance explained by the k-th PC.

Total variance is preserved: lambda_1 + lambda_2 + ... + lambda_p = sum of original variances. The proportion of variance explained by the k-th PC is lambda_k / (lambda_1 + ... + lambda_p). The cumulative proportion of variance explained by the first K components is (lambda_1 + ... + lambda_K) / (lambda_1 + ... + lambda_p).

For example, suppose you have 5 variables and the eigenvalues are 3.2, 1.5, 0.8, 0.3, 0.2 (total = 6.0). The first PC explains 3.2/6.0 = 53.3% of the variance, the first two together explain (3.2 + 1.5)/6.0 = 78.3%, and the first three explain (3.2 + 1.5 + 0.8)/6.0 = 91.7%. In this case, three components capture over 90% of the information in five variables.

Each PC is a linear combination of the original variables: PC_k = v_{k1}*X_1 + v_{k2}*X_2 + ... + v_{kp}*X_p, where the coefficients (loadings) v_{ki} come from the k-th eigenvector. The loadings tell you which original variables contribute most to each component and help with interpretation.

Standardization: Covariance vs. Correlation Matrix

A critical decision in PCA is whether to use the covariance matrix or the correlation matrix. This choice is equivalent to deciding whether to standardize the variables before applying PCA.

If variables are measured on different scales or in different units (e.g., claim count vs. premium amount in dollars vs. policyholder age in years), you must standardize first (subtract the mean, divide by the standard deviation) and use the correlation matrix. Without standardization, variables with larger variance dominate the first PCs regardless of their importance. For instance, a premium variable ranging from 0 to 100,000 would overwhelm an age variable ranging from 18 to 85.

If all variables are measured in the same units and on comparable scales, PCA on the covariance matrix is appropriate and preserves the original variance structure. In actuarial practice, standardization is almost always necessary because rating variables have diverse units.

When using the correlation matrix (standardized variables), each variable has variance 1, and the sum of eigenvalues equals p (the number of variables). The Kaiser criterion, which retains PCs with eigenvalues greater than 1, has a natural interpretation in this context: keep a component only if it explains more variance than a single original variable.

Choosing the Number of Components

Three common methods guide the choice of how many PCs to retain. The scree plot displays eigenvalues in decreasing order. Look for an "elbow" where the eigenvalues transition from large to small; retain the components before the elbow. This is somewhat subjective but widely used.

The cumulative variance threshold retains enough PCs to explain a target percentage of total variance, typically 80% or 90%. This is straightforward and objective, but the right threshold depends on the application.

The Kaiser criterion retains PCs with eigenvalues greater than 1 (when using the correlation matrix). The rationale is that a component should explain at least as much variance as a single standardized variable to be worth keeping. This method can over-retain or under-retain in some situations, so it should be used alongside other criteria.

In practice, these methods may suggest different numbers of components. Use domain judgment alongside the statistical criteria. If the goal is dimensionality reduction for a subsequent regression, cross-validation of the overall model's prediction error provides the most reliable guidance.

Principal Component Regression (PCR)

Principal component regression combines PCA with linear regression. Instead of regressing Y on the original predictors X_1, ..., X_p, you first compute the PCs and then regress Y on the first K principal components (where K < p). This addresses multicollinearity because the PCs are uncorrelated by construction, and it reduces dimensionality when K is much smaller than p.

The procedure is: (1) standardize the predictors, (2) compute PCs, (3) choose K components to retain, (4) regress Y on PC_1, ..., PC_K. The resulting coefficients can be back-transformed to the original variable scale, but interpretation is often done in terms of the PCs themselves.

A limitation of PCR is that the PCs are chosen to explain variance in X, not to predict Y. A component that explains little variance in X might be highly predictive of Y, and PCR would discard it. Partial least squares (PLS) regression addresses this by finding components that maximize covariance with Y rather than variance in X.

Interpreting Principal Components

The loadings (eigenvector coefficients) tell you what each PC represents. If the first PC has large positive loadings on all variables, it represents an overall level or size factor. If the second PC has positive loadings on some variables and negative loadings on others, it represents a contrast between those groups.

For example, in auto insurance with variables for driver age, vehicle age, vehicle value, and annual mileage, the first PC might load positively on all four (an overall "exposure" factor), while the second PC might contrast vehicle value (positive loading) against vehicle age (negative loading), representing a "vehicle quality" factor.

Be cautious about over-interpreting PCs, especially when loadings are similar in magnitude across many variables. Also note that the sign of a PC is arbitrary (multiplying all loadings by -1 gives an equally valid component), so focus on relative magnitudes and directions.

Study Tips for Exam SRM

For Exam SRM, focus on these essential PCA concepts. First, understand the eigenvalue decomposition and how eigenvalues relate to variance explained. Second, know when to use the covariance versus correlation matrix (standardize when variables have different scales). Third, be able to apply the Kaiser criterion and interpret a scree plot. Fourth, understand PCR as an alternative to variable selection for handling multicollinearity. Common exam questions give eigenvalues and ask for the proportion of variance explained, present loadings and ask for interpretation, or ask you to choose between PCA on the covariance versus correlation matrix given a description of the data. Practice these until they are routine.