Simple Linear Regression

Simple linear regression models the relationship between a response variable Y and a single predictor X: Y_i = beta_0 + beta_1 * X_i + epsilon_i, where epsilon_i ~ N(0, sigma^2) are independent errors. The ordinary least squares (OLS) estimators minimize the sum of squared residuals: beta_1-hat = sum((X_i - X-bar)(Y_i - Y-bar)) / sum((X_i - X-bar)^2) and beta_0-hat = Y-bar - beta_1-hat * X-bar. These estimators are unbiased and have the smallest variance among all linear unbiased estimators (by the Gauss-Markov theorem).

The coefficient beta_1 measures the expected change in Y for a one-unit increase in X. The intercept beta_0 is the expected value of Y when X = 0 (which may or may not have a meaningful interpretation depending on the context). The standard error of beta_1-hat is sigma / sqrt(sum(X_i - X-bar)^2), which decreases with more data and with greater spread in the X values.

Multiple Linear Regression

Multiple linear regression extends the model to p predictors: Y = X * beta + epsilon, where X is the n x (p+1) design matrix (including a column of ones for the intercept), beta is the (p+1) x 1 parameter vector, and epsilon ~ N(0, sigma^2 * I). The OLS estimator is beta-hat = (X'X)^{-1} * X'Y. The covariance matrix of beta-hat is sigma^2 * (X'X)^{-1}, with sigma^2 estimated by MSE = SSE/(n-p-1).

Each coefficient beta_j measures the expected change in Y for a one-unit increase in X_j, holding all other predictors constant. This "holding constant" interpretation is central to multiple regression and differs from the simple regression coefficient, which does not control for other variables. When predictors are correlated, the simple and multiple regression coefficients for the same variable can differ substantially.

R-squared and Model Significance

The coefficient of determination R^2 = SSR/SST = 1 - SSE/SST measures the proportion of variance in Y explained by the model. R^2 always increases (or stays the same) when a predictor is added, even if that predictor is useless, which makes it unreliable for comparing models with different numbers of predictors. Adjusted R^2 = 1 - (SSE/(n-p-1))/(SST/(n-1)) penalizes for additional predictors and can decrease when a useless predictor is added.

The overall F-test evaluates whether any predictor contributes significantly: F = (SSR/p) / (SSE/(n-p-1)). Under H0 (all slopes equal zero), F follows an F-distribution with (p, n-p-1) degrees of freedom. Individual t-tests evaluate each coefficient: t_j = beta_j-hat / SE(beta_j-hat), with n-p-1 degrees of freedom. A significant overall F-test with non-significant individual t-tests often signals multicollinearity.

Multicollinearity

Multicollinearity occurs when predictor variables are highly correlated with each other. It inflates the standard errors of regression coefficients, making individual t-tests unreliable even when the overall F-test is significant. The variance inflation factor (VIF) for predictor j is VIF_j = 1/(1 - R_j^2), where R_j^2 is the R-squared from regressing X_j on all other predictors. A VIF exceeding 5 or 10 indicates problematic multicollinearity.

Remedies for multicollinearity include removing one of the correlated predictors, combining correlated predictors into a single index, using principal components regression, or applying ridge regression. In actuarial applications, multicollinearity often arises when using multiple related rating variables (for example, age and years of driving experience in auto insurance).

Diagnostics and Residual Analysis

Residual analysis checks the four assumptions of linear regression: linearity, independence, normality, and constant variance (homoscedasticity). A plot of residuals versus fitted values should show no pattern; a curved pattern indicates non-linearity, and a funnel shape indicates heteroscedasticity. A QQ plot of residuals against normal quantiles assesses normality; departures from a straight line suggest non-normal errors.

Leverage measures how far an observation's X values are from the center of the data. High-leverage points have unusual predictor values and can disproportionately influence the regression line. Cook's distance combines leverage and residual size to measure the overall influence of each observation on the fitted model. Observations with Cook's distance exceeding 4/n warrant investigation as potential influential points.

Model Selection

Model selection criteria help choose among competing models. AIC = -2 * ln(L) + 2k balances goodness of fit against model complexity, where L is the maximized likelihood and k is the number of parameters. BIC = -2 * ln(L) + k * ln(n) applies a stronger penalty for complexity, favoring simpler models, especially for large n. Lower values of AIC or BIC indicate better models. Adjusted R^2 serves a similar purpose in a more intuitive scale.

Stepwise selection procedures automate variable selection. Forward selection starts with no predictors and adds the most significant one at each step. Backward elimination starts with all predictors and removes the least significant one at each step. Bidirectional (stepwise) selection combines both approaches. These procedures are convenient but can produce different final models depending on the direction and significance thresholds used.

Common Pitfalls

A frequent mistake is interpreting R^2 as a measure of model adequacy. A high R^2 does not guarantee that the model is correct; a misspecified model can still explain a large proportion of variance. Another common error is interpreting regression coefficients causally when the data is observational. Regression measures association, not causation, unless the study design supports causal inference.

Students also sometimes forget to check assumptions before reporting results. Regression results are unreliable when assumptions are violated, and standard errors, confidence intervals, and p-values all depend on the model being correctly specified.

Study Tips

Exam MAS-I tests regression mechanics, interpretation of computer output, and diagnostic evaluation. Practice reading regression output tables and extracting key information: coefficient estimates, standard errors, t-statistics, p-values, R^2, adjusted R^2, and the F-statistic. Be comfortable computing these quantities from summary statistics. Know how to interpret residual plots and identify assumption violations. Practice model comparison using AIC, BIC, and adjusted R^2, and understand why each criterion makes different trade-offs between fit and complexity.

Regression Analysis Review for Exam MAS-I