Linear Regression: Assumptions, Estimation, and Inference for Exam SRM

The Simple Linear Regression Model

Simple linear regression models the relationship Y = beta_0 + beta_1*X + epsilon, where epsilon ~ N(0, sigma^2) independently. The four key assumptions are: linearity (the true relationship is linear), independence (errors are independent), normality (errors are normally distributed), and constant variance (homoscedasticity: Var(epsilon) = sigma^2 for all X). These are often abbreviated as LINE.

The model separates the systematic component (beta_0 + beta_1*X) from the random component (epsilon). The intercept beta_0 represents the expected value of Y when X = 0. The slope beta_1 represents the expected change in Y for a one-unit increase in X. In actuarial applications, X might be driver age and Y might be expected claim cost, or X might be building age and Y might be expected maintenance expense.

The assumption that errors are normally distributed is needed for inference (confidence intervals and hypothesis tests) but not for the least squares estimates themselves. The Gauss-Markov theorem requires only linearity, independence, and constant variance to guarantee that OLS estimators are the best linear unbiased estimators.

Least Squares Estimation

The least squares estimators minimize the sum of squared residuals: SSE = sum of (Yi - beta_0 - beta_1*Xi)^2. Taking partial derivatives and setting them to zero yields the normal equations. The solutions are beta_1_hat = S_xy / S_xx, where S_xy = sum((Xi - X_bar)(Yi - Y_bar)) and S_xx = sum((Xi - X_bar)^2). The intercept is beta_0_hat = Y_bar - beta_1_hat * X_bar.

An equivalent formula for the slope is beta_1_hat = (sum(Xi*Yi) - n*X_bar*Y_bar) / (sum(Xi^2) - n*X_bar^2). The fitted regression line always passes through the point (X_bar, Y_bar). The residuals e_i = Yi - Y_hat_i sum to zero and are uncorrelated with the fitted values.

The computation can also be expressed using correlation: beta_1_hat = r * (S_Y / S_X), where r is the sample correlation coefficient and S_X, S_Y are the sample standard deviations of X and Y respectively.

Properties of OLS Estimators

Under the model assumptions, the OLS estimators are unbiased: E[beta_1_hat] = beta_1 and E[beta_0_hat] = beta_0. They are the Best Linear Unbiased Estimators (BLUE) by the Gauss-Markov theorem, meaning no other unbiased linear estimator has smaller variance. Var(beta_1_hat) = sigma^2 / S_xx and Var(beta_0_hat) = sigma^2 * (1/n + X_bar^2/S_xx).

The unbiased estimator of sigma^2 is MSE = SSE/(n-2), where SSE = sum of (Yi - Y_hat_i)^2 and the denominator reflects the two estimated parameters (beta_0 and beta_1). The standard error of beta_1_hat is SE(beta_1_hat) = sqrt(MSE / S_xx). These standard errors are used in constructing confidence intervals and test statistics.

Note that Var(beta_1_hat) decreases as S_xx increases. This means the slope estimate is more precise when the predictor values are more spread out. In actuarial contexts, having a diverse range of risk characteristics in the data improves the precision of the estimated relationships.

ANOVA Decomposition and R-Squared

The total variability in Y is decomposed as SST = SSR + SSE, where SST = sum((Yi - Y_bar)^2) is the total sum of squares, SSR = sum((Y_hat_i - Y_bar)^2) is the regression sum of squares, and SSE = sum((Yi - Y_hat_i)^2) is the error sum of squares.

The coefficient of determination R^2 = SSR/SST = 1 - SSE/SST measures the proportion of variance in Y explained by the linear relationship with X. In simple linear regression, R^2 = r^2, the square of the sample correlation coefficient. R^2 ranges from 0 (the model explains nothing) to 1 (the model explains everything).

Be cautious about interpreting R^2 in isolation. A high R^2 does not prove causation, nor does it mean the model is appropriate (a linear fit can have high R^2 even when the true relationship is nonlinear). A low R^2 does not mean X is unrelated to Y; it means other factors contribute substantial variability.

Hypothesis Testing and Confidence Intervals

To test H0: beta_1 = 0 (no linear relationship), use the t-statistic t = beta_1_hat / SE(beta_1_hat), which follows a t-distribution with n-2 degrees of freedom under H0. Reject H0 if |t| > t_{alpha/2, n-2} for a two-sided test. The p-value is the probability of observing a t-statistic as extreme as the computed value under H0.

The 100(1-alpha)% confidence interval for beta_1 is beta_1_hat +/- t_{alpha/2, n-2} * SE(beta_1_hat). Similarly for beta_0. These intervals quantify the uncertainty in the estimated parameters. On Exam SRM, you may be given the standard error and asked to compute the test statistic or confidence interval.

The F-test for overall model significance in simple regression is equivalent to the t-test for beta_1: F = MSR/MSE = t^2. This equivalence holds only in simple regression; in multiple regression, the F-test and individual t-tests answer different questions.

Prediction and Confidence Bands

Two types of intervals are used for predictions at a new value X = x_0. The confidence interval for E[Y|X=x_0] (the mean response) has standard error SE_mean = sqrt(MSE * (1/n + (x_0 - X_bar)^2/S_xx)). The prediction interval for a new individual observation Y_new has standard error SE_pred = sqrt(MSE * (1 + 1/n + (x_0 - X_bar)^2/S_xx)).

The prediction interval is always wider than the confidence interval because it accounts for both the uncertainty in estimating the regression line and the variability of individual observations around the line. Both intervals are narrowest at X = X_bar and widen as x_0 moves away from X_bar, reflecting extrapolation uncertainty.

On Exam SRM, understand the distinction between these two intervals. A confidence interval targets the average response; a prediction interval targets a single future observation. Using the wrong interval is a common exam mistake.

Study Tips and Common Pitfalls

The most common mistake is interpreting the slope as a causal effect. Regression measures association, not causation. Another frequent error is extrapolating the regression line far beyond the range of observed X values, where the linear relationship may not hold.

For Exam SRM, practice computing all the key quantities (beta_hat values, SSE, SSR, SST, R^2, standard errors, t-statistics) from summary statistics. Many problems provide sum(Xi), sum(Yi), sum(Xi^2), sum(Yi^2), sum(Xi*Yi), and n, from which you can compute everything. Also practice interpreting computer output tables that show coefficients, standard errors, t-statistics, and p-values. At least 8 practice problems covering estimation, testing, and prediction will build the fluency needed for exam day.