Statistics and Regression Cheat Sheet

Linear Regression

• OLS estimates: beta-hat = (X'X)^(-1) X'Y
• R-squared: R^2 = 1 - SSE/SST = SSR/SST
• Adjusted R-squared: R^2_adj = 1 - (1-R^2)(n-1)/(n-p-1)
• F-statistic: F = (SSR/p) / (SSE/(n-p-1))
• t-statistic: t = beta-hat_j / SE(beta-hat_j)
• VIF: VIF_j = 1/(1 - R^2_j), values > 10 indicate multicollinearity

Generalized Linear Models

• Components: Random (Y from exponential family), Systematic (eta = X beta), Link (g(mu) = eta)
• Canonical links: Normal: identity, Poisson: log, Binomial: logit, Gamma: inverse
• Deviance: D = 2(log L_saturated - log L_fitted)
• AIC: AIC = -2 log L + 2p
• BIC: BIC = -2 log L + p ln(n)

Logistic Regression

• Model: log(p/(1-p)) = beta_0 + beta_1 x_1 + ...
• Odds ratio: OR = e^(beta_j) for a one-unit increase in x_j
• Classification threshold: default 0.5, adjust based on costs

Model Selection

• Forward selection: Start empty, add variable with smallest p-value
• Backward elimination: Start full, remove variable with largest p-value
• AIC: Lower is better, penalizes complexity less than BIC
• Cross-validation: k-fold CV error estimates out-of-sample performance