Hypothesis Testing Review for Exam SRM

The Hypothesis Testing Framework

A hypothesis test provides a formal procedure for using data to decide between two competing claims about a population parameter. The null hypothesis H0 represents the default position (often "no effect" or "no difference"), while the alternative hypothesis H1 represents the claim you are testing for. The test uses data to assess whether there is sufficient evidence to reject H0 in favor of H1.

The procedure involves four steps. Step 1: State H0 and H1. Step 2: Choose a test statistic and significance level alpha (typically 0.05). Step 3: Compute the test statistic from the data and determine the p-value. Step 4: Compare the p-value to alpha. If p-value < alpha, reject H0. If p-value >= alpha, fail to reject H0 (do not "accept" H0, because failing to find evidence against it is not the same as proving it true).

The p-value is the probability, under H0, of observing a test statistic at least as extreme as the one actually obtained. A small p-value (say 0.003) means the observed data would be very unlikely if H0 were true, providing strong evidence against H0. A large p-value (say 0.45) means the observed data is consistent with H0.

Type I and Type II Errors

Two types of errors can occur. A Type I error (false positive) occurs when H0 is true but the test incorrectly rejects it. The probability of a Type I error equals the significance level alpha. A Type II error (false negative) occurs when H0 is false but the test fails to reject it. The probability of a Type II error is denoted beta.

The power of a test is 1 - beta, the probability of correctly rejecting H0 when it is false. Higher power means the test is more likely to detect a real effect. Power depends on four factors: the sample size n (larger n increases power), the effect size (larger effects are easier to detect), the significance level alpha (larger alpha increases power but also increases the Type I error rate), and the variability of the data (less noise increases power).

In actuarial contexts, the consequences of each error type differ. In fraud detection, a Type I error (flagging a legitimate claim as fraud) wastes investigation resources and annoys honest policyholders. A Type II error (missing actual fraud) costs money. The relative costs influence the choice of alpha and the decision threshold.

A worked example: suppose you are testing whether a new rating factor improves a pricing model. H0: the coefficient beta = 0 (the factor has no effect). You set alpha = 0.05. If the true beta is 0.3 with standard error 0.15, the power calculation gives: under H1, the t-statistic is approximately 0.3/0.15 = 2.0. With large n, the power is P(Z > 1.96 - 2.0) + P(Z < -1.96 - 2.0) = P(Z > -0.04) = approximately 0.516. This modest power suggests you need a larger sample to reliably detect the effect.

t-Tests for Regression Coefficients

In linear regression, the t-test for an individual coefficient tests H0: beta_j = 0 against H1: beta_j != 0. The test statistic is t = beta_j_hat / SE(beta_j_hat), which follows a t-distribution with n - p - 1 degrees of freedom under H0 (where p is the number of predictors). Reject H0 if |t| exceeds the critical value t_{alpha/2, n-p-1}, or equivalently, if the p-value is less than alpha.

A significant t-test means the predictor has a statistically significant linear relationship with Y, controlling for the other predictors in the model. However, statistical significance does not imply practical significance. A coefficient can be statistically significant (small p-value) but practically unimportant (tiny effect size), especially with large datasets.

In logistic regression and GLMs, the analogous test is the Wald test: z = beta_j_hat / SE(beta_j_hat), which follows approximately a standard normal distribution for large samples.

F-Test for Overall Significance and Nested Model Comparison

The F-test for overall regression significance tests H0: beta_1 = beta_2 = ... = beta_p = 0 (no predictors are useful) against H1: at least one beta_j != 0. The test statistic is F = (SSR/p) / (SSE/(n-p-1)) = (R^2/p) / ((1-R^2)/(n-p-1)), which follows an F-distribution with p and n-p-1 degrees of freedom.

The partial F-test compares nested models (a full model and a reduced model that drops q predictors). The test statistic is F = ((SSE_reduced - SSE_full) / q) / (SSE_full / (n-p-1)), with q and n-p-1 degrees of freedom. This tests whether the q additional predictors in the full model are jointly significant.

Likelihood Ratio Test

The likelihood ratio test (LRT) compares nested models using their maximized likelihoods. The test statistic is G^2 = -2 * (l_reduced - l_full), where l denotes the log-likelihood. Under H0 (the reduced model is correct), G^2 follows a chi-squared distribution with degrees of freedom equal to the difference in the number of parameters.

The LRT is used extensively with GLMs, logistic regression, and other models estimated by maximum likelihood. It is generally more reliable than the Wald test, especially for small samples or when coefficients are large. For Exam SRM, the LRT is the primary tool for comparing nested models in the GLM framework.

A worked example: a logistic regression model with 5 predictors has log-likelihood l_full = -312.5. Dropping two predictors gives l_reduced = -315.8. The test statistic is G^2 = -2(-315.8 - (-312.5)) = -2(-3.3) = 6.6. Under H0, this follows chi-squared with 2 degrees of freedom. The critical value at alpha = 0.05 is 5.99. Since 6.6 > 5.99, reject H0 and conclude the two predictors are jointly significant.

Multiple Testing Issues

When conducting many hypothesis tests simultaneously, the probability of at least one false positive (incorrectly rejecting a true H0) increases rapidly. If you perform m independent tests each at significance level alpha, the family-wise error rate (FWER) is 1 - (1 - alpha)^m. For m = 20 tests at alpha = 0.05, the FWER is 1 - 0.95^20 = 0.642, meaning there is a 64.2% chance of at least one false positive.

The Bonferroni correction addresses this by testing each hypothesis at level alpha/m instead of alpha. With m = 20 and overall alpha = 0.05, each test uses alpha = 0.0025. This controls the FWER at alpha but is conservative (it has lower power to detect real effects).

In the context of Exam SRM, multiple testing is particularly relevant for stepwise variable selection. Forward selection, backward elimination, and stepwise regression involve many implicit hypothesis tests (testing each candidate variable for addition or removal at each step). This inflates the Type I error rate, meaning the selected model may include variables that are not truly significant. This is one important motivation for using AIC, BIC, or cross-validation for variable selection instead of relying solely on p-values.

Study Tips for Exam SRM

For Exam SRM, be fluent in the following. First, the hypothesis testing framework: stating hypotheses, computing test statistics, interpreting p-values, and understanding Type I/II errors. Second, the t-test for individual regression coefficients and its interpretation. Third, the F-test for comparing nested linear models. Fourth, the likelihood ratio test for comparing nested GLMs. Fifth, the Bonferroni correction and why multiple testing inflates false positive rates. Practice computing test statistics and making decisions, and be prepared for questions that ask you to compare the t-test, F-test, and likelihood ratio test in different contexts.