Chi-Squared, Kolmogorov-Smirnov, and Anderson-Darling Tests
Compare the three main goodness-of-fit tests used for loss distribution validation on Exam STAM.
Overview of Goodness-of-Fit Testing
After fitting a parametric distribution to loss data, you need objective methods to evaluate whether the model provides an adequate fit. Goodness-of-fit tests formalize this evaluation by comparing the observed data to what the fitted model predicts. If the discrepancy between observed and expected is too large, the test rejects the null hypothesis that the data came from the fitted distribution. Three tests dominate the Exam STAM syllabus: chi-squared, Kolmogorov-Smirnov (KS), and Anderson-Darling (AD).
Each test measures the discrepancy differently, which means they have different strengths and sensitivities. The choice of test depends on the data format (grouped vs. individual), the sample size, and which aspects of fit matter most for the application. In insurance, accurate tail modeling is often critical, which influences which test is most appropriate.
The Chi-Squared Test
The chi-squared test groups data into c intervals and compares observed counts O_j to expected counts E_j = n * (F(c_j) - F(c_{j-1})), where F is the fitted CDF and c_j are the interval boundaries. The test statistic is Q = sum from j=1 to c of (O_j - E_j)^2 / E_j. Under the null hypothesis that the data follows the fitted distribution, Q approximately follows a chi-squared distribution with c - 1 - p degrees of freedom, where p is the number of parameters estimated from the data.
For the chi-squared approximation to be valid, each expected count E_j should be at least 5. If some intervals have expected counts below 5, adjacent intervals should be merged. This requirement means the test needs reasonably large samples and that very fine groupings are inappropriate for small data sets.
For example, suppose you fit an exponential distribution with theta_hat = 1500 to 200 loss observations and use 5 intervals: [0,500), [500,1000), [1000,2000), [2000,4000), and [4000, infinity). You compute E_j for each interval using the fitted exponential CDF. With p = 1 estimated parameter, degrees of freedom = 5 - 1 - 1 = 3. If Q = 6.2 and the chi-squared critical value at the 5% level with 3 degrees of freedom is 7.815, you do not reject the null hypothesis, concluding that the exponential model fits adequately at the 5% significance level.
Choosing Intervals for Chi-Squared
The results of the chi-squared test depend on how the data is grouped, which introduces an element of subjectivity. Different groupings can lead to different conclusions. Common approaches include equal-probability intervals (each interval has the same expected count under the fitted model) and equal-width intervals. Equal-probability intervals are generally preferred because they ensure each cell contributes roughly equally to the test statistic and all expected counts are adequate.
The number of intervals also matters. Too few intervals may miss localized poor fit, while too many intervals produce small expected counts that violate the chi-squared approximation. A common guideline is to use between 5 and 15 intervals for typical insurance data sets. Exam STAM problems typically specify the intervals, so you do not need to choose them yourself, but understanding the trade-offs helps you interpret results.
The Kolmogorov-Smirnov Test
The KS test uses the maximum absolute difference between the empirical CDF F_n(x) and the fitted CDF F(x). The test statistic is D = max over all x of |F_n(x) - F(x)|. Because F_n is a step function, the maximum occurs at one of the data points. In practice, D is computed by checking two quantities at each ordered observation x_i: D_i^+ = |i/n - F(x_i)| and D_i^- = |(i-1)/n - F(x_i)|, then taking D = max over all i of max(D_i^+, D_i^-).
The KS test works with ungrouped (individual) data and avoids the arbitrary grouping decisions required by the chi-squared test. Its critical values depend on the sample size n. Standard KS tables assume the distribution is fully specified (parameters known, not estimated from data). When parameters are estimated, the critical values must be adjusted downward (the distribution of D is stochastically smaller), though Exam STAM often provides the appropriate critical values.
The main limitation of the KS test is that it is most sensitive to discrepancies near the center of the distribution, where the empirical CDF is changing most rapidly. It is relatively insensitive to differences in the tails, which is a significant drawback for insurance applications where tail accuracy is paramount.
The Anderson-Darling Test
The Anderson-Darling test addresses the KS test's weakness in the tails by applying a weight function that emphasizes tail deviations. The test statistic is A^2 = -n - (1/n) * sum from i=1 to n of (2i-1) * [ln(F(x_{(i)})) + ln(1 - F(x_{(n+1-i)}))], where x_{(1)} <= x_{(2)} <= ... <= x_{(n)} are the ordered observations.
The weight function underlying the AD test is 1/[F(x)*(1-F(x))], which becomes very large when F(x) is near 0 or 1, corresponding to the extreme tails of the distribution. This means that a single poorly modeled observation in the tail can produce a large AD statistic, making the test highly sensitive to tail misfit.
For insurance applications, the Anderson-Darling test is often the most informative choice. If the fitted distribution understates the probability of large losses, the AD test is more likely to detect this problem than either the chi-squared test (which averages discrepancies across intervals) or the KS test (which focuses on the median region). Critical values for the AD test depend on the sample size and the distribution family being tested, and Exam STAM problems typically provide them.
Comparing the Three Tests
Each test has distinct advantages and limitations that make it suitable for different situations. The chi-squared test is the most versatile: it works with both grouped and individual data (after grouping), handles any distribution family, and has well-known critical values. However, its results depend on the choice of intervals. The KS test requires no grouping decisions and works with individual data, but is insensitive to tail discrepancies. The AD test provides superior tail sensitivity and also works with individual data, but has more complex critical values and requires the data to be ungrouped.
In terms of data requirements, the chi-squared test can handle grouped data where individual observations are not available, which is common in insurance data reported by loss range. The KS and AD tests both require individual observations. If you have individual data and tail accuracy is important, the AD test is generally the best choice. If you only have grouped data, the chi-squared test is your option.
Common Pitfalls
Students frequently forget to subtract the number of estimated parameters p from the chi-squared degrees of freedom. If you estimate two parameters and use six intervals, the degrees of freedom are 6 - 1 - 2 = 3, not 5. This adjustment is necessary because parameter estimation uses information from the data, reducing the effective degrees of freedom for the test.
Another common error is miscomputing the KS statistic. Remember to check both D_i^+ and D_i^- at each data point, not just one of them. The maximum discrepancy can occur either just before or just after a jump in the empirical CDF. Also be careful about the ordering: the observations must be sorted from smallest to largest before computing the statistic.
For the AD test, a frequent mistake is getting the index wrong in the summation formula. Note that the formula uses both F(x_{(i)}) and F(x_{(n+1-i)}), which pairs the smallest and largest observations, the second smallest and second largest, and so on. Carefully tracking these indices is essential for correct computation.
Study Tips
For Exam STAM, practice computing all three test statistics from small data sets (5 to 10 observations). Set up your work in a structured table with columns for i, x_{(i)}, F(x_{(i)}), i/n, (i-1)/n, and the relevant differences. Memorize the chi-squared degrees of freedom formula and know the critical values for common significance levels. Understand which test is most appropriate for each situation and be prepared to explain your reasoning. When a problem does not specify which test to use, consider the data format and the importance of tail accuracy to guide your choice.