Robust Statistics: Handling Outliers in Insurance Data
Apply robust statistical methods to handle outliers and heavy-tailed insurance data for Exam MAS-I.
Why Robustness Matters in Insurance
Insurance data frequently contains outliers. Catastrophic claims can be thousands of times larger than typical claims. Data entry errors introduce extreme values. Heavy-tailed loss distributions generate occasional observations far from the mean. Standard statistical methods based on least squares or maximum likelihood can be severely affected by these outliers, producing misleading parameter estimates and invalid inference.
Consider fitting a mean to auto insurance claim data. If 99 claims range from 500 to 5,000 (average around 2,000) but one catastrophic claim is 500,000, the sample mean jumps to approximately 7,000, nearly four times the typical claim. The sample mean is not representative of the data at all. Robust statistics provide estimators that remain reliable even when a fraction of the data is contaminated by outliers.
Breakdown Point
The breakdown point of an estimator is the smallest fraction of the data that must be changed to make the estimator arbitrarily bad (arbitrarily large or small, or undefined). It measures the maximum proportion of "bad" observations the estimator can tolerate before it breaks down. The sample mean has a breakdown point of 1/n: changing just one observation to a sufficiently extreme value can make the mean arbitrarily large. This is the worst possible breakdown point.
The sample median has a breakdown point of approximately 50%, the best possible for any location estimator. You would need to replace more than half the data with extreme values before the median becomes arbitrarily bad. This makes the median far more robust than the mean for estimating the center of heavy-tailed distributions, though it is less efficient (higher variance) when the data actually is normally distributed.
Robust Location Estimators
The trimmed mean removes a fixed percentage of the largest and smallest observations before computing the average. A 10% trimmed mean discards the largest 10% and smallest 10%, then averages the remaining 80%. Its breakdown point equals the trimming proportion (10% in this case). The trimmed mean balances robustness and efficiency: it resists moderate contamination while still using most of the data.
The Winsorized mean replaces extreme observations with the nearest non-extreme value instead of discarding them. A 10% Winsorized mean replaces the largest 10% of observations with the value at the 90th percentile and the smallest 10% with the value at the 10th percentile, then computes the mean. This avoids the data loss of trimming while still limiting the influence of extremes.
Robust Scale Estimators
The sample standard deviation, like the sample mean, is highly sensitive to outliers (breakdown point 1/n). The median absolute deviation (MAD) provides a robust alternative: MAD = median(|x_i - median(x)|). To estimate the standard deviation of a normal distribution, multiply the MAD by 1.4826: sigma_hat = 1.4826 * MAD. The MAD has a breakdown point of 50%, making it far more robust than the standard deviation.
The interquartile range (IQR) = Q3 - Q1 is another robust scale measure with a 25% breakdown point. For a normal distribution, sigma = IQR / 1.349. Robust scale estimates are important in actuarial work for setting outlier detection thresholds, constructing robust confidence intervals, and initializing iterative robust estimation procedures.
The Influence Function
The influence function IF(x; T, F) measures the effect of adding a single observation at value x to an infinitely large sample from distribution F on the estimator T. Formally, IF(x; T, F) = lim_{epsilon to 0} (T((1-epsilon)*F + epsilon*delta_x) - T(F)) / epsilon, where delta_x is a point mass at x.
For the sample mean, IF(x) = x - mu, which is unbounded. Adding an extreme observation has an influence proportional to its distance from the mean. For the median, IF(x) = sign(x - median) / (2*f(median)), which is bounded regardless of how extreme x is. Bounded influence functions indicate robustness; unbounded influence functions indicate sensitivity to outliers. The concept helps compare estimators and design robust alternatives.
Robust Regression
In regression, a single outlying observation can dramatically tilt the fitted line or plane. Ordinary least squares (OLS) minimizes sum((y_i - x_i'*beta)^2), and the squared residual term means that large residuals dominate the objective function. M-estimation replaces the squared loss with a less sensitive loss function: minimize sum(rho(r_i / sigma)), where r_i is the i-th residual and rho is a chosen loss function.
Huber's loss function is the most common choice: rho(u) = u^2/2 for |u| <= c and c*|u| - c^2/2 for |u| > c, where c is a tuning constant (often c = 1.345 for 95% efficiency under normality). This function behaves like squared loss for small residuals and like absolute loss for large residuals, limiting the influence of extreme observations. The resulting estimator is more resistant to outliers than OLS while maintaining good efficiency for well-behaved data.
Least Trimmed Squares
Least trimmed squares (LTS) provides an even more robust regression method. It minimizes the sum of the h smallest squared residuals, where h is typically chosen as approximately n/2 + (p+1)/2. This gives a breakdown point of approximately 50%, the highest possible for regression. LTS effectively fits the regression to the "best" half of the data, ignoring the rest. It is computationally more expensive than M-estimation but provides much higher robustness.
Common Pitfalls
A frequent mistake is using robust methods indiscriminately. When data is clean and approximately normal, robust estimators have higher variance than standard methods (they "pay" for their robustness in efficiency). Always check for outliers first, and use robust methods when there is evidence of contamination or when the consequences of being misled by outliers are severe.
Another error is confusing robustness with resistance to model misspecification. Robust statistics protect against outliers in the data, not against using the wrong model. A robust estimator applied under a fundamentally wrong model will still give wrong answers; it will just be less affected by individual extreme observations.
Study Tips
Exam MAS-I covers basic robustness concepts, including breakdown points, influence functions, and robust alternatives to standard estimators. Memorize the breakdown points of the mean (1/n), median (50%), and trimmed mean (equal to the trimming proportion). Know Huber's loss function and how M-estimation differs from OLS. Understand the MAD as a robust alternative to the standard deviation. Be prepared to explain why robust methods are important for insurance data and to compare the robustness properties of different estimators.