Handling Missing Data in Actuarial Datasets

Types of Missing Data

Understanding why data is missing is the first step in choosing an appropriate handling method. The statistical framework distinguishes three types. Missing Completely at Random (MCAR): the probability of being missing does not depend on any observed or unobserved data. This is the strongest and rarest assumption. An example would be data lost due to a random computer malfunction unrelated to the data values.

Missing at Random (MAR): the probability of being missing depends on observed data but not on the missing values themselves. For example, younger policyholders may be less likely to report their income, but conditional on age (which is observed), the missingness does not depend on the actual income level. MAR is the most commonly assumed mechanism in practice.

Missing Not at Random (MNAR): the probability of being missing depends on the unobserved value itself. For example, high-income individuals may be less likely to report income precisely because it is high. MNAR is the most problematic because no fully satisfactory statistical correction exists without additional modeling assumptions about the missingness mechanism.

Diagnosing the Missingness Mechanism

Determining the type of missingness is crucial but often difficult. You can test for MCAR by comparing the distribution of observed variables between observations with and without missing values. If these distributions differ significantly, MCAR is unlikely. For example, if observations missing income tend to be younger, missingness depends on age, ruling out MCAR.

Distinguishing MAR from MNAR is more challenging because it involves the relationship between missingness and the unobserved values, which by definition cannot be directly tested. Domain knowledge is essential: ask whether it is plausible that the probability of missingness depends on the missing value itself. In insurance applications, understanding the data collection process helps identify the mechanism.

Creating a missingness pattern matrix (showing which variables are missing for each observation) can reveal systematic patterns. If certain variables tend to be missing together, this suggests a common cause that can inform the choice of handling method.

Complete Case Analysis

Complete case analysis (listwise deletion) discards any observation with at least one missing value. Under MCAR, this produces unbiased estimates because the complete cases are a random sample of the full dataset. However, it loses information (reduced sample size, wider confidence intervals) and can be severely biased under MAR or MNAR.

In actuarial datasets, complete case analysis can be particularly problematic when missingness is concentrated in certain subgroups. For example, if newer policyholders are more likely to have missing driving history information, dropping them biases the model toward the characteristics of longer-tenured policyholders.

Despite its limitations, complete case analysis is appropriate when the proportion of missing data is very small (less than 5%) and MCAR is plausible. It has the advantage of simplicity and does not require any modeling assumptions beyond MCAR.

Single Imputation Methods

Mean or median imputation replaces missing values with the variable's mean (for continuous variables) or mode (for categorical variables). This preserves the sample size but distorts the distribution by reducing variance and weakening correlations between variables. It can also bias regression coefficients because the imputed values all cluster at the center.

Regression imputation predicts missing values using a regression model fitted on the observed data. This preserves relationships between variables better than mean imputation but still underestimates variance because the imputed values fall exactly on the regression line with no residual noise. Adding random noise (stochastic regression imputation) partially addresses this.

The indicator (dummy variable) method adds a binary variable indicating whether the original value was missing and replaces missing values with a constant (often zero or the mean). This is commonly used in actuarial practice, especially for tree-based models, because it allows the model to learn whether missingness itself is informative. However, it can bias coefficient estimates in linear models.

Multiple Imputation

Multiple imputation addresses the limitations of single imputation by properly accounting for the uncertainty due to missing values. The process has three steps: (1) create m (typically 5 to 20) plausible completed datasets, each with slightly different imputed values drawn from the predictive distribution of the missing data; (2) fit the model to each completed dataset separately; (3) combine results using Rubin's rules, which pool the coefficient estimates and adjust the standard errors to reflect both within-imputation and between-imputation variability.

The pooled estimate is the average of the m individual estimates. The pooled variance accounts for the variation across imputations: Var_total = Var_within + (1 + 1/m) * Var_between, where Var_within is the average of the m variance estimates and Var_between is the variance of the m point estimates. This properly inflates the standard errors to reflect imputation uncertainty.

Multiple imputation is valid under MAR and produces approximately unbiased estimates with correctly sized confidence intervals. It is the gold standard for handling missing data in most statistical applications.

Tree-Based and Algorithm-Specific Approaches

Some modeling algorithms handle missing values natively without requiring preprocessing. Certain implementations of gradient boosting (such as XGBoost and LightGBM) learn optimal split directions for missing data during training. When a variable is selected for a split, observations with missing values in that variable are sent to whichever child node produces the better fit. This often outperforms explicit imputation.

Random forests can use a proximity-based imputation approach: observations that frequently land in the same terminal node are considered similar, and missing values are imputed using observed values from similar observations. k-nearest neighbors imputation uses the k most similar complete observations (based on other variables) to impute the missing value.

For Exam SRM, understanding when algorithms handle missing data natively versus when explicit imputation is needed is an important practical distinction. Tree-based methods are generally more robust to missing data than linear models or SVMs.

Study Tips and Common Pitfalls

The most common mistake is using mean imputation without considering its consequences. While convenient, it distorts the data distribution and can bias model results. Another frequent error is applying a missing data method that assumes MCAR when the data is actually MAR or MNAR, leading to biased conclusions.

On Exam SRM, be able to identify the type of missingness from a problem description and recommend an appropriate method. Know the advantages and disadvantages of each approach: complete case analysis (simple but wasteful), mean imputation (preserves n but distorts distribution), multiple imputation (gold standard but complex), and algorithm-specific methods (convenient but not universally available). Practice at least 4 problems involving missing data scenarios and method selection.