Handling Missing Data in Actuarial Datasets

Types of Missing Data

Missing Completely at Random (MCAR): the probability of being missing does not depend on any observed or unobserved data. This is the strongest and rarest assumption. Missing at Random (MAR): the probability of being missing depends on observed data but not on the missing values themselves. For example, younger policyholders may be less likely to report certain information, but conditional on age, missingness is random. Missing Not at Random (MNAR): the probability of being missing depends on the unobserved value itself. For example, high-income individuals may be less likely to report income.

The type of missingness determines which methods are appropriate.

Simple Methods

Complete case analysis (listwise deletion): discard any observation with missing values. Valid under MCAR but loses information and can introduce bias otherwise. Mean/median imputation: replace missing values with the variable's mean or median. Preserves the sample size but distorts the distribution (reduces variance) and can bias relationships. Indicator method: add a binary variable indicating whether the value was missing and impute with a constant. This is commonly used in actuarial practice for tree-based models.

Advanced Methods

Multiple imputation: creates several plausible completed datasets, fits the model to each, and combines results using Rubin's rules. This properly accounts for the uncertainty due to imputation. Model-based imputation: uses regression or other models to predict missing values from observed variables. For tree-based methods (random forests, gradient boosting), some implementations handle missing values natively by learning optimal splits for missing data. For Exam SRM, understand the assumptions behind each method and when each is appropriate. In practice, understanding why data is missing is as important as choosing an imputation method.