Zero-Inflated and Hurdle Models for Insurance Data

The Excess Zeros Problem

Insurance claim count data frequently contains more zeros (policyholders with no claims) than standard count models predict. A Poisson distribution with mean 0.10 predicts P(Y = 0) = exp(-0.10) = 0.905, but the actual proportion of zero-claim policyholders might be 0.95 or higher. This discrepancy, called zero inflation, cannot be handled by simply adjusting the Poisson mean. A smaller mean would match the zero proportion but underestimate the frequency and size of the non-zero counts.

Two classes of models address excess zeros: zero-inflated models and hurdle models. Both separate the data-generating process into components, but they differ in their interpretation of why zeros occur and in their mathematical structure. Understanding these differences is important for choosing the right model for a given actuarial application.

Zero-Inflated Poisson (ZIP) Model

The zero-inflated Poisson (ZIP) model assumes the population consists of two latent subgroups. A fraction pi of the population is in the "always zero" group (structural zeros), meaning they cannot generate any events regardless of their covariates. The remaining fraction (1 - pi) follows a standard Poisson distribution. The probabilities are:

• P(Y = 0) = pi + (1 - pi) * exp(-lambda)
• P(Y = k) = (1 - pi) * exp(-lambda) * lambda^k / k! for k >= 1

The mean and variance of a ZIP random variable are E[Y] = (1 - pi) * lambda and Var(Y) = (1 - pi) * lambda * (1 + pi * lambda). Since the variance exceeds the mean (the factor (1 + pi * lambda) > 1), the ZIP model naturally accommodates overdispersion caused by excess zeros.

Interpreting the ZIP Model

The structural zero interpretation is crucial. In a fraud detection model, the always-zero group might represent legitimate policyholders who never commit fraud. In an auto insurance model, some policyholders might have garaged vehicles that are never driven, making claims impossible. The mixing parameter pi represents the probability that a randomly selected policyholder belongs to this zero-generating group.

Both pi and lambda can depend on covariates, potentially different ones. The zero-inflation parameter pi typically uses a logistic link: logit(pi) = gamma_0 + gamma_1 * w_1 + ... The Poisson mean lambda uses a log link: ln(lambda) = beta_0 + beta_1 * x_1 + ... This flexibility allows different risk factors to affect the probability of being a structural zero versus the claim rate among non-structural-zero policyholders.

Zero-Inflated Negative Binomial (ZINB) Model

If the non-zero subgroup still shows overdispersion beyond what the ZIP model can explain, the zero-inflated negative binomial (ZINB) model replaces the Poisson component with a negative binomial. This handles both excess zeros (through the zero-inflation parameter pi) and additional overdispersion in the count component (through the negative binomial dispersion parameter k).

The ZINB model is the most flexible standard count model. However, this flexibility comes at a cost: more parameters require more data for stable estimation, and distinguishing between the zero-inflation and overdispersion components can be difficult with limited data. In practice, start with the simpler models (Poisson, negative binomial, ZIP) and move to ZINB only if simpler models fail diagnostics.

Hurdle Models

Hurdle models take a different approach to excess zeros. Instead of mixing two subpopulations, they model the data-generating process in two explicit stages. Stage 1 is a binary model for whether any event occurs: P(Y = 0) = 1 - p and P(Y > 0) = p. Stage 2 is a truncated-at-zero count model for the positive counts: P(Y = k | Y > 0) follows a zero-truncated Poisson or negative binomial distribution.

The key difference from zero-inflated models is that hurdle models treat all zeros as coming from one process (the binary "hurdle"), while zero-inflated models allow zeros to come from two sources (structural zeros and sampling zeros from the Poisson component). In a hurdle model, once the hurdle is crossed (Y > 0), the count distribution governs how many events occur.

Interpreting Hurdle Models

Hurdle models have a natural two-stage interpretation that is often more intuitive than zero-inflated models. The binary component models the "propensity to claim": what determines whether a policyholder files any claim at all? The count component models the "claim intensity given at least one claim": once a policyholder starts claiming, how many claims do they file?

For example, in health insurance, the hurdle might represent the decision to seek medical care (influenced by access, health literacy, and copayment levels), while the count component represents the number of visits once care is initiated (influenced by condition severity and treatment protocols). Different covariates may matter for each component, and hurdle models allow this naturally.

Comparing ZIP and Hurdle Models

The choice between zero-inflated and hurdle models depends on the data-generating mechanism. If you believe there are truly two subpopulations (some units cannot generate events, others follow a count process), a zero-inflated model is appropriate. If you believe all units face the same type of process but must cross a threshold before events occur, a hurdle model is more appropriate.

From a practical standpoint, hurdle models are often easier to interpret because the two stages have clear real-world meanings. Zero-inflated models can be harder to explain because they imply that some observed zeros are "structural" (from the always-zero group) while others are "sampling" zeros (from the Poisson group that happened to produce zero). In many actuarial applications, this distinction is not clearly supported by the problem context.

Common Pitfalls

A frequent mistake is applying a zero-inflated model when a simpler model (negative binomial) would suffice. The negative binomial can accommodate moderate excess zeros through its overdispersion parameter. Test for zero inflation specifically (using the Vuong test or comparing AIC/BIC) before adding the complexity of a zero-inflated component.

Another error is confusing the interpretation of the zero-inflation parameter pi with the probability of zero claims. In a ZIP model, P(Y = 0) = pi + (1 - pi) * exp(-lambda), not pi alone. The parameter pi represents the structural zero probability, not the total zero probability. Students also sometimes forget that in hurdle models, the count component is a truncated distribution (conditioning on Y > 0), not the standard distribution.

Study Tips

Exam MAS-I tests your understanding of both model types, their differences, and their application to insurance count data. Be able to write out the probability functions for ZIP and hurdle Poisson models. Know how to compute E[Y] and Var(Y) for a ZIP model. Understand the conceptual difference: zero-inflated models mix two subpopulations, while hurdle models separate the process into two stages. Practice computing probabilities from both models given specific parameter values. Be prepared to recommend one model over the other for a given scenario and to justify your choice based on the data-generating mechanism.