Advanced Regression: Tweedie and Zero-Inflated Models

Tweedie Distribution

The Tweedie distribution is an exponential dispersion model with variance function V(mu) = mu^p, where p is the power parameter. Different values of p yield different distributions: p = 0 gives the Normal, p = 1 gives the Poisson, values of p between 1 and 2 give the compound Poisson-Gamma (most useful for insurance), p = 2 gives the Gamma, and p = 3 gives the Inverse Gaussian. The Tweedie family provides a unifying framework that connects several distributions commonly used in actuarial modeling.

The compound Poisson-Gamma case (1 < p < 2) is particularly important for insurance applications because it naturally handles data that contains both exact zeros and positive continuous values. Insurance pure premium data has exactly this structure: most policyholders have zero claims in a given period (contributing the point mass at zero), while those who do have claims generate positive loss amounts (contributing the continuous component). The Tweedie GLM models the pure premium directly without requiring separate frequency and severity models.

Tweedie GLM Implementation

In a Tweedie GLM, the response variable Y (typically the pure premium per exposure) follows a Tweedie distribution with E[Y] = mu and Var(Y) = phi * mu^p, where phi is the dispersion parameter. A log link function is standard: ln(mu_i) = x_i' * beta. The exposure enters as a weight in the likelihood rather than as an offset, unlike the Poisson frequency model.

The power parameter p can be estimated from the data using profile likelihood or can be fixed based on domain knowledge. When p is close to 1, the data resembles Poisson-like behavior with many zeros and small positive values. When p is close to 2, the data looks more Gamma-like with fewer zeros and larger positive values. The estimated p often falls between 1.5 and 1.8 for auto insurance pure premium data, reflecting a moderate proportion of zeros and a right-skewed distribution of positive losses.

One advantage of the Tweedie approach over separate frequency-severity modeling is parsimony. Instead of fitting two models (one for claim probability or frequency, one for claim severity) and combining them, the Tweedie GLM fits a single model. This can be beneficial when the number of observations is limited or when the covariates affecting frequency and severity are the same. However, separate models offer more flexibility when different factors drive frequency and severity.

Zero-Inflated Models

Zero-inflated models address the common situation where a dataset contains more zeros than a standard count distribution would predict. The zero-inflated Poisson (ZIP) model assumes there are two types of zeros: structural zeros (from a population that never produces events, such as fraudulent policies or data errors) and sampling zeros (from a population that could produce events but happened not to in the observation period). The model has two components: a binary model (typically logistic regression) for the probability pi of being a structural zero, and a Poisson model with parameter lambda for the count process among non-structural zeros.

The probability mass function of the ZIP model is: P(Y = 0) = pi + (1 minus pi) * exp(-lambda) and P(Y = k) = (1 minus pi) * exp(-lambda) * lambda^k / k! for k >= 1. The mean is E[Y] = (1 minus pi) * lambda and the variance is Var(Y) = (1 minus pi) * lambda * (1 + pi * lambda). Since pi * lambda > 0, the variance always exceeds the mean, so the ZIP model handles overdispersion arising from excess zeros.

Zero-Inflated Negative Binomial and Hurdle Models

The zero-inflated negative binomial (ZINB) model extends the ZIP by replacing the Poisson component with a negative binomial, handling both excess zeros and additional overdispersion beyond what the zeros alone explain. This is useful when the positive counts also show more variability than a Poisson model would predict.

Hurdle models provide an alternative to zero-inflated models. A hurdle model separates the process into two parts: a binary model for whether Y = 0 or Y > 0, and a truncated count model for the positive values. The key difference from zero-inflated models is conceptual: hurdle models do not distinguish between structural and sampling zeros. All zeros are treated as arising from the same process. The hurdle model may use logistic regression for the zero/non-zero decision and a truncated Poisson or truncated negative binomial for positive counts.

The choice between zero-inflated and hurdle models depends on whether the distinction between structural and sampling zeros is meaningful for the application. In insurance, a hurdle model may be more natural: the first stage models whether a claim occurs (frequency), and the second stage models the number of claims given that at least one occurred.

Model Selection and Diagnostics

Selecting between a standard GLM, a zero-inflated model, and a hurdle model requires comparing their fit to the data. AIC and BIC provide quantitative comparison, with lower values indicating better fit after penalizing for model complexity. Vuong's test is a likelihood-ratio-based test specifically designed for comparing non-nested models like ZIP versus standard Poisson. A significant positive test statistic favors the zero-inflated model.

Diagnostic checks include comparing predicted and observed proportions of zeros, examining residual patterns, and checking whether the additional complexity of a zero-inflated or hurdle model is justified by the improvement in fit. A rootogram (a plot comparing observed frequencies to predicted frequencies, with bars hanging from the predicted values) provides a clear visual comparison of how well each model captures the distribution of counts, especially at zero and small values.

Common Pitfalls

A common mistake is using a zero-inflated model when the excess zeros arise from overdispersion rather than a separate zero-generating process. In such cases, a negative binomial GLM may be simpler and equally effective. Another pitfall is allowing both pi and lambda to depend on the same covariates in a zero-inflated model, which can create identifiability issues. Consider whether some covariates should only affect the zero-inflation probability while others only affect the count rate.

With Tweedie models, students sometimes confuse the power parameter p with the dispersion parameter phi. The power parameter determines the shape of the variance function and connects the Tweedie to specific distribution families. The dispersion parameter is a scaling factor that does not change the distributional family. Both affect the variance, but they play different roles in the model.