Overdispersion: Detection and Modeling Approaches

What Is Overdispersion

Overdispersion occurs when the observed variance of a response variable exceeds what the assumed statistical model predicts. For count data modeled with a Poisson distribution, the Poisson assumption implies Var(Y) = E[Y] = mu. Overdispersion means Var(Y) > mu. In insurance claim count data, overdispersion is the rule rather than the exception. Claim frequencies almost always show more variability than a Poisson model predicts.

The dispersion ratio phi = Var(Y) / E[Y] quantifies the degree of overdispersion. A Poisson model implies phi = 1. Values of phi between 2 and 10 are common in insurance data. For example, if the average claim count per policy is 0.15 but the variance is 0.45, the dispersion ratio is phi = 0.45 / 0.15 = 3, indicating substantial overdispersion.

Causes of Overdispersion

Unobserved heterogeneity is the most common cause. Even after controlling for observable risk factors (age, territory, vehicle type), policyholders differ in unobservable ways (driving skill, risk tolerance, exposure patterns). These unobserved differences create extra variability in claim counts beyond what the Poisson model allows. If each policyholder has their own true Poisson rate Lambda_i, but Lambda_i varies across policyholders, the unconditional distribution of claim counts has variance exceeding the mean.

Clustering occurs when events tend to happen together. A single accident might generate multiple claims (bodily injury, property damage, medical payments), creating correlated counts. Excess zeros arise when a large fraction of policyholders have zero claims, more than the Poisson distribution predicts. This can happen when some policyholders face near-zero risk (structural zeros) or when claims are underreported.

Detecting Overdispersion

The simplest diagnostic compares the Pearson chi-squared statistic or the deviance to the residual degrees of freedom. The Pearson statistic is X^2 = sum((y_i - mu_hat_i)^2 / mu_hat_i). Under a correctly specified Poisson model, X^2 / (n - p) should be approximately 1, where n is the number of observations and p is the number of estimated parameters. A ratio substantially greater than 1 indicates overdispersion.

For a concrete example, suppose you fit a Poisson GLM to 1,000 policies with 5 parameters and obtain a Pearson statistic of 2,850. The ratio is 2,850 / 995 = 2.86, strongly suggesting overdispersion. The Dean-Lawless test provides a formal hypothesis test: it tests H0: phi = 1 against H1: phi > 1 using a score test based on the Poisson regression residuals.

Consequences of Ignoring Overdispersion

If you fit a Poisson model to overdispersed data, the point estimates of regression coefficients remain consistent (unbiased for large samples). However, the standard errors are too small because the Poisson model underestimates the true variability. This leads to confidence intervals that are too narrow, p-values that are too small, and too many variables declared "statistically significant." In actuarial pricing, this means overfitting: the model appears to detect patterns that are really just noise.

For example, if the true dispersion is phi = 3, a Poisson model underestimates standard errors by a factor of sqrt(3) = 1.73. A coefficient with a z-statistic of 2.5 under the Poisson model would have a corrected z-statistic of only 2.5 / 1.73 = 1.44, which is not significant at the 5% level. Failing to account for overdispersion can lead to including irrelevant risk factors in a pricing model.

Quasi-Poisson Models

The quasi-Poisson model is the simplest fix for overdispersion. It introduces a dispersion parameter phi so that Var(Y) = phi * mu while keeping E[Y] = mu. The regression coefficients are estimated the same way as in a Poisson model (the point estimates are identical), but the standard errors are multiplied by sqrt(phi_hat), where phi_hat is estimated from the Pearson statistic. This approach adjusts inference without changing the model for the mean.

The quasi-Poisson model is not a true probability model (there is no distribution with Var(Y) = phi * mu for arbitrary phi), but it provides valid large-sample inference through the quasi-likelihood framework. It is useful when the primary goal is correct inference about regression coefficients rather than predicting the full distribution of counts.

Negative Binomial Models

The negative binomial model provides a proper probability distribution for overdispersed counts. It can be derived as a Poisson-Gamma mixture: if Y | Lambda ~ Poisson(Lambda) and Lambda ~ Gamma(k, k/mu), then the marginal distribution of Y is negative binomial with E[Y] = mu and Var(Y) = mu + mu^2/k. The parameter k controls the degree of overdispersion; smaller k means more overdispersion. As k approaches infinity, the negative binomial converges to the Poisson.

The negative binomial model is preferred over quasi-Poisson when you need a full distributional model (for example, to simulate data or compute tail probabilities). It naturally arises from the assumption of unobserved heterogeneity: if each policyholder's true Poisson rate is drawn from a Gamma distribution, the resulting claim count distribution is negative binomial.

Other Approaches

Mixed models with random effects address overdispersion by explicitly modeling the unobserved heterogeneity that causes it. A Poisson mixed model with a random intercept for each policyholder allows each policyholder's baseline claim rate to vary, capturing the heterogeneity that a fixed-effects Poisson model misses. Zero-inflated models specifically address excess zeros by mixing a point mass at zero with a count distribution. These models are covered in detail in separate articles.

Common Pitfalls

A frequent mistake is testing for overdispersion after fitting the wrong mean model. If important covariates are omitted, the residual variance will be inflated, mimicking overdispersion. Always ensure your mean model is reasonably well specified before concluding that overdispersion is present. Another error is applying a quasi-Poisson adjustment but then using the model for prediction as if it were a proper Poisson model. The quasi-Poisson model adjusts standard errors but does not change predicted probabilities; for distributional predictions, use a negative binomial.

Study Tips

Exam MAS-I tests your ability to detect overdispersion, understand its consequences, and choose appropriate remedies. Know the Pearson statistic diagnostic and the Dean-Lawless test. Be able to explain why overdispersion causes standard errors to be too small under a Poisson model. Understand the key difference between quasi-Poisson (adjusts standard errors only) and negative binomial (provides a full probability model). Know the Poisson-Gamma mixture derivation of the negative binomial and the variance formula Var(Y) = mu + mu^2/k. Practice interpreting model output from both Poisson and negative binomial regressions and comparing the standard errors.