Truncated and Censored Data in Loss Modeling

Why Insurance Data Is Modified

Insurance data is rarely a complete, unaltered sample from the underlying loss distribution. Policy features like deductibles and limits systematically modify the data that insurers observe. A deductible means losses below a threshold are not reported to the insurer, creating left truncation. A policy limit means losses above a cap are reported only as reaching the limit, creating right censoring. Understanding these modifications and their effect on statistical estimation is essential for accurate modeling and is heavily tested on Exam STAM.

The distinction between truncation and censoring is fundamental. Truncation removes observations entirely from the data set: you do not know how many losses fell below the deductible, and those losses leave no trace in the data. Censoring modifies observations but keeps them in the data set: you know a loss occurred and that it exceeded the policy limit, but you do not know the exact amount. Both modifications distort the observed data relative to the underlying loss distribution, and ignoring them biases parameter estimates.

Left Truncation from Deductibles

When a policy has a deductible d, losses below d are not reported. The insurer only observes losses that exceed d. This creates a left-truncated sample: the observed data consists of values drawn from the conditional distribution of X given X > d. The conditional density is f(x | X > d) = f(x) / S(d) for x > d, where f is the underlying (ground-up) density and S(d) = 1 - F(d) is the survival function at d.

For maximum likelihood estimation, each observed loss x_i > d contributes f(x_i) / S(d) to the likelihood. The log-likelihood is l(theta) = sum of [ln f(x_i; theta) - ln S(d; theta)]. The subtraction of ln S(d) is the crucial adjustment for truncation. Without it, the MLE would be based on the wrong conditional distribution and the parameter estimates would be biased.

For example, suppose losses follow an exponential distribution with mean theta and the deductible is d = 500. The conditional density for observed losses is f(x|X>500) = (1/theta) * exp(-x/theta) / exp(-500/theta) = (1/theta) * exp(-(x-500)/theta) for x > 500. Notice that the conditional distribution of the excess X - 500 is also exponential with the same mean theta, which is a consequence of the memoryless property. This special property does not hold for other distributions.

Right Censoring from Policy Limits

When a policy has a limit u, losses exceeding u are reported as u. The insurer knows a loss occurred and that the ground-up amount exceeds u, but not the actual value. This creates right-censored observations. For an uncensored observation x < u, the likelihood contribution is f(x). For a censored observation (recorded as u, meaning the actual loss exceeds u), the likelihood contribution is S(u) = P(X > u).

If there are n_1 uncensored observations and n_2 censored observations, the log-likelihood is l(theta) = sum over uncensored of ln f(x_i; theta) + n_2 * ln S(u; theta). The censored observations contribute through the survival function rather than the density. Ignoring censoring and treating all observations at u as exact values of u would underestimate the mean and produce incorrect parameter estimates.

Combined Truncation and Censoring

In practice, policies often have both a deductible d and a limit u, creating data that is simultaneously left-truncated and right-censored. The observed data consists of: (1) exact loss amounts x for losses where d < x < d + u (the insurer pays x - d), and (2) censored observations for losses where x >= d + u (the insurer pays the full limit u). Losses below d are unobserved.

For the ground-up loss distribution, the likelihood contribution for an uncensored observation x is f(x) / S(d), combining the density at x with the truncation adjustment. For a censored observation at d + u (meaning the ground-up loss exceeds d + u), the contribution is S(d + u) / S(d). The full log-likelihood is: l(theta) = sum over uncensored of [ln f(x_i) - ln S(d)] + n_c * [ln S(d+u) - ln S(d)], where n_c is the number of censored observations.

Some problems express losses as payment amounts (x - d for uncensored) rather than ground-up amounts. Be careful to translate back to the ground-up scale when constructing the likelihood, or equivalently, write the likelihood in terms of the per-payment random variable Y = X - d | X > d.

Impact on Naive Estimation

Ignoring data modifications leads to systematic biases. If you treat left-truncated data as a complete sample, the sample mean overstates the true population mean because all observations exceed the deductible. For an exponential distribution with mean theta = 1000 and deductible d = 500, the mean of the observed (truncated) data is theta + d = 1500, not theta = 1000. Using the sample mean as an estimate of theta would be badly biased.

If you treat right-censored observations as exact values at the limit, the sample mean understates the true mean because large losses are recorded at a value below their actual amount. Both biases can be severe, and they can partly offset each other when both truncation and censoring are present, but the net effect depends on the distribution and the values of d and u.

MLE with Modified Data

Maximum likelihood estimation handles truncation and censoring naturally through proper likelihood construction. The key is to write the correct likelihood contribution for each type of observation: f(x)/S(d) for truncated uncensored observations, S(u)/S(d) for truncated censored observations, and the standard contributions f(x) or S(u) when only one modification is present.

For the exponential distribution with truncation at d, the MLE has a simple closed form: theta_hat = (sum of x_i)/n - d = x_bar - d, where x_bar is the mean of the observed (truncated) data. For other distributions, the MLE typically requires numerical optimization of the log-likelihood. Setting derivatives of the log-likelihood equal to zero produces score equations that must be solved iteratively.

Common Pitfalls

A frequent error is confusing the deductible d in the ground-up loss scale with the payment scale. If the problem gives payment amounts (what the insurer pays), an observed payment of y corresponds to a ground-up loss of y + d. Make sure you know which scale the data is on before constructing the likelihood. Another common mistake is forgetting the S(d) denominator in the truncation adjustment, or applying it only to uncensored observations when it should appear for all observations (both censored and uncensored are conditional on exceeding d).

Students also sometimes confuse left truncation with left censoring. Left censoring occurs when you know an observation is below a threshold but not its exact value (rare in insurance). Left truncation means observations below the threshold are entirely absent from the data. The distinction matters because the likelihood contributions are different.

Study Tips

Exam STAM frequently presents data sets with deductibles and limits and asks you to write the correct likelihood, compute MLEs, or compare estimates with and without adjustment for data modifications. Practice writing likelihoods for each combination: truncation only, censoring only, and both together. Work through the exponential case (which has closed-form MLEs) to build intuition, then practice with Pareto and lognormal distributions where numerical methods are needed. Always start by identifying the data modification type before attempting any calculation.