Point Estimation: MLE, MOM, and Minimum Variance
Compare maximum likelihood, method of moments, and minimum variance estimation for Exam MAS-I.
Maximum Likelihood Estimation
Maximum likelihood estimation (MLE) finds the parameter values that maximize the probability of observing the data. For a sample x_1, x_2, ..., x_n from a distribution with PDF or PMF f(x; theta), the likelihood function is L(theta) = product of f(x_i; theta). In practice, we maximize the log-likelihood l(theta) = sum of ln(f(x_i; theta)) because the logarithm converts products to sums, simplifying calculus and avoiding numerical underflow.
The MLE is found by taking the derivative of the log-likelihood with respect to theta, setting it equal to zero, and solving. For multi-parameter models, this involves a system of equations (the score equations). For example, for a Normal(mu, sigma^2) sample, the MLEs are mu-hat = x-bar (the sample mean) and sigma^2-hat = (1/n) * sum(x_i - x-bar)^2 (the biased sample variance). Note that the MLE of sigma^2 divides by n, not n-1, so it is biased but consistent.
Properties of MLEs
MLEs possess several desirable large-sample properties. Consistency means theta-hat converges in probability to the true parameter value as n grows. Asymptotic normality means that sqrt(n) * (theta-hat - theta) converges in distribution to N(0, 1/I_1(theta)), where I_1(theta) is the Fisher information per observation. Asymptotic efficiency means that no consistent estimator has smaller asymptotic variance, so the MLE achieves the Cramer-Rao lower bound in large samples.
The invariance property states that if theta-hat is the MLE of theta, then g(theta-hat) is the MLE of g(theta) for any function g. This is extremely convenient: if you find the MLE of a rate parameter lambda, then 1/lambda-hat is automatically the MLE of the mean. No other estimation method has this general invariance property.
Fisher Information and the Cramer-Rao Bound
The Fisher information I(theta) = -E[d^2/dtheta^2 ln f(X; theta)] = E[(d/dtheta ln f(X; theta))^2] measures how much information a single observation carries about theta. The total information from n independent observations is n * I(theta). The Cramer-Rao lower bound states that for any unbiased estimator T of theta, Var(T) >= 1/(n * I(theta)).
If an unbiased estimator achieves the Cramer-Rao bound, it is called efficient or a minimum variance unbiased estimator (MVUE). Not all distributions admit an efficient estimator, but when one exists, it is unique. The exponential family of distributions always admits an efficient estimator for the natural parameter, which is why exponential family distributions are so important in statistical theory.
Method of Moments
The method of moments (MOM) equates population moments to sample moments and solves for parameters. For a distribution with k unknown parameters, equate the first k population moments E[X], E[X^2], ..., E[X^k] to their sample counterparts (1/n) * sum(x_i), (1/n) * sum(x_i^2), ..., (1/n) * sum(x_i^k). This produces k equations in k unknowns.
For example, for a Gamma(alpha, theta) distribution, E[X] = alpha*theta and E[X^2] = alpha*theta^2 + (alpha*theta)^2 = alpha*theta^2 * (alpha + 1). Setting E[X] = x-bar and E[X^2] = (1/n)*sum(x_i^2) gives two equations. Solving yields alpha-hat = (x-bar)^2 / (m_2 - x-bar^2) and theta-hat = (m_2 - x-bar^2) / x-bar, where m_2 = (1/n)*sum(x_i^2).
MOM estimators are simple to compute and always consistent (by the law of large numbers), but they are generally less efficient than MLEs. They may also produce estimates outside the parameter space (for example, a negative variance estimate), which is a practical disadvantage.
Sufficient Statistics and the Rao-Blackwell Theorem
A sufficient statistic T(X) captures all the information in the data about theta: the conditional distribution of the data given T does not depend on theta. By the factorization theorem, T is sufficient if and only if the likelihood can be written as L(theta) = g(T, theta) * h(x), where g depends on the data only through T and h does not depend on theta.
The Rao-Blackwell theorem states that if T is a sufficient statistic and U is any unbiased estimator of theta, then E[U|T] is at least as good as U (its variance is no larger). If T is also complete (no non-trivial function of T has zero expectation for all theta), then E[U|T] is the unique MVUE. The Lehmann-Scheffe theorem formalizes this: any unbiased estimator that is a function of a complete sufficient statistic is the MVUE.
Comparing Estimation Methods
For many standard distributions, MLE and MOM produce the same estimator. For the Exponential distribution, both methods yield theta-hat = x-bar. For the Poisson, both give lambda-hat = x-bar. The methods diverge for more complex distributions. The Normal variance MLE is (1/n)*sum(x_i - x-bar)^2 (biased), while the MOM using the second central moment gives the same result (also biased). The unbiased estimator s^2 = (1/(n-1))*sum(x_i - x-bar)^2 is neither MLE nor MOM but is the MVUE.
When choosing between methods, MLE is generally preferred for its efficiency and invariance properties, especially in complex models or with non-standard data (censored, truncated). MOM is useful as a quick starting point, for providing initial values for iterative MLE algorithms, or when the likelihood is difficult to optimize.
Common Pitfalls
A frequent mistake is forgetting that MLE may produce biased estimators. Consistency and asymptotic efficiency are large-sample properties; in small samples, bias can be significant. Another common error is computing Fisher information using the wrong formula or forgetting the negative sign in I(theta) = -E[d^2/dtheta^2 ln f(X; theta)]. Also, when computing the Cramer-Rao bound for a sample of size n, remember to multiply the single-observation Fisher information by n.
Study Tips
Exam MAS-I tests your ability to derive MLEs and MOM estimators for common distributions, compute Fisher information, apply the Cramer-Rao bound, and use the Rao-Blackwell theorem. Practice deriving estimators from scratch for Poisson, Exponential, Normal, Gamma, and Beta distributions. Know the properties of each estimation method and be prepared to compare them. Understand sufficient statistics and the factorization theorem, as these concepts connect to both estimation quality and the structure of exponential family distributions.