Maximum Likelihood Estimation (MLE) is a method used to estimate the parameters of a statistical model. The goal of MLE is to find the parameter values that maximize the likelihood function.
Likelihood Function
Suppose we have a set of observations \( \{x_1, x_2, \ldots, x_n\} \) that are independently and identically distributed (i.i.d) according to some probability distribution with parameter \( \theta \). The likelihood function \( L(\theta) \) is defined as the joint probability (or probability density) of the observed data as a function of \( \theta \):
$$ L(\theta) = P(x_1, x_2, \ldots, x_n \mid \theta) = \prod_{i=1}^n P(x_i \mid \theta) $$
Log-Likelihood Function
In practice, it is often easier to work with the natural logarithm of the likelihood function, known as the log-likelihood function:
$$ \ell(\theta) = \log L(\theta) = \log \left( \prod_{i=1}^n P(x_i \mid \theta) \right) = \sum_{i=1}^n \log P(x_i \mid \theta) $$
Maximum Likelihood Estimate
The maximum likelihood estimate (MLE) of the parameter \( \theta \) is the value that maximizes the log-likelihood function:
$$ \hat{\theta} = \arg \max_\theta \ell(\theta) $$
Example: Normal Distribution
Consider a set of i.i.d observations \( \{x_1, x_2, \ldots, x_n\} \) from a normal distribution with unknown mean \( \mu \) and known variance \( \sigma^2 \). The probability density function is:
$$ f(x \mid \mu) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \left( -\frac{(x – \mu)^2}{2\sigma^2} \right) $$
The log-likelihood function is:
$$ \ell(\mu) = \sum_{i=1}^n \log f(x_i \mid \mu) = -\frac{n}{2} \log (2 \pi \sigma^2) – \frac{1}{2\sigma^2} \sum_{i=1}^n (x_i – \mu)^2 $$
To find the MLE, we take the derivative of \( \ell(\mu) \) with respect to \( \mu \) and set it to zero:
$$ \frac{d\ell(\mu)}{d\mu} = \frac{1}{\sigma^2} \sum_{i=1}^n (x_i – \mu) = 0 $$
Solving for \( \mu \), we get the MLE:
$$ \hat{\mu} = \frac{1}{n} \sum_{i=1}^n x_i $$
Properties of MLE
Under certain regularity conditions, the MLE has several desirable properties:
Consistency:
The MLE converges in probability to the true parameter value as the sample size increases.
Asymptotic Normality:
The distribution of the MLE approaches a normal distribution as the sample size increases.
Efficiency:
The MLE achieves the Cramér-Rao lower bound, meaning it has the lowest possible variance among unbiased estimators.
