Maximum likelihood estimation

(Redirected from Maximum likelihood)

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters by maximizing a likelihood function, so that under the assumed statistical model the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate.[1] The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.[2][3][4]

If the likelihood function is differentiable, the derivative test for determining maxima can be applied. In some cases, the first-order conditions of the likelihood function can be solved explicitly; for instance, the ordinary least squares estimator maximizes the likelihood of the linear regression model.[5] Under most circumstances, however, numerical methods will be necessary to find the maximum of the likelihood function.

From the point of view of Bayesian inference, MLE is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters. In frequentist inference, MLE is a special case of an extremum estimator, with the objective function being the likelihood.

Principles

From a statistical standpoint, the observations ${\displaystyle \mathbf {y} =(y_{1},y_{2},\ldots ,y_{T})}$ are a random sample from an unknown population. The goal of statistical analysis is to make inference about the population that is most likely to have generated the sample, specifically the probability distribution corresponding to the population.[6] Associated with each probability distribution is a unique vector ${\displaystyle \theta =\left[\theta _{1},\,\theta _{2},\ldots \,,\theta _{k}\right]^{\mathsf {T}}}$ of parameters that index the probability distribution within a parametric family ${\displaystyle \{f(\cdot \,;\theta )\mid \theta \in \Theta \}}$. As ${\displaystyle \theta }$ changes in value, different probability distributions are generated. The idea of maximum likelihood is to re-express the joint probability of the sample data ${\displaystyle f(y_{1},y_{2},\ldots ,y_{T};\theta )}$ as a likelihood function ${\displaystyle {\mathcal {L}}(\theta \,;\mathbf {y} )}$ that treats ${\displaystyle \theta }$ as a variable. For independent and identically distributed random variables, the likelihood function is defined as

${\displaystyle L(\theta \,;\mathbf {y} )=f(y_{1};\theta )f(y_{2};\theta )\dots f(y_{T};\theta )=\prod _{t=1}^{T}f(y_{t};\theta )}$

and evaluated at the observed data sample. The goal is then to find the values of the model parameter that maximize the likelihood function over the parameter space ${\displaystyle \Theta }$. Intuitively, this selects the parameter values that make the observed data most probable. The problem is thus to find the supremum value of the likelihood function by choice of the parameter

${\displaystyle {\mathcal {L}}({\widehat {\theta \,}}\,;\mathbf {y} )=\sup _{\theta \in \Theta }L(\theta \,;\mathbf {y} )}$

where the estimator ${\displaystyle {\widehat {\theta \,}}={\widehat {\theta \,}}(\mathbf {y} )}$ is function of the sample. A sufficient but not necessary condition for its existence is for the likelihood function to be continuous over a parameter space ${\displaystyle \Theta }$ that is compact.[7] For an open ${\displaystyle \Theta }$ the likelihood function may be unbounded (increasing without ever reaching a supremum value).

In practice, it is often convenient to work with the natural logarithm of the likelihood function, called the log-likelihood:

${\displaystyle \ell (\theta \,;\mathbf {y} )=\ln {\mathcal {L}}(\theta \,;\mathbf {y} ),}$

Since the logarithm is a monotonic function, the maximum of ${\displaystyle \ell (\theta \,;\mathbf {y} )}$ occurs at the same value of ${\displaystyle \theta }$ as does the maximum of ${\displaystyle {\mathcal {L}}}$.[8] If ${\displaystyle \ell (\theta \,;\mathbf {y} )}$ is differentiable in ${\displaystyle \theta }$, the necessary conditions for the occurrence of a maximum (or a minimum) are

${\displaystyle {\frac {\partial \ell }{\partial \theta _{1}}}=0,\quad {\frac {\partial \ell }{\partial \theta _{2}}}=0,\quad \ldots ,\quad {\frac {\partial \ell }{\partial \theta _{k}}}=0}$

known as the likelihood equations. For some models, these equations can be explicitly solved for ${\displaystyle {\widehat {\theta \,}}}$, but in general no closed-form solution to the maximization problem is known or available, and an MLE can only be found via numerical optimization. Another problem is that in finite samples, there may exist multiple roots for the likelihood equations.[9] Whether the identified root ${\displaystyle {\widehat {\theta \,}}}$ of the likelihood equations is indeed a (local) maximum depends on whether the matrix of second-order partial and cross-partial derivatives,

${\displaystyle \mathbf {H} ({\widehat {\theta \,}})={\begin{bmatrix}\left.{\frac {\partial ^{2}\ell }{\partial \theta _{1}^{2}}}\right|_{\theta ={\widehat {\theta \,}}}&\left.{\frac {\partial ^{2}\ell }{\partial \theta _{1}\partial \theta _{2}}}\right|_{\theta ={\widehat {\theta \,}}}&\dots &\left.{\frac {\partial ^{2}\ell }{\partial \theta _{1}\partial \theta _{k}}}\right|_{\theta ={\widehat {\theta \,}}}\\\left.{\frac {\partial ^{2}\ell }{\partial \theta _{2}\partial \theta _{1}}}\right|_{\theta ={\widehat {\theta \,}}}&\left.{\frac {\partial ^{2}\ell }{\partial \theta _{2}^{2}}}\right|_{\theta ={\widehat {\theta \,}}}&\dots &\left.{\frac {\partial ^{2}\ell }{\partial \theta _{2}\partial \theta _{k}}}\right|_{\theta ={\widehat {\theta \,}}}\\\vdots &\vdots &\ddots &\vdots \\\left.{\frac {\partial ^{2}\ell }{\partial \theta _{k}\partial \theta _{1}}}\right|_{\theta ={\widehat {\theta \,}}}&\left.{\frac {\partial ^{2}\ell }{\partial \theta _{k}\partial \theta _{2}}}\right|_{\theta ={\widehat {\theta \,}}}&\dots &\left.{\frac {\partial ^{2}\ell }{\partial \theta _{k}^{2}}}\right|_{\theta ={\widehat {\theta \,}}}\end{bmatrix}}}$

known as the Hessian matrix is negative semi-definite at ${\displaystyle {\widehat {\theta \,}}}$, which indicates local concavity. Conveniently, most common probability distributions—in particular the exponential family—are logarithmically concave.[10][11]

Properties

A maximum likelihood estimator is an extremum estimator obtained by maximizing, as a function of θ, the objective function ${\displaystyle {\widehat {\ell \,}}(\theta \,;x)}$. If the data are independent and identically distributed, then we have

${\displaystyle {\widehat {\ell \,}}(\theta \,;x)={\frac {1}{n}}\sum _{i=1}^{n}\ln f(x_{i}\mid \theta ),}$

this being the sample analogue of the expected log-likelihood ${\displaystyle \ell (\theta )=\operatorname {E} [\,\ln f(x_{i}\mid \theta )\,]}$, where this expectation is taken with respect to the true density.

Maximum-likelihood estimators have no optimum properties for finite samples, in the sense that (when evaluated on finite samples) other estimators may have greater concentration around the true parameter-value.[12] However, like other estimation methods, maximum likelihood estimation possesses a number of attractive limiting properties: As the sample size increases to infinity, sequences of maximum likelihood estimators have these properties:

• Consistency: the sequence of MLEs converges in probability to the value being estimated.
• Efficiency, i.e. it achieves the Cramér–Rao lower bound when the sample size tends to infinity. This means that no consistent estimator has lower asymptotic mean squared error than the MLE (or other estimators attaining this bound).
• Second-order efficiency after correction for bias.

Consistency

Under the conditions outlined below, the maximum likelihood estimator is consistent. The consistency means that if the data were generated by ${\displaystyle f(\cdot \,;\theta _{0})}$ and we have a sufficiently large number of observations n, then it is possible to find the value of θ0 with arbitrary precision. In mathematical terms this means that as n goes to infinity the estimator ${\displaystyle {\widehat {\theta \,}}}$ converges in probability to its true value:

${\displaystyle {\widehat {\theta \,}}_{\mathrm {mle} }\ {\xrightarrow {\text{p}}}\ \theta _{0}.}$

Under slightly stronger conditions, the estimator converges almost surely (or strongly):

${\displaystyle {\widehat {\theta \,}}_{\mathrm {mle} }\ {\xrightarrow {\text{a.s.}}}\ \theta _{0}.}$

Note that, in practical applications, data is never generated by ${\displaystyle f(\cdot \,;\theta _{0})}$. Rather, ${\displaystyle f(\cdot \,;\theta _{0})}$ is a model, often in idealized form, of the process that generated the data. It is a common aphorism in statistics that all models are wrong. Thus, true consistency does not occur in practical applications. Nevertheless, consistency is often considered to be a desirable property for an estimator to have.

To establish consistency, the following conditions are sufficient.[13]

1. Identification of the model:
${\displaystyle \theta \neq \theta _{0}\quad \Leftrightarrow \quad f(\cdot \mid \theta )\neq f(\cdot \mid \theta _{0}).}$

In other words, different parameter values θ correspond to different distributions within the model. If this condition did not hold, there would be some value θ1 such that θ0 and θ1 generate an identical distribution of the observable data. Then we would not be able to distinguish between these two parameters even with an infinite amount of data—these parameters would have been observationally equivalent.

The identification condition is absolutely necessary for the ML estimator to be consistent. When this condition holds, the limiting likelihood function (θ|·) has unique global maximum at θ0.
2. Compactness: the parameter space Θ of the model is compact.

The identification condition establishes that the log-likelihood has a unique global maximum. Compactness implies that the likelihood cannot approach the maximum value arbitrarily close at some other point (as demonstrated for example in the picture on the right).

Compactness is only a sufficient condition and not a necessary condition. Compactness can be replaced by some other conditions, such as:

• both concavity of the log-likelihood function and compactness of some (nonempty) upper level sets of the log-likelihood function, or
• existence of a compact neighborhood N of θ0 such that outside of N the log-likelihood function is less than the maximum by at least some ε > 0.
3. Continuity: the function ln f(x|θ) is continuous in θ for almost all values of x:
${\displaystyle \operatorname {P} \!{\big [}\;\ln f(x\mid \theta )\;\in \;C^{0}(\Theta )\;{\big ]}=1.}$
The continuity here can be replaced with a slightly weaker condition of upper semi-continuity.
4. Dominance: there exists D(x) integrable with respect to the distribution f(x|θ0) such that
${\displaystyle {\big |}\ln f(x\mid \theta ){\big |}
By the uniform law of large numbers, the dominance condition together with continuity establish the uniform convergence in probability of the log-likelihood:
${\displaystyle \sup _{\theta \in \Theta }\left|{\widehat {\ell \,}}(\theta \mid x)-\ell (\theta )\,\right|\ {\xrightarrow {\text{p}}}\ 0.}$

The dominance condition can be employed in the case of i.i.d. observations. In the non-i.i.d. case, the uniform convergence in probability can be checked by showing that the sequence ${\displaystyle {\widehat {\ell \,}}(\theta \mid x)}$ is stochastically equicontinuous. If one wants to demonstrate that the ML estimator ${\displaystyle {\widehat {\theta \,}}}$ converges to θ0 almost surely, then a stronger condition of uniform convergence almost surely has to be imposed:

${\displaystyle \sup _{\theta \in \Theta }{\big \|}\;{\widehat {\ell \,}}(x\mid \theta )-\ell (\theta )\;{\big \|}\ {\xrightarrow {\text{a.s.}}}\ 0.}$

Additionally, if (as assumed above) the data were generated by ${\displaystyle f(\cdot \,;\theta _{0})}$, then under certain conditions, it can also be shown that the maximum likelihood estimator converges in distribution to a normal distribution. Specifically,[14]

${\displaystyle {\sqrt {n}}\left({\widehat {\theta \,}}_{\mathrm {mle} }-\theta _{0}\right)\ {\xrightarrow {d}}\ {\mathcal {N}}(0,\,I^{-1})}$

where I is the Fisher information matrix.

Functional invariance

The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, if ${\displaystyle {\widehat {\theta \,}}}$ is the MLE for ${\displaystyle \theta }$, and if ${\displaystyle g(\theta )}$ is any transformation of ${\displaystyle \theta }$, then the MLE for ${\displaystyle \alpha =g(\theta )}$ is by definition[15]

${\displaystyle {\widehat {\alpha }}=g(\,{\widehat {\theta \,}}\,).\,}$

It maximizes the so-called profile likelihood:

${\displaystyle {\bar {L}}(\alpha )=\sup _{\theta :\alpha =g(\theta )}L(\theta ).\,}$

The MLE is also invariant with respect to certain transformations of the data. If ${\displaystyle y=g(x)}$ where ${\displaystyle g}$ is one to one and does not depend on the parameters to be estimated, then the density functions satisfy

${\displaystyle f_{Y}(y)={\frac {f_{X}(x)}{|g'(x)|}}}$

and hence the likelihood functions for ${\displaystyle X}$ and ${\displaystyle Y}$ differ only by a factor that does not depend on the model parameters.

For example, the MLE parameters of the log-normal distribution are the same as those of the normal distribution fitted to the logarithm of the data.

Higher-order properties

As noted above, the maximum likelihood estimator is n-consistent and asymptotically efficient, meaning that it reaches the Cramér–Rao bound:

${\displaystyle {\sqrt {n}}({\widehat {\theta \,}}_{\text{mle}}-\theta _{0})\ \ {\xrightarrow {d}}\ \ {\mathcal {N}}(0,\ I^{-1}),}$

where ${\displaystyle I}$ is the Fisher information matrix:

${\displaystyle I_{jk}=\operatorname {E} {\bigg [}\;{-{\frac {\partial ^{2}\ln f_{\theta _{0}}(X_{t})}{\partial \theta _{j}\,\partial \theta _{k}}}}\;{\bigg ]}.}$

In particular, it means that the bias of the maximum likelihood estimator is equal to zero up to the order ​1n . However, when we consider the higher-order terms in the expansion of the distribution of this estimator, it turns out that θmle has bias of order ​1n. This bias is equal to (componentwise)[16]

${\displaystyle b_{h}\equiv \operatorname {E} {\bigg [}\;({\widehat {\theta }}_{\mathrm {mle} }-\theta _{0})_{h}\;{\bigg ]}={\frac {1}{n}}\sum _{i,j,k=1}^{m}I^{hi}I^{jk}{\big (}{\frac {1}{2}}K_{ijk}+J_{j,ik}{\big )}}$

where ${\displaystyle I^{jk}}$ denotes the (j,k)-th component of the inverse Fisher information matrix ${\displaystyle I^{-1}}$, and

${\displaystyle {\tfrac {1}{2}}K_{ijk}+J_{j,ik}=\operatorname {E} {\bigg [}\;{\frac {1}{2}}{\frac {\partial ^{3}\ln f_{\theta _{0}}(X_{t})}{\partial \theta _{i}\,\partial \theta _{j}\,\partial \theta _{k}}}+{\frac {\partial \ln f_{\theta _{0}}(X_{t})}{\partial \theta _{j}}}{\frac {\partial ^{2}\ln f_{\theta _{0}}(X_{t})}{\partial \theta _{i}\,\partial \theta _{k}}}\;{\bigg ]}.}$

Using these formulae it is possible to estimate the second-order bias of the maximum likelihood estimator, and correct for that bias by subtracting it:

${\displaystyle {\widehat {\theta \,}}_{\text{mle}}^{*}={\widehat {\theta \,}}_{\text{mle}}-{\widehat {b\,}}.}$

This estimator is unbiased up to the terms of order ​1n, and is called the bias-corrected maximum likelihood estimator.

This bias-corrected estimator is second-order efficient (at least within the curved exponential family), meaning that it has minimal mean squared error among all second-order bias-corrected estimators, up to the terms of the order ​1n2. It is possible to continue this process, that is to derive the third-order bias-correction term, and so on. However the maximum likelihood estimator is not third-order efficient.[17]

Relation to Bayesian inference

A maximum likelihood estimator coincides with the most probable Bayesian estimator given a uniform prior distribution on the parameters. Indeed, the maximum a posteriori estimate is the parameter θ that maximizes the probability of θ given the data, given by Bayes' theorem:

${\displaystyle \operatorname {P} (\theta \mid x_{1},x_{2},\ldots ,x_{n})={\frac {f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )\operatorname {P} (\theta )}{\operatorname {P} (x_{1},x_{2},\ldots ,x_{n})}}}$

where ${\displaystyle P(\theta )}$ is the prior distribution for the parameter θ and where ${\displaystyle \operatorname {P} (x_{1},x_{2},\ldots ,x_{n})}$ is the probability of the data averaged over all parameters. Since the denominator is independent of θ, the Bayesian estimator is obtained by maximizing ${\displaystyle f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )\operatorname {P} (\theta )}$ with respect to θ. If we further assume that the prior ${\displaystyle P(\theta )}$ is a uniform distribution, the Bayesian estimator is obtained by maximizing the likelihood function ${\displaystyle f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )}$. Thus the Bayesian estimator coincides with the maximum likelihood estimator for a uniform prior distribution ${\displaystyle \operatorname {P} (\theta )}$.

Examples

Discrete uniform distribution

Consider a case where n tickets numbered from 1 to n are placed in a box and one is selected at random (see uniform distribution); thus, the sample size is 1. If n is unknown, then the maximum likelihood estimator ${\displaystyle {\widehat {n}}}$ of n is the number m on the drawn ticket. (The likelihood is 0 for n < m, ​1n for n ≥ m, and this is greatest when n = m. Note that the maximum likelihood estimate of n occurs at the lower extreme of possible values {mm + 1, ...}, rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) The expected value of the number m on the drawn ticket, and therefore the expected value of ${\displaystyle {\widehat {n}}}$, is (n + 1)/2. As a result, with a sample size of 1, the maximum likelihood estimator for n will systematically underestimate n by (n − 1)/2.

Discrete distribution, finite parameter space

Suppose one wishes to determine just how biased an unfair coin is. Call the probability of tossing a ‘headp. The goal then becomes to determine p.

Suppose the coin is tossed 80 times: i.e. the sample might be something like x1 = H, x2 = T, ..., x80 = T, and the count of the number of heads "H" is observed.

The probability of tossing tails is 1 − p (so here p is θ above). Suppose the outcome is 49 heads and 31 tails, and suppose the coin was taken from a box containing three coins: one which gives heads with probability p = ​13, one which gives heads with probability p = ​12 and another which gives heads with probability p = ​23. The coins have lost their labels, so which one it was is unknown. Using maximum likelihood estimation the coin that has the largest likelihood can be found, given the data that were observed. By using the probability mass function of the binomial distribution with sample size equal to 80, number successes equal to 49 but for different values of p (the "probability of success"), the likelihood function (defined below) takes one of three values:

{\displaystyle {\begin{aligned}\operatorname {P} {\big [}\;\mathrm {H} =49\mid p={\tfrac {1}{3}}\;{\big ]}&={\binom {80}{49}}({\tfrac {1}{3}})^{49}(1-{\tfrac {1}{3}})^{31}\approx 0.000,\\[6pt]\operatorname {P} {\big [}\;\mathrm {H} =49\mid p={\tfrac {1}{2}}\;{\big ]}&={\binom {80}{49}}({\tfrac {1}{2}})^{49}(1-{\tfrac {1}{2}})^{31}\approx 0.012,\\[6pt]\operatorname {P} {\big [}\;\mathrm {H} =49\mid p={\tfrac {2}{3}}\;{\big ]}&={\binom {80}{49}}({\tfrac {2}{3}})^{49}(1-{\tfrac {2}{3}})^{31}\approx 0.054.\end{aligned}}}

The likelihood is maximized when p = ​23, and so this is the maximum likelihood estimate for p.

Discrete distribution, continuous parameter space

Now suppose that there was only one coin but its p could have been any value 0 ≤ p ≤ 1. The likelihood function to be maximised is

${\displaystyle L(p)=f_{D}(\mathrm {H} =49\mid p)={\binom {80}{49}}p^{49}(1-p)^{31},}$

and the maximisation is over all possible values 0 ≤ p ≤ 1.

likelihood function for proportion value of a binomial process (n = 10)

One way to maximize this function is by differentiating with respect to p and setting to zero:

{\displaystyle {\begin{aligned}{0}&{}={\frac {\partial }{\partial p}}\left({\binom {80}{49}}p^{49}(1-p)^{31}\right),\\[8pt]{0}&{}=49p^{48}(1-p)^{31}-31p^{49}(1-p)^{30}\\[8pt]&{}=p^{48}(1-p)^{30}\left[49(1-p)-31p\right]\\[8pt]&{}=p^{48}(1-p)^{30}\left[49-80p\right]\end{aligned}}}

which has solutions p = 0, p = 1, and p = ​4980. The solution which maximizes the likelihood is clearly p = ​4980 (since p = 0 and p = 1 result in a likelihood of zero). Thus the maximum likelihood estimator for p is ​4980.

This result is easily generalized by substituting a letter such as s in the place of 49 to represent the observed number of 'successes' of our Bernoulli trials, and a letter such as n in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields ​sn which is the maximum likelihood estimator for any sequence of n Bernoulli trials resulting in s 'successes'.

Continuous distribution, continuous parameter space

For the normal distribution ${\displaystyle {\mathcal {N}}(\mu ,\sigma ^{2})}$ which has probability density function

${\displaystyle f(x\mid \mu ,\sigma ^{2})={\frac {1}{{\sqrt {2\pi \sigma ^{2}}}\ }}\exp {\left(-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}\right)},}$

the corresponding probability density function for a sample of n independent identically distributed normal random variables (the likelihood) is

${\displaystyle f(x_{1},\ldots ,x_{n}\mid \mu ,\sigma ^{2})=\prod _{i=1}^{n}f(x_{i}\mid \mu ,\sigma ^{2})=\left({\frac {1}{2\pi \sigma ^{2}}}\right)^{n/2}\exp \left(-{\frac {\sum _{i=1}^{n}(x_{i}-\mu )^{2}}{2\sigma ^{2}}}\right),}$.

This family of distributions has two parameters: θ = (μσ); so we maximize the likelihood, ${\displaystyle {\mathcal {L}}(\mu ,\sigma )=f(x_{1},\ldots ,x_{n}\mid \mu ,\sigma )}$, over both parameters simultaneously, or if possible, individually.

Since the logarithm function itself is a continuous strictly increasing function over the range of the likelihood, the values which maximize the likelihood will also maximize its logarithm (the log-likelihood itself is not necessarily strictly increasing). The log-likelihood can be written as follows:

${\displaystyle \log {\Big (}{\mathcal {L}}(\mu ,\sigma ){\Big )}=-{\frac {\,n\,}{2}}\log(2\pi \sigma ^{2})-{\frac {1}{2\sigma ^{2}}}\sum _{i=1}^{n}(\,x_{i}-\mu \,)^{2}}$

(Note: the log-likelihood is closely related to information entropy and Fisher information.)

We now compute the derivatives of this log-likelihood as follows.

{\displaystyle {\begin{aligned}0&={\frac {\partial }{\partial \mu }}\log {\Big (}{\mathcal {L}}(\mu ,\sigma ){\Big )}=0-{\frac {\;-2\!n({\bar {x}}-\mu )\;}{2\sigma ^{2}}}.\end{aligned}}}

where ${\displaystyle {\bar {x}}}$ is the sample mean. This is solved by

${\displaystyle {\widehat {\mu }}={\bar {x}}=\sum _{i=1}^{n}{\frac {\,x_{i}\,}{n}}.}$

This is indeed the maximum of the function, since it is the only turning point in μ and the second derivative is strictly less than zero. Its expected value is equal to the parameter μ of the given distribution,

${\displaystyle \operatorname {E} {\big [}\;{\widehat {\mu }}\;{\big ]}=\mu ,\,}$

which means that the maximum likelihood estimator ${\displaystyle {\widehat {\mu }}}$ is unbiased.

Similarly we differentiate the log-likelihood with respect to σ and equate to zero:

{\displaystyle {\begin{aligned}0&={\frac {\partial }{\partial \sigma }}\log {\Big (}{\mathcal {L}}(\mu ,\sigma ){\Big )}=-{\frac {\,n\,}{\sigma }}+{\frac {1}{\sigma ^{3}}}\sum _{i=1}^{n}(\,x_{i}-\mu \,)^{2}.\end{aligned}}}

which is solved by

${\displaystyle {\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-\mu )^{2}.}$

Inserting the estimate ${\displaystyle \mu ={\widehat {\mu }}}$ we obtain

${\displaystyle {\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}={\frac {1}{n}}\sum _{i=1}^{n}x_{i}^{2}-{\frac {1}{n^{2}}}\sum _{i=1}^{n}\sum _{j=1}^{n}x_{i}x_{j}.}$

To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables (statistical error) ${\displaystyle \delta _{i}\equiv \mu -x_{i}}$. Expressing the estimate in these variables yields

${\displaystyle {\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}(\mu -\delta _{i})^{2}-{\frac {1}{n^{2}}}\sum _{i=1}^{n}\sum _{j=1}^{n}(\mu -\delta _{i})(\mu -\delta _{j}).}$

Simplifying the expression above, utilizing the facts that ${\displaystyle \operatorname {E} {\big [}\;\delta _{i}\;{\big ]}=0}$ and ${\displaystyle \operatorname {E} {\big [}\;\delta _{i}^{2}\;{\big ]}=\sigma ^{2}}$, allows us to obtain

${\displaystyle \operatorname {E} {\big [}\;{\widehat {\sigma }}^{2}\;{\big ]}={\frac {\,n-1\,}{n}}\sigma ^{2}.}$

This means that the estimator ${\displaystyle {\widehat {\sigma }}}$ is biased. However, ${\displaystyle {\widehat {\sigma }}}$ is consistent.

Formally we say that the maximum likelihood estimator for ${\displaystyle \theta =(\mu ,\sigma ^{2})}$ is

${\displaystyle {\widehat {\theta \,}}=\left({\widehat {\mu }},{\widehat {\sigma }}^{2}\right).}$

In this case the MLEs could be obtained individually. In general this may not be the case, and the MLEs would have to be obtained simultaneously.

The normal log-likelihood at its maximum takes a particularly simple form:

${\displaystyle \log {\Big (}{\mathcal {L}}({\widehat {\mu }},{\widehat {\sigma }}){\Big )}={\frac {\,-n\;\;}{2}}{\big (}\,\log(2\pi {\widehat {\sigma }}^{2})+1\,{\big )}}$

This maximum log-likelihood can be shown to be the same for more general least squares, even for non-linear least squares. This is often used in determining likelihood-based approximate confidence intervals and confidence regions, which are generally more accurate than those using the asymptotic normality discussed above.

Non-independent variables

It may be the case that variables are correlated, that is, not independent. Two random variables ${\displaystyle y_{1}}$ and ${\displaystyle y_{2}}$ are independent only if their joint probability density function is the product of the individual probability density functions, i.e.

${\displaystyle f(y_{1},y_{2})=f(y_{1})f(y_{2})\,}$

Suppose one constructs an order-d Gaussian vector out of random variables ${\displaystyle (y_{1},\ldots ,y_{d})}$, where each variable has means given by ${\displaystyle (\mu _{1},\ldots ,\mu _{d})}$. Furthermore, let the covariance matrix be denoted by ${\displaystyle {\mathit {\Sigma }}}$. The joint probability density function of these d random variables is then follows a multivariate normal distribution given by:

${\displaystyle f(y_{1},\ldots ,y_{d})={\frac {1}{(2\pi )^{n/2}{\sqrt {{\text{det}}({\mathit {\Sigma }})}}}}\exp \left(-{\frac {1}{2}}\left[y_{1}-\mu _{1},\ldots ,y_{d}-\mu _{d}\right]{\mathit {\Sigma }}^{-1}\left[x_{1}-\mu _{1},\ldots ,y_{d}-\mu _{d}\right]^{\mathrm {T} }\right)}$

In the bivariate case, the joint probability density function is given by:

${\displaystyle f(y_{1},y_{2})={\frac {1}{2\pi \sigma _{1}\sigma _{2}{\sqrt {1-\rho ^{2}}}}}\exp \left[-{\frac {1}{2(1-\rho ^{2})}}\left({\frac {(y_{1}-\mu _{1})^{2}}{\sigma _{1}^{2}}}-{\frac {2\rho (y_{1}-\mu _{1})(y_{2}-\mu _{2})}{\sigma _{1}\sigma _{2}}}+{\frac {(y_{2}-\mu _{2})^{2}}{\sigma _{2}^{2}}}\right)\right]}$

In this and other cases where a joint density function exists, the likelihood function is defined as above, in the section "principles," using this density.

Iterative procedures

Except for special cases, the likelihood equations

${\displaystyle {\frac {\partial \ell (\theta ;\mathbf {y} )}{\partial \theta }}=0}$

cannot be solved for explicitly for an estimator ${\displaystyle {\widehat {\theta }}={\widehat {\theta }}(\mathbf {y} )}$. Instead, they need to be solved iteratively: starting from an initial guess of ${\displaystyle \theta }$ (say ${\displaystyle {\widehat {\theta }}_{1}}$), one seeks to obtain a convergent sequence ${\displaystyle \left\{{\widehat {\theta }}_{r}\right\}}$. Many methods for this kind of optimization problem are available,[18][19] but the most commonly used ones are hill climbing algorithms based on an updating formula of the form

${\displaystyle {\widehat {\theta }}_{r+1}={\widehat {\theta }}_{r}+\lambda _{r}\mathbf {d} _{r}({\widehat {\theta }})}$

where the vector ${\displaystyle \mathbf {d} _{r}({\widehat {\theta }})}$ indicates the direction of the rth "step," and the scalar ${\displaystyle \lambda _{r}}$ captures the "step length."[20][21] In the classic Newton–Raphson method,

${\displaystyle \lambda _{r}=1}$ and ${\displaystyle \mathbf {d} _{r}({\widehat {\theta }})=-\mathbf {H} _{r}^{-1}({\widehat {\theta }})\mathbf {s} _{r}({\widehat {\theta }})}$

where ${\displaystyle \mathbf {s} _{r}({\widehat {\theta }})}$ is the score and ${\displaystyle \mathbf {H} _{r}^{-1}({\widehat {\theta }})}$ is the inverse of the Hessian matrix of the log-likelihood function, both evaluated the rth iteration.[22][23] But because the calculation of the Hessian matrix is computationally costly, numerous alternatives have been proposed. The popular Berndt–Hall–Hall–Hausman algorithm approximates the Hessian with the outer product of the expected gradient, such that

${\displaystyle \mathbf {d} _{r}({\widehat {\theta }})=-\left[{\frac {1}{T}}\sum _{t=1}^{T}{\frac {\partial \ell (\theta ;\mathbf {y} )}{\partial \theta }}\left({\frac {\partial \ell (\theta ;\mathbf {y} )}{\partial \theta }}\right)^{\mathsf {T}}\right]^{-1}\mathbf {s} _{r}({\widehat {\theta }})}$

Other quasi-Newton methods use more elaborate secant updates, such as the Davidon–Fletcher–Powell formula or the Broyden–Fletcher–Goldfarb–Shanno algorithm. It is important to note, though, that these methods may converge to a stationary point that is not necessarily a local or global maximum,[24] but rather a local minimum or a saddle point. In practice, it is therefore important to assess the validity of the obtained solution to the likelihood equations, by verifying that the Hessian, evaluated at the solution, is both negative definite and well-conditioned.[25]

History

Ronald Fisher in 1913

Early users of maximum likelihood were Carl Friedrich Gauss, Pierre-Simon Laplace, Thorvald N. Thiele, and Francis Ysidro Edgeworth.[26][27] However, its widespread use rose between 1912 and 1922 when Ronald Fisher recommended, widely popularized, and carefully analyzed maximum-likelihood estimation (with fruitless attempts at proofs).[28]

Maximum-likelihood estimation finally transcended heuristic justification in a proof published by Samuel S. Wilks in 1938, now called Wilks' theorem.[29] The theorem shows that the error in the logarithm of likelihood values for estimates from multiple independent observations is asymptotically χ 2-distributed, which enables convenient determination of a confidence region around any estimate of the parameters. The only difficult part of Wilks’ proof depends on the expected value of the Fisher information matrix, which is provided by a theorem proven by Fisher.[30] Wilks continued to improve on the generality of the theorem throughout his life, with his most general proof published in 1962.[31]

Reviews of the development of maximum likelihood estimation have been provided by a number of authors.[32][33][34][35][36][37][38][39]

Related concepts

• Akaike information criterion, a criterion to compare statistical models, based on MLE
• Extremum estimator, a more general class of estimators to which MLE belongs
• Fisher information, information matrix, its relationship to covariance matrix of ML estimates
• Mean squared error, a measure of how 'good' an estimator of a distributional parameter is (be it the maximum likelihood estimator or some other estimator)
• RANSAC, a method to estimate parameters of a mathematical model given data that contains outliers
• Rao–Blackwell theorem, which yields a process for finding the best possible unbiased estimator (in the sense of having minimal mean squared error); the MLE is often a good starting place for the process
• Wilks’ theorem provides a means of estimating the size and shape of the region of roughly equally-probable estimates for the population's parameter values, using the information from a single sample, using a chi-squared distribution

References

1. ^ Rossi, Richard J. (2018). Mathematical Statistics : An Introduction to Likelihood Based Inference. New York: John Wiley & Sons. p. 227. ISBN 978-1-118-77104-4.
2. ^ Hendry, David F.; Nielsen, Bent (2007). Econometric Modeling: A Likelihood Approach. Princeton: Princeton University Press. ISBN 978-0-691-13128-3.
3. ^ Chambers, Raymond L.; Steel, David G.; Wang, Suojin; Welsh, Alan (2012). Maximum Likelihood Estimation for Sample Surveys. Boca Raton: CRC Press. ISBN 978-1-58488-632-7.
4. ^ Ward, Michael Don; Ahlquist, John S. (2018). Maximum Likelihood for Social Science : Strategies for Analysis. New York: Cambridge University Press. ISBN 978-1-107-18582-1.
5. ^ Press, W. H.; Flannery, B. P.; Teukolsky, S. A.; Vetterling, W. T. (1992). "Least Squares as a Maximum Likelihood Estimator". Numerical Recipes in FORTRAN: The Art of Scientific Computing (2nd ed.). Cambridge: Cambridge University Press. pp. 651–655. ISBN 0-521-43064-X.
6. ^ Myung, I. J. (2003). "Tutorial on Maximum Likelihood Estimation". Journal of Mathematical Psychology. 47 (1): 90–100. doi:10.1016/S0022-2496(02)00028-7.
7. ^ Gourieroux, Christian; Monfort, Alain (1995). Statistics and Econometrics Models. Cambridge University Press. p. 161. ISBN 0-521-40551-3.
8. ^ Kane, Edward J. (1968). Economic Statistics and Econometrics. New York: Harper & Row. p. 179.
9. ^ Small, Christoper G.; Wang, Jinfang (2003). "Working with Roots". Numerical Methods for Nonlinear Estimating Equations. Oxford University Press. pp. 74–124. ISBN 0-19-850688-0.
10. ^ Kass, Robert E.; Vos, Paul W. (1997). "Geometrical Foundations of Asymptotic Inference". New York: John Wiley & Sons. p. 14. ISBN 0-471-82668-5.
11. ^ Papadopoulos, Alecos (September 25, 2013). "Why we always put log() before the joint pdf when we use MLE (Maximum likelihood Estimation)?". Stack Exchange.
12. ^ Pfanzagl (1994, p. 206)
13. ^ By Theorem 2.5 in Newey, Whitney K.; McFadden, Daniel (1994). "Chapter 36: Large sample estimation and hypothesis testing". In Engle, Robert; McFadden, Dan (eds.). Handbook of Econometrics, Vol.4. Elsevier Science. pp. 2111–2245. ISBN 978-0-444-88766-5.
14. ^ By Theorem 3.3 in Newey, Whitney K.; McFadden, Daniel (1994). "Chapter 36: Large sample estimation and hypothesis testing". In Engle, Robert; McFadden, Dan (eds.). Handbook of Econometrics, Vol.4. Elsevier Science. pp. 2111–2245. ISBN 978-0-444-88766-5.
15. ^ Zacks, Shelemyahu (1971). The Theory of Statistical Inference. New York: John Wiley & Sons. p. 223. ISBN 0-471-98103-6.
16. ^ See formula 20 in Cox, David R.; Snell, E. Joyce (1968). "A general definition of residuals". Journal of the Royal Statistical Society, Series B. 30 (2): 248–275. JSTOR 2984505.
17. ^ Kano, Yutaka (1996). "Third-order efficiency implies fourth-order efficiency". Journal of the Japan Statistical Society. 26: 101–117. doi:10.14490/jjss1995.26.101.
18. ^ Fletcher, R. (1987). Practical Methods of Optimization (Second ed.). New York: John Wiley & Sons. ISBN 0-471-91547-5.
19. ^ Nocedal, Jorge; Wright, Stephen J. (2006). Numerical Optimization (Second ed.). New York: Springer. ISBN 0-387-30303-0.
20. ^ Daganzo, Carlos (1979). Multinomial Probit : The Theory and its Application to Demand Forecasting. New York: Academic Press. pp. 61–78. ISBN 0-12-201150-3.
21. ^ Gould, William; Pitblado, Jeffrey; Poi, Brian (2010). Maximum Likelihood Estimation with Stata (Fourth ed.). College Station: Stata Press. pp. 13–20. ISBN 978-1-59718-078-8.
22. ^ Amemiya, Takeshi (1985). Advanced Econometrics. Cambridge: Harvard University Press. pp. 137–138. ISBN 0-674-00560-0.
23. ^ Sargan, Denis (1988). "Methods of Numerical Optimization". Lecture Notes on Advanced Econometric Theory. Oxford: Basil Blackwell. pp. 161–169. ISBN 0-631-14956-2.
24. ^ See theorem 10.1 in Avriel, Mordecai (1976). Nonlinear Programming: Analysis and Methods. Englewood Cliffs: Prentice-Hall. pp. 293–294.
25. ^ Gill, Philip E.; Murray, Walter; Wright, Margaret H. (1981). Practical Optimization. London: Academic Press. pp. 312–313. ISBN 0-12-283950-1.
26. ^ Edgeworth, Francis Y. (Sep 1908). "On the probable errors of frequency-constants". Journal of the Royal Statistical Society. 71 (3): 499–512. doi:10.2307/2339293. JSTOR 2339293.
27. ^ Edgeworth, Francis Y. (Dec 1908). "On the probable errors of frequency-constants". Journal of the Royal Statistical Society. 71 (4): 651–678. doi:10.2307/2339378. JSTOR 2339378.
28. ^ Pfanzagl, Johann, with the assistance of R. Hamböker (1994). Parametric Statistical Theory. Walter de Gruyter. pp. 207–208. ISBN 978-3-11-013863-4.CS1 maint: Multiple names: authors list (link)
29. ^ Wilks, S. S. (1938). "The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses". Annals of Mathematical Statistics. 9: 60–62. doi:10.1214/aoms/1177732360.
30. ^ Owen, Art B. (2001). Empirical Likelihood. London: Chapman & Hall/Boca Raton, FL: CRC Press. ISBN 978-1584880714.
31. ^ Wilks, Samuel S. (1962), Mathematical Statistics, New York: John Wiley & Sons. ISBN 978-0471946502.
32. ^ Savage, Leonard J. (1976). "On rereading R. A. Fisher". The Annals of Statistics. 4 (3): 441–500. doi:10.1214/aos/1176343456. JSTOR 2958221.
33. ^ Pratt, John W. (1976). "F. Y. Edgeworth and R. A. Fisher on the efficiency of maximum likelihood estimation". The Annals of Statistics. 4 (3): 501–514. doi:10.1214/aos/1176343457. JSTOR 2958222.
34. ^ Stigler, Stephen M. (1978). "Francis Ysidro Edgeworth, statistician". Journal of the Royal Statistical Society, Series A. 141 (3): 287–322. doi:10.2307/2344804. JSTOR 2344804.
35. ^ Stigler, Stephen M. (1986). The history of statistics: the measurement of uncertainty before 1900. Harvard University Press. ISBN 978-0-674-40340-6.
36. ^ Stigler, Stephen M. (1999). Statistics on the table: the history of statistical concepts and methods. Harvard University Press. ISBN 978-0-674-83601-3.
37. ^ Hald, Anders (1998). A history of mathematical statistics from 1750 to 1930. New York, NY: Wiley. ISBN 978-0-471-17912-2.
38. ^ Hald, Anders (1999). "On the history of maximum likelihood in relation to inverse probability and least squares". Statistical Science. 14 (2): 214–222. doi:10.1214/ss/1009212248. JSTOR 2676741.
39. ^ Aldrich, John (1997). "R. A. Fisher and the making of maximum likelihood 1912–1922". Statistical Science. 12 (3): 162–176. doi:10.1214/ss/1030037906. MR 1617519.