Universal Methods for Constructing Estimators

WarningDraft

This note is still being edited: the text may contain typos, inaccuracies, and locally unpolished passages. If you find something, please let the author know (contact information on the course page).

Let \(\Theta \subset \mathbb {R}^{k}\). Consider some family of distributions (a parametric model) \(\left\{ \mathbb {P}_\theta , \; \theta \in \Theta \right\}\) and a sample \(X_{1}, \ldots , X_{n}\) from an unknown distribution \(\mathbb {P}_{\theta }\). In this chapter we consider the 3 most common universal ways of constructing estimators of the parameter \(\theta\).

1 The Method of Moments

Recall that the \(k\)-th moment (\(k \in \mathbb {N}\)) of a random variable \(X\) is the expectation \(\mathbb {E}\left[X^k\right]\). The \(k\)-th sample moment of a sample \(X_1, \ldots , X_n\) is the quantity

\[ \overline{X^k} := \frac{1}{n}\sum _{i=1}^n X_i^k \]

The idea of the method of moments (abbreviated MM) is the following: if some moment of the r.v. \(X_1\) can easily be expressed in terms of \(\theta\), then we can replace the moment by the sample moment and \(\theta\) by its estimator \(\widehat{\theta }\), thereby obtaining an estimator.

ExampleExample 1

Suppose we have a sample from the exponential distribution \(\operatorname {Exp}(\theta ), \; \theta > 0\) with density \[ f_\theta (x) = \theta \cdot e^{-\theta x} \cdot \; \mathbb {1}_{\mathbb {R}_+}(x) \] Construct a method-of-moments estimator for \(\theta\).

SolutionSolution

Here \(\theta \in \Theta = (0, +\infty )\), the parameter is one-dimensional. Next, \(\mathbb {E}_{\theta }\left[X_1\right] = \frac{1}{\theta }\). Replacing the expectation (1st moment) by the sample mean (1st sample moment), and \(\theta\) by \(\widehat{\theta }\), we get \[ \overline{X} = \frac{1}{\widehat{\theta }} \] i.e. \(\widehat{\theta } = \frac{1}{\overline{X}}\).

AnswerAnswer

\(\widehat{\theta }_{\operatorname {MM}} = \frac{1}{\overline{X}}\)

In the case of a multidimensional parameter, several moments can be used to find an estimator.

ExampleExample 2

Find method-of-moments estimators for the family of distributions \(\mathcal{N}\left(a, \sigma^{2}\right)\). Treat \(\sigma^{2}\) as a single parameter here. That is, we need to estimate the variance \(\sigma^{2}\) of the normal distribution itself, not the standard deviation \(\sigma\).

SolutionSolution

Here \(\theta =\left(a, \sigma^{2}\right), \Theta =\mathbb {R} \times (0, \infty )\). We have \[ \begin{cases} \mathbb {E}_{\theta }\left[X_1\right] & = a \\ \mathbb {E}_{\theta }\left[X_1^2\right] & = \sigma ^2 + a^2 \end{cases} \] Replacing the moments by the sample moments, and the parameters by their estimators, we get the system \[ \begin{cases} \overline{X} & = \widehat{a} \\ \overline{X^2} & = \widehat{\sigma ^2} + \widehat{a}^2 \end{cases} \] Solving the system: \[ \begin{cases} \overline{X} & = \widehat{a} \\ \overline{X^2} & = \widehat{\sigma ^2} + \widehat{a}^2 \end{cases} \iff \begin{cases} \widehat{a} & = \overline{X} \\ \widehat{\sigma ^2} & = \overline{X^2} - \overline{X}^2 \end{cases} \]

AnswerAnswer

\(\widehat{a}_{\operatorname {MM}} = \overline{X}, \quad \widehat{\sigma^2}_{\operatorname {MM}} = \overline{X^2} - \overline{X}^2\).

ProblemProblem 1

Find method-of-moments estimators for the following distributions

1. \(\mathscr {N}\left(a, \sigma^2\right), \; a \in \mathbb {R}, \sigma > 0\);

2. \(\Gamma (\alpha , \beta )\). Recall that the density of the gamma distribution equals

\[ f_{\alpha , \beta } = \frac{\beta ^\alpha }{\Gamma (\alpha )}x^{\alpha - 1}e^{-\beta x} \; \mathbb {1}_{x \in [0, +\infty )}, \]

where \(\alpha , \beta > 0\). The gamma function: \(\Gamma (\alpha ) = \int_0^{\infty } x^{\alpha - 1}e^{-x}dx\). This is a continuous analogue of the factorial function, namely \(\Gamma (n+1) = n!, \; n \in \mathbb {Z}_+\). In general, \(\Gamma (\alpha + 1) = \alpha \Gamma (\alpha ), \; \forall \alpha > 0\). Another useful value: \(\Gamma (\frac{1}{2}) = \sqrt{\pi }\).

3. \(U[a,b], \; -\infty < a < b < +\infty\)

4. \(\operatorname {Pois}\left(\lambda \right), \; \lambda > 0\);

5. \(\operatorname {Bin}\left(m, p\right), \; m \in \mathbb {N}, p \in (0,1)\);

6. \(\operatorname {Geom}\left(p\right)\). Recall that \(\operatorname {Geom}\left(p\right)\) is a discrete distribution with support \(\mathbb {Z}_+\) and the following probability mass function: \(f_{p}(k) = (1-p)^kp, \; k \in \mathbb {Z}_+\).

7. \(\operatorname {Beta}(\lambda_1, \lambda_2)\). Recall that the density of the beta distribution equals

\[ f_{\lambda _1, \lambda _2}(x) = \frac{x^{\lambda _1-1}(1 - x)^{\lambda _2 - 1}}{B(\lambda _1, \lambda _2)} \; \mathbb {1}_{x \in [0,1]}, \]

\(\lambda_1, \lambda_2 > 0\). The beta function \(B(\lambda_1, \lambda_2)\):

\[ B(\lambda _1, \lambda _2) = \int _0^1 x^{\lambda _1 - 1}(1 - x)^{\lambda _2 - 1} dx = \frac{\Gamma (\lambda _1)\Gamma (\lambda _2)}{\Gamma (\lambda _1 + \lambda _2)} \]

Note. Each part is worth 0.3 points.

In general, the distribution may not have direct moments at all. However, if other expectations \(\mathbb {E}_{\theta }\left[g(X_1)\right]\) exist, for example \(\mathbb {E}_{\theta }\left[\ln (X_1)\right]\), then we can try to construct an estimator based on them. \(g\) is called a trial function; if \(g(x) = x^k\), \(k\in \mathbb {N}\), then \(g\) is called a standard trial function. In the examples above we used standard trial functions. Now let’s try using nonstandard ones.

ExampleExample 3

Suppose we have a sample \(X_1,\ldots , X_n\) from the log-logistic distribution \(\operatorname {LL}(\alpha , \beta ), \; \alpha , \beta > 0\) with density \[ p(x) = \frac{(\beta /\alpha ) (x/\alpha )^{\beta - 1}}{(1 + (x/\alpha )^\beta )^2} \; \mathbb {1}_{\mathbb {R}_+}(x) \] Construct method-of-moments estimators for \(\alpha , \beta\).

SolutionSolution

Here \(\theta = (\alpha , \beta )^T \in \Theta = (0, +\infty ) \times (0, +\infty )\), the parameter is two-dimensional.

For \(m > 0\) we have

\[ \begin{align} \mathbb {E}_{\theta }\left[X^m_1\right] & = \int _{0}^{\infty } \frac{(\beta /\alpha ) (x/\alpha )^{\beta - 1}}{(1 + (x/\alpha )^\beta )^2} \cdot x^m \cdot dx = \\ & = \beta \alpha ^m \int _{0}^{\infty } \frac{(x/\alpha )^{\beta - 1 + m}}{(1 + (x/\alpha )^\beta )^2} d(x/\alpha ) = \bigg\rvert y = x/\alpha \bigg\rvert = \\ & = \beta \alpha ^m \int _{0}^{\infty } \frac{y^{\beta - 1 + m}}{(1 + y^\beta )^2}dy \end{align} \]

Next, make the substitution \(z = \frac{1}{1 + y^\beta }, \; y = \left(\frac{1 - z}{z}\right)^{\frac{1}{\beta }}, \; dy = -\frac{1}{\beta }\left(\frac{1 - z}{z}\right)^{\frac{1}{\beta } - 1} z^{-2} dz\) (here \(y : 0 \to \infty\) corresponds to \(z : 1 \to 0\)). We obtain

\[ \begin{align} \mathbb {E}_{\theta }\left[X^m_1\right] & = \beta \alpha ^m \cdot \left(-\int _{1}^{0} \left(\frac{1 - z}{z}\right)^{1 - \frac{1}{\beta } + \frac{m}{\beta }} z^2 \frac{1}{\beta }\left(\frac{1 - z}{z}\right)^{\frac{1}{\beta } - 1} z^{-2} dz\right) \end{align} \]

Switch the sign together with the order of the integration limits, cancel \(z^2 \cdot z^{-2} = 1\) and \(\beta \cdot \frac{1}{\beta } = 1\), and add the exponents at \(\frac{1-z}{z}\): \(\left(1 - \frac{1}{\beta } + \frac{m}{\beta }\right) + \left(\frac{1}{\beta } - 1\right) = \frac{m}{\beta }\). We obtain

\[ \begin{align} \mathbb {E}_{\theta }\left[X^m_1\right] & = \alpha ^m \int _0^1 \left(\frac{1-z}{z}\right)^{\frac{m}{\beta }} dz = \alpha ^m \int _0^1 z^{-\frac{m}{\beta }}\left(1 - z\right)^{\frac{m}{\beta }} dz = \\ & = \begin{cases} \alpha ^m B(1 - \frac{m}{\beta }, 1 + \frac{m}{\beta }), & \text{ if } 1 - \frac{m}{\beta } > 0 \iff \beta > m \\ \infty , & \text{ otherwise} \end{cases} \end{align} \]

For \(\beta < 1\) the expectation \(\mathbb {E}_{\theta }\left[X_1\right]\), and consequently all other moments of \(X_1\), do not exist, so it is impossible to construct a method-of-moments estimator based on standard trial functions.

However, it is known that the random variable \(\ln (X_1)\) has the logistic distribution \(\operatorname {Logistic}(\mu , s)\) with parameters \(\mu = \ln (\alpha ), \; s = 1/\beta\). It is known that for this distribution the expectation and variance exist for any values of the parameters and equal \(\mu\), \(s^2 \pi^2/ 3\) respectively. Then

\[ \begin{align} \mathbb {E}\left[\ln (X_1)\right] & = \mu = \ln (\alpha ), \\ \mathbb {E}\left[\ln ^2(X_1)\right] & = \operatorname {Var}\left[\ln (X_1)\right] + \left(\mathbb {E}\left[\ln (X_1)\right]\right)^2 = \frac{s^2\pi ^2}{3} + \mu ^2 = \\ & = \frac{\pi ^2}{3\beta ^2} + \ln ^2(\alpha ) \end{align} \]

To construct method-of-moments estimators, replace the expectations by sample characteristics, and the parameters by their estimators:

\[ \begin{align} \mathbb {E}\left[\ln (X_1)\right] \; & \longrightarrow \; \overline{\ln (X)} = \frac{1}{n}\sum _{i=1}^n \ln (X_i), \\ \mathbb {E}\left[\ln ^2(X_1)\right] \; & \longrightarrow \; \overline{\ln ^2(X)} = \frac{1}{n}\sum _{i=1}^n \ln ^2(X_i) \\ \alpha \; & \longrightarrow \; \widehat{\alpha }, \\ \beta \; & \longrightarrow \; \widehat{\beta } \end{align} \]

We obtain

\[ \begin{cases} \overline{\ln (X)} & = \ln (\widehat{\alpha }) \\ \overline{\ln ^2(X)} & = \frac{\pi ^2 }{3\widehat{\beta }^2} + \ln ^2(\widehat{\alpha }) \end{cases} \iff \begin{cases} \widehat{\alpha } & = e^{\overline{\ln (X)}} \\ \widehat{\beta } & = \pi \frac{1}{\sqrt{3\left( \overline{\ln ^2(X)} - \overline{\ln (X)}^2\right)}} \end{cases} \]

AnswerAnswer

\[ \widehat{\alpha }_{\operatorname {MM}} = e^{\overline{\ln (X)}}, \qquad \widehat{\beta }_{\operatorname {MM}} = \pi \frac{1}{\sqrt{3 \cdot \left( \overline{\ln ^2(X)} - \overline{\ln (X)}^2\right)}} \]

Recall that \(\operatorname {Pareto}(x_0, \alpha )\), \(x_0, \alpha > 0\) is a continuous distribution with density

\[ f_{x_0, \alpha }(x) = \frac{\alpha x_0^\alpha }{x^{\alpha + 1}}\; \mathbb {1}_{x \in [ x_0, +\infty )}. \]

\(x_0\) is interpreted as a shift of the distribution, and \(\alpha\) as the shape parameter.

Recall that \(\operatorname {Cauchy}(x_0, \gamma )\), \(x_0 \in \mathbb {R}\), \(\gamma > 0\) is a continuous distribution with density

\[ f_{x_0, \gamma }(x) = \frac{1}{\pi \gamma \left(1 + \left(\frac{x - x_0}{\gamma }\right)^2\right)} = \frac{\gamma }{\pi \left(\gamma ^2 + (x - x_0)^2\right)}. \]

\(x_0\) is interpreted as a shift of the distribution, and \(\gamma\) as the scale.

ProblemProblem 2

Find method-of-moments estimators for the parameters of the following distributions

1. the distribution \(\operatorname {Pareto}(1, \alpha ), \; \alpha > 0\) with density \(f_{\alpha }(x) = \frac{\alpha }{x^{\alpha + 1}}\; \mathbb {1}_{[1, +\infty )}(x)\);

2. the distribution \(\operatorname {Cauchy}(0, \gamma ), \; \gamma > 0\) with density \(f_{\gamma }(x) = \frac{\gamma }{\pi (\gamma^2 + x^2)}\).

Let us give a general description of the method of moments for constructing an estimator of \(\theta\). To construct an estimator, we choose Borel functions \(g_{1}, \ldots , g_{k}: \mathbb {R} \rightarrow \mathbb {R}\) (the number of functions \(k\) equals the dimension of the parameter \(\theta\)) such that for every \(i \in \left\{ 1, \ldots , k\right\}\) and every \(\theta \in \Theta\) there exists a finite expectation \(\mathbb {E}_{\theta }\left[g_i(X_1)\right]\). Denote

\[ \begin{cases} m_1(\theta ) & := \mathbb {E}_{\theta }\left[g_1(X_1)\right], \\ & \vdots \\ m_k(\theta ) & := \mathbb {E}_{\theta }\left[g_k(X_1)\right] \end{cases} \qquad \text{ or } \qquad m(\theta ) = \begin{pmatrix} \mathbb {E}_{\theta }\left[g_1(X_1)\right] \\ \vdots \\ \mathbb {E}_{\theta }\left[g_k(X_1)\right] \end{pmatrix}, \quad \text{ where } \quad m(\theta ) := \begin{pmatrix} m_1(\theta ) \\ \vdots \\ m_k(\theta ) \end{pmatrix} \]

Replace the parameter \(\theta\) by its estimator \(\widehat{\theta }\), and the expectations \(\mathbb {E}_{\theta }\left[g_i(X_1)\right]\) by the sample expectations

\[ \overline{g_i(X)} = \frac{1}{n}\sum _{j=1}^n g_i(X_j) \]

We obtain a system of equations for \(\widehat{\theta }\):

\[ \begin{cases} m_1(\widehat{\theta }) & = \overline{g_1(X)}, \\ & \vdots \\ m_k(\widehat{\theta }) & = \overline{g_k(X)} \end{cases} \]

Suppose this system has a unique solution. Then this unique solution \(\theta^{*}(X) = \widehat{\theta }_{\operatorname {MM}}(X)\) is called the method-of-moments estimator with trial functions \(g_{1}, \ldots , g_{k}\). The functions \(g_{i}(x)=x^{i}, i \in \{ 1, \ldots , k\}\), are called standard trial functions. When standard trial functions are used, \(\mathbb {E}\left[g_i(X_1)\right]\) is some moment of the random variable.

Theorem 1 If \(m: \Theta \rightarrow m(\Theta )\) is a bijection and the function \(m^{-1}\) can be extended to a function defined on all of \(\mathbb {R}^{k}\) and continuous at every point of \(m(\Theta )\), then the method-of-moments estimator is strongly consistent.

Moreover, the statement of this theorem can be strengthened as follows.

ExampleExample 4

Prove that, under the conditions of the theorem above, the method-of-moments estimator is asymptotically normal if the function \(m^{-1}\) can be extended to a function defined on all of \(\mathbb {R}^{k}\) and differentiable at every point of \(m(\Theta )\), and, moreover, for every \(i \in \{ 1, \ldots , k\}\) and every \(\theta \in \Theta\) the inequality \[ 0 < \operatorname {Var}_{\theta }\left[g_i(X_1)\right] < +\infty \] holds.

SolutionSolution

So, the method-of-moments estimator equals \(\theta^{*}=m^{-1}\left(\overline{g(X)}\right)\), where \(\overline{g(X)}=\left(\overline{g_{1}(X)}, \ldots , \overline{g_{k}(X)}\right)^{T}\). By the multivariate central limit theorem, for every \(\theta \in \Theta\) we have \[ \sqrt{n}\left(\overline{g(X)}-m(\theta )\right) \xrightarrow {d_{\theta }} \xi \sim \mathcal{N}(0, \Sigma (\theta )), \] where \(\Sigma (\theta )\) is the covariance matrix of the vector \(\left(g_{1}\left(X_{1}\right), \ldots , g_{k}\left(X_{1}\right)\right)^{T}\). By the delta method (\(h=m^{-1}\), \(b_{n}=1 / \sqrt{n}\), \(a=m(\theta )\), \(\xi_{n}=\sqrt{n}\left(\overline{g(X)}-m(\theta )\right)\)), for \(k = 1\) we have \[ \sqrt{n}\left(m^{-1}\left(\overline{g(X)}\right) - \theta \right) \xrightarrow []{d_\theta } \left\langle \xi , \nabla (m^{-1})\vert _{m(\theta )} \right\rangle , \] and the limiting distribution is Gaussian: \[ \left\langle \xi , \nabla (m^{-1})\vert _{m(\theta )} \right\rangle \sim \mathscr {N}\left(0, \left(\nabla (m^{-1})\vert _{m(\theta )}\right)^T \cdot \Sigma (\theta ) \cdot \nabla (m^{-1})\vert _{m(\theta )}\right). \] For \(k > 1\) the argument is the same, except that the delta method is applied to the vector-valued function \(h = m^{-1}: \mathbb {R}^{k} \to \mathbb {R}^{k}\) (componentwise): the gradient is replaced by the Jacobian matrix \(J := \frac{\partial m^{-1}}{\partial x}\big\vert_{m(\theta )}\), and the limiting distribution turns out to be \(\mathscr {N}\left(0, J \Sigma (\theta ) J^T\right)\) — asymptotic normality (now with an asymptotic covariance matrix) is preserved.

Note that the method-of-moments estimator is not necessarily unbiased.

ProblemProblem 3

Let \(X_1, \ldots , X_n\) be a sample from the distribution \(\operatorname {Pois}\left(\lambda \right), \; \lambda > 0\). Find an unbiased estimator of the parameter \(\lambda^3\).

2 The Maximum Likelihood Method

Let \(\mathcal{P}=\left\{ \mathbb {P}_{\theta }, \theta \in \Theta \right\}\) be a parametric family of distributions. We will consider 2 cases:

• All \(\mathbb {P}_{\theta }\) are absolutely continuous distributions. In this case let \(f_{\theta }(x)\) be the density of the distribution \(\mathbb {P}_{\theta }\).

• All \(\mathbb {P}_{\theta }\) are discrete distributions (with support in \(\mathbb {Z}\)). In this case let \(f_{\theta }(x)\) be the probability mass function of the distribution \(\mathbb {P}_{\theta }\).

In both cases we will call the family \(\mathcal{P}\) dominated.

Let \(\mathcal{P}=\left\{ \mathbb {P}_{\theta }, \theta \in \Theta \right\}\) be a dominated family of distributions, \(\vec{X} = (X_{1}, \ldots , X_{n})^T\) a sample from a distribution \(\mathbb {P} \in \mathcal{P}\). If \(f_\theta (x)\) is the density (probability mass function) of \(X_1\), then the density (probability mass function) \(f\) of the sample vector \(X = (X_1, \ldots , X_n)^T\) equals the product of the densities of the components, since the components are independent:

\[ f_\theta (\overrightarrow {x}) = f(\overrightarrow {x} ; \theta ) := f_\theta (x_1) \cdot f_\theta (x_2) \cdot \ldots \cdot f_{\theta }(x_n) \]

where \(\overrightarrow {x} = \left(x_1, \ldots , x_n\right)^T\). The joint density of the sample \(f\) depends on both \(\theta\) and \(\overrightarrow {x}\). However, it is customary when considering the density to fix the parameter \(\theta\) and let \(f\) vary with \(\overrightarrow {x}\).

The likelihood function of the sample \(\vec{X}\) is (the joint) density of the sample, if \(\vec{x}\) is treated as fixed and it is allowed to vary with the parameter \(\theta\):

\[ L(\theta ) = L(\theta ; \vec{x}) := f_\theta (x_1) \cdot \ldots \cdot f_{\theta }(x_n) \]

Sometimes the sample \(\vec{X}\) itself is substituted in place of \(\vec{x}\), and then the likelihood \(L(\theta ; \vec{X})\) becomes a random function: for each \(\omega \in \Omega\) it is its own function \(\Theta \to \mathbb {R}\):

\[ \left(L(\theta )\right)(\omega ) = L(\theta ; \vec{X}(\omega )) \]

For a fixed \(\theta\), the likelihood in this case is a random variable.

The maximum likelihood estimator (MLE) of the parameter \(\theta\) is

\[ \widehat{\theta }_{\operatorname {MLE}} := \operatorname *{arg\, max} _{\theta \in \Theta } L\left(\theta \right) \]

that is, the value \(\theta \in \Theta\) at which the likelihood function attains its maximum for fixed \(X_{1}, \ldots , X_{n}\).

In practice it is easier to look for the maximum not of the likelihood itself, but of the log-likelihood function:

\[ \ell (\theta ) := \ln L\left(\theta \right) \]

The logarithm is a strictly increasing function, so the point at which the maximum is attained does not change upon taking the logarithm. It is easier to work with the logarithm since it turns a product into a sum.

ExampleExample 5

Let \(X_{1}, \ldots , X_{n}\) be a sample from the distribution \(\operatorname {Bern}(\theta )\), \(\theta \in (0,1) = \Theta\). Find the maximum likelihood estimator of the parameter \(\theta\).

SolutionSolution

The probability mass function of the Bernoulli distribution:

\[ f_{\theta }(x) = \mathbb {P}\left(X_1 = x\right) = \begin{cases} \theta , & \text{ if } x = 1 \\ 1 - \theta , & \text{ if } x = 0 \end{cases} \]

Let us obtain a convenient representation for \(f_\theta\):

\[ f_\theta (x) = \theta ^x \cdot (1-\theta )^{1-x} \cdot \; \mathbb {1}_{\left\{ 0,1\right\} }(x) \]

The likelihood function:

\[ L(\theta ) = \prod _{i=1}^{n}f_{\theta }(X_i) = \theta ^{\sum X_{i}}(1-\theta )^{n-\sum X_{i}} \]

and the log-likelihood function (which, as a rule, is easier to differentiate to find the maximum) equals

\[ \begin{align} \ell (\theta ) & = \sum _{i=1}^n X_{i} \ln \theta + \left(n-\sum _{i=1}^n X_{i}\right) \ln (1-\theta ) = \\ & = n \cdot \left(\overline{X}\ln (\theta ) + \left(1 - \overline{X}\right)\ln (1-\theta )\right) \end{align} \]

To find the maximum likelihood estimator, differentiate \(\ell\) with respect to \(\theta\) (treating \(X_1, \ldots , X_n\) as fixed):

\[ \frac{\partial \ell (\theta )}{\partial \theta } = n \cdot \left( \frac{\overline{X}}{\theta }-\frac{1-\overline{X}}{1-\theta }\right) \]

Bring to a common denominator:

\[ \frac{\partial \ell (\theta )}{\partial \theta } = n \cdot \frac{\overline{X}(1-\theta ) - \theta (1-\overline{X})}{\theta (1-\theta )} = n \cdot \frac{\overline{X} - \theta }{\theta (1 - \theta )} \]

Set the derivative equal to zero:

\[ 0 = \frac{\partial \ell (\theta )}{\partial \theta } = n \cdot \frac{\overline{X} - \theta }{\theta (1 - \theta )} \iff \theta = \overline{X} \]

Is \(\widehat{\theta } := \overline{X}\) the point at which the likelihood (or, equivalently, the log-likelihood) attains its maximum? Note that for \(0 < \theta < \overline{X}\) the derivative \(\frac{\partial \ell (\theta )}{\partial \theta }\) is positive, and for \(\overline{X} < \theta < 1\) the derivative is negative. Hence \(\widehat{\theta } = \overline{X}\) is a global maximum point of the function \(\ell (\theta )\) for fixed \(X_1,\ldots , X_n\) and \(\theta \in (0,1)\).

AnswerAnswer

\(\widehat{\theta }_{\operatorname {MLE}} = \overline{X}\)

Geometrically, the MLE is the maximum point of the likelihood function (or, equivalently, of its logarithm) over the parameter \(\theta\) for a fixed, observed sample.

The log-likelihood function \(\ell(\theta)\) for a specific sample from \(\operatorname{Bern}(\theta)\): the unique maximum is attained at the MLE \(\hat\theta=\overline{X}\). The button below generates a new sample with the true parameter \(\theta_{\text{true}}\) – note how the shape of the curve and the position of the maximum change from sample to sample, but on average the maximum stays near \(\theta_{\text{true}}\); the larger \(n\), the sharper the peak (the more confident the estimate).

n = 20 θ_true = 0.5

Xᵢ =
θ̂ = X̄ =

ProblemProblem 4

Find maximum likelihood estimators for the following distributions:

1. \(\mathscr {N}\left(a, \sigma^2\right)\) in three cases: when only one of the parameters is unknown, and when both parameters are unknown;

2. \(\Gamma (\alpha , \beta ), \; \alpha ,\beta > 0\), if the parameter \(\alpha\) is known;

3. \(U[a,b], \; -\infty < a < b < +\infty\)

4. \(\operatorname {Pois}\left(\lambda \right), \; \lambda > 0\);

5. \(\operatorname {Bin}\left(m, p\right), \; m \in \mathbb {N}, p \in (0,1)\), if the parameter \(m\) is known;

6. \(\operatorname {Geom}\left(p\right), \; p \in (0,1)\).

7. \(\operatorname {Pareto}(\theta , 2), \; \theta > 0\).

Note. Each part is worth 0.3 points.

ProblemProblem 5

Let \(X_1, \ldots , X_n\) be a sample from a shifted exponential distribution with density \[ f_{\alpha , \beta } = \frac{1}{\alpha } e^{\frac{\beta - x}{\alpha }} \; \mathbb {1}_{x \in [\beta , +\infty )}, \] where \(\theta = (\alpha , \beta )^T \in \mathbb {R}_+ \times \mathbb {R}\). Find the maximum likelihood estimator for \(\theta\). Prove that the resulting estimator \(\widehat{\alpha }\) for \(\alpha\) is asymptotically normal, and find its asymptotic variance.

ProblemProblem 6

Find the maximum likelihood estimator of the shift parameter \(x_0\) in the Cauchy distribution \(\operatorname {Cauchy}(x_0, 1)\), if the sample consists of

1. one observation,

2. two observations.

ProblemProblem 7

Let \(X_1, \ldots , X_n\) be a sample from the distribution \(e^{U[0, \theta ]}, \; \theta > 0\). That is, if \(\xi \sim U[0, \theta ]\), then \(X_1 \sim e^\xi\). Find the maximum likelihood estimator of the parameter \(\theta\) and check it for consistency.

ProblemProblem 8

Let \(X_1, \ldots , X_n\) be a sample from the Laplace distribution with density \[ f_\theta (x) = \frac{1}{2}e^{-\left|x - \theta \right|} \] Find the maximum likelihood estimator of the parameter \(\theta\).

Let us turn to the properties of the maximum likelihood estimator.

Theorem 2 Let \(\Theta \subset \mathbb {R}\), suppose the regularity conditions hold (see, for example, (Ивченко and Медведев 2010, sec. 3.5.4)), the likelihood function has only one local maximum, lying inside \(\Theta\) and coinciding with the maximum likelihood estimator, and, finally, the likelihood function is three times differentiable in \(\theta\) and there exists a function \(M(x)\) not depending on \(\theta\) such that for all \(\theta \in \Theta\) \[ \left|\frac{\partial ^{3} f_{\theta }(x)}{\partial \theta ^{3}}\right| \leq M(x), \qquad \mathbb {E}_{\theta }\left[M(X_1)\right] < \infty \] Then the maximum likelihood estimator is an asymptotically normal estimator of \(\theta\) with asymptotic variance \[ n \left(\operatorname {Var}_{\theta }\left[\frac{\partial \ell (\theta )}{\partial \theta }\right]\right)^{-1}, \] where \(\ell (\theta ) = \ln L(\theta )\) is the log-likelihood function of the entire sample (as defined above).

ExampleExample 6

Let \(X_1, \ldots , X_n\) be a sample from \(\operatorname {Pois}\left(\lambda \right)\), \(\lambda > 0\). Find the asymptotic variance of the maximum likelihood estimator \(\widehat{\lambda }_{\operatorname {MLE}}\).

SolutionSolution

The log-likelihood function of the sample: \[ \ell (\lambda ) = \sum _{i=1}^n X_i \ln \lambda - n\lambda - \sum _{i=1}^n \ln (X_i!). \] Then \[ \frac{\partial \ell (\lambda )}{\partial \lambda } = \frac{\sum _{i=1}^n X_i}{\lambda } - n, \] from which \[ \operatorname {Var}_{\lambda }\left[\frac{\partial \ell (\lambda )}{\partial \lambda }\right] = \frac{\sum _{i=1}^n \operatorname {Var}_{\lambda }\left[X_i\right]}{\lambda ^2} = \frac{n\lambda }{\lambda ^2} = \frac{n}{\lambda }. \] By the theorem, the asymptotic variance of \(\widehat{\lambda }_{\operatorname {MLE}}\) equals \(n \cdot \left(\frac{n}{\lambda }\right)^{-1} = \lambda\). This can also be confirmed directly: since \(\widehat{\lambda }_{\operatorname {MLE}} = \overline{X}\) (see the method of moments for the Poisson distribution), by the CLT \[ \sqrt{n}\left(\overline{X} - \lambda \right) \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, \operatorname {Var}\left[X_1\right]\right) = \mathscr {N}\left(0, \lambda \right), \] which coincides exactly with the answer obtained from the general theorem.

AnswerAnswer

The asymptotic variance equals \(\lambda\).

ProblemProblem 9

Let \(X_1, \ldots , X_n\) be a sample from the distribution \(U[0, \theta ], \; \theta > 0\), with \(n \geq 2\). Find an unbiased estimator of the parameter \(\frac{1}{\theta }\).

3 The Method of Sample Quantiles

Definition 1 Let \(\mathbf{P}\) be some distribution, \(F\) its distribution function. We will regard

\[ F: \mathbb {R} \cup \left\{ -\infty , +\infty \right\} \to [0,1] \]

Let, in addition, \(p \in [0,1]\). The quantile function of the distribution \(\mathbf{P}\) is the generalized inverse function to \(F\):

\[ Q: [0,1] \to \mathbb {R} \cup \left\{ -\infty , +\infty \right\} , \qquad Q(p) := \inf \{ x: F(x) \geq p\} \]

\(Q(p)\) is also called the quantile of level \(p \in [0,1]\).

The median of the distribution \(F\) is the quantile of level \(\frac{1}{2}\), i.e. the value \(Q(0.5)\).

If \(F\) is bijective, i.e. strictly increasing and without discontinuities, then the quantile function is simply the inverse function of \(F\): \(Q = F^{-1}\). In the general case, however, \(F\) may have jumps and flat intervals, and the “generalized” nature of the inverse function is precisely in how \(Q\) handles them:

The quantile function as a generalized inverse

On the left — the distribution function \(F(x)\) with a jump at \(x=2\) and a plateau on \([4,5]\). On the right — the quantile function \(Q(p) = \inf\{x : F(x) \geq p\}\) (same axes, rotated: the level \(p\) is always on the vertical axis). Move the slider or press the buttons — the red point on both plots shows the pair \((Q(p), p)\).

\(F(x)\)

\(Q(p) = \inf\{x : F(x) \geq p\}\)

Level \(p =\) 0.50 Q(p) = 2.00

Definition 2 Let \(X_{1}, \ldots , X_{n}\) be a sample from an unknown distribution \(\mathbb {P}\). The statistic \[ \widehat{Q}(p) := X_{(\lceil np \rceil )} = \begin{cases} X_{([n p]+1)}, & \text{ if } n p \notin \mathbb {Z} \\ X_{(n p)}, & \text{ if } n p \in \mathbb {Z}\end{cases} \] is called the sample \(p\)-quantile.

Definition 3 The sample median of a sample \(X_{1}, \ldots X_{n}\) refers either to the statistic \[ \widehat{\mu } := \widehat{Q}(0.5) = \begin{cases} X_{(k+1)}, & \text{ if } n=2 k+1 \\ X_{(k)}, & \text{ if } n=2 k\end{cases} \] or to \[ \widehat{\mu } := \begin{cases} X_{(k+1)}, & \text{ if } n=2 k+1 \\ \frac{X_{(k)}+X_{(k+1)}}{2}, & \text{ if } n=2 k\end{cases} \]

The definitions of the sample median differ slightly when \(n\) is even. In the first case we take the left value of the 2 central ones, and in the second the average of the 2 central ones. We will mainly use the second convention, although there is practically no difference between them, especially from the point of view of asymptotic properties.

Theorem 3 (on the asymptotic normality of the sample quantile) Let \(X_{1}, \ldots X_{n}\) be a sample from an absolutely continuous one-dimensional distribution \(\mathbb {P}\) with density \(f\). Let \(p \in (0,1)\) and \(Q(p)\) be the \(p\)-quantile of the distribution \(\mathbb {P}\), with \(f\) continuously differentiable in a neighborhood of the point \(Q(p)\) and \(f\left(Q(p)\right)>0\). Then \(\widehat{Q}(p)\) is an asymptotically normal estimator of \(Q(p)\) with asymptotic variance \(\frac{p(1-p)}{f^{2}\left(Q(p)\right)}\): \[ \sqrt{n} \left(\widehat{Q}(p) - Q(p)\right) \xrightarrow [n\to \infty ]{d} \mathcal{N}\left(0, \frac{p(1-p)}{f^{2}\left(Q(p)\right)}\right) \]

In particular, for the median (\(p = \frac12\)) the asymptotic variance equals \(\frac{1}{4f^2\left(Q(0.5)\right)}\).

The asymptotic behavior of \(\widehat{\mu }\) coincides with that of \(\widehat{Q}(0.5)\), regardless of which definition of \(\widehat{\mu }\) is used.

ExampleExample 7

Let \(X_{1}, \ldots , X_{n}\) be a sample from \(\operatorname {Cauchy}(\theta , 1)\). Find an asymptotically normal estimator of the parameter \(\theta\) and its asymptotic variance.

SolutionSolution

Density:

\[ f_{\theta }(x)=\frac{1}{\pi \left(1+(x-\theta )^{2}\right)} \]

Since a random variable with this distribution has no finite expectation, we cannot use the central limit theorem.

Let us use the sample median. \(Q(0.5)=\theta\), so, by the theorem on the asymptotic normality of the sample median, \(\hat{\mu }\) is an asymptotically normal estimator of \(\theta\) with asymptotic variance

\[ \frac{1}{4 f_{\theta }^{2}(\theta )}=\frac{\pi ^{2}}{4} \]

Switch the distribution, move the (N) slider to look at the trajectories for different sample sizes, or press the button to regenerate the sample.

Sample mean and sample median: robustness to heavy tails

A single sample realization, \(\theta = 0\). For Cauchy, the mean occasionally makes large jumps and does not converge, while the median always converges.

Distribution Cauchy(0, 1) Normal(0, 1) \(N =\) 1

\(n =\) \(\bar X_n =\) median \(=\)

ProblemProblem 10

Propose an asymptotically normal estimator of the parameter \(\theta^2\) in the Cauchy distribution model \(\operatorname {Cauchy}(\theta , 1)\) with shift \(\theta\) (see the previous problem), and find its asymptotic variance.

References

Ивченко, Г. И., and Ю. И. Медведев. 2010. Введение в Математическую Статистику. ЛКИ. https://disk.yandex.ru/i/CwMvHKeSVCbYHA.