This set of notes is still being edited: the text may contain typos, inaccuracies, and locally unpolished spots. If you find something, please let the author know (contact information on the course page).
Let us continue studying the asymptotic properties of various estimators. Suppose we have a sample \(X_1, \ldots , X_n\) from the distribution \(\mathscr {N}\left(0, \sigma^2\right)\) with unknown parameter \(\sigma > 0\), which we want to estimate. Recall that
\[
\begin{align} \mathbb {E}\left[X^p\right] = \begin{cases} 0, & p \text{ -- odd} \\ \sigma ^p (p-1)!!, & p \text{ -- even} \end{cases} \end{align}
\]
In particular, \(\mathbb {E}\left[X_1^2\right] = \sigma^2, \mathbb {E}\left[X_1^4\right] = 3\sigma^4\). Sample means approximately coincide with the corresponding expectations, \(\overline{X^2} \approx \mathbb {E}\left[X_1^2\right] = \sigma^2, \overline{X^4} \approx \mathbb {E}\left[X_1^4\right] = 3\sigma^4\). Consequently, one can try to estimate \(\sigma\) in several ways at once:
\[
\widehat{\sigma }_1(X) = \sqrt{\overline{X^2}}, \quad \widehat{\sigma }_2(X) = \sqrt{\frac{\overline{X^4}}{3\overline{X^2}}}, \quad \widehat{\sigma }_3(X) = \sqrt[4]{\frac{\overline{X^4}}{3}}, \quad \ldots
\]
One can expect that these statistics will be approximately equal to the unknown parameter \(\sigma\). But will they, for example, be asymptotically normal? After all, we know that, by the CLT applied to the squares and fourth powers of the original sample, \(\overline{X^2}, \overline{X^4}\) are asymptotically normal estimators of \(\sigma^2, 3\sigma^4\).
To answer the questions above, we need a few new tools. First, we need a theorem on the so-called inheritance of asymptotic normality, i.e., some statement that would give asymptotic normality of the estimator \(f(\widehat{\theta })\) for some function \(f\), given that we know the asymptotic normality of \(\widehat{\theta }\). Such a statement would help in answering the question of the asymptotic normality of \(\widehat{\sigma }_1\), \(\widehat{\sigma }_3\).
Second, the samples \((X_1^2, X_2^2, \ldots , X_n^2)\) and \((X_1^4, X_2^4, \ldots , X_n^4)\) are dependent on each other. We cannot expect that applying the CLT separately to each of the samples will lead to an exact answer to the question about the limiting distribution of
\[
\widehat{\sigma }_2 = \sqrt{\frac{\overline{X^4}}{3\overline{X^2}}}
\]
We need some new version of the CLT that would take these dependencies into account, that would answer the question about the limiting distribution of the two-dimensional random vector \((\overline{X^2}, \overline{X^4})^T\). We need a multivariate version of the CLT, which we would apply to the (infinite) sample from a 2-dimensional distribution
\[
\begin{pmatrix} X_1^2 \\ X_1^4\end{pmatrix}, \begin{pmatrix} X_2^2 \\ X_2^4\end{pmatrix}, \begin{pmatrix} X_3^2 \\ X_3^4\end{pmatrix}, \ldots , \begin{pmatrix} X_n^2 \\ X_n^4\end{pmatrix}, \ldots .
\]
For this we need to give a definition of convergence for random vectors.
Convergence of Random Vectors
Modes of convergence in the multidimensional case are analogous to the one-dimensional case.
Definition 1 Let \(\left\{ \xi^{(n)}\right\}_{n \in \mathbb {N}}\) be a sequence of random vectors of dimension \(m\), defined on the same probability space \((\Omega , \mathcal{F}, \mathbb {P})\), \(\xi^{(n)} = (\xi^{(n)}_1, \ldots , \xi^{(n)}_m)^T\).
The sequence \(\xi^{(n)}\) converges almost surely to a random vector \(\xi\) as \(n \rightarrow \infty\) (written \(\xi^{(n)} \stackrel{\text{ a.s. }}{\longrightarrow } \xi\)) if
\[
\mathbb {P}\left(\omega : \lim _{n \to \infty } \xi ^{(n)}(\omega ) = \xi (\omega )\right) = 1
\]
The sequence \(\xi^{(n)}\) converges in probability to a random vector \(\xi\) as \(n \rightarrow \infty\) (written \(\xi^{(n)} \xrightarrow [n \to \infty ]{\mathbb {P}} \xi\)) if for every \(\varepsilon >0\) we have
\[
\mathbb {P}\left(\left\| \xi ^{(n)} - \xi \right\| _{2} \geq \varepsilon \right) \xrightarrow [n \to \infty ]{} 0,
\]
where
\[
\left\| \overrightarrow {x}\right\| _{p} := \sqrt[p]{\left|x_1\right|^p + \ldots + \left|x_m\right|^p}
\]
for \(\overrightarrow {x} = (x_1, \ldots , x_m) \in \mathbb {R}^{m}\).
Note that instead of the norm \(\left\| \cdot \right\|_{2}\) one could use any other norm \(\left\| \cdot \right\|_{p}, \; p \geq 1\) and even \(\left\| \overrightarrow {x}\right\|_{\infty } = \max \left\{ \left|x_i\right|, \; i = 1, \ldots , m\right\}\), since in finite-dimensional spaces all norms are equivalent.
The sequence \(\xi^{(n)}\) converges in the Lebesgue space \(L^{p}\) to a random vector \(\xi\) as \(n \rightarrow \infty\) (written \(\xi^{(n)} \xrightarrow [n \to \infty ]{L^{p}} \xi\)) if
\[
\mathbb {E}\left[\left\| \xi ^{(n)} - \xi \right\| _{p}^p\right] \xrightarrow [n \to \infty ]{} 0,
\]
The sequence \(\xi^{(n)}\) converges in distribution to a random vector \(\xi\) as \(n \rightarrow \infty\) (written \(\xi^{(n)} \xrightarrow [n \to \infty ]{d} \xi\)) if for every bounded continuous function \(f: \mathbb {R}^{m} \to \mathbb {R}\) we have
\[
\mathbb {E}\left[f(\xi ^{(n)})\right] \xrightarrow [n \to \infty ]{} \mathbb {E}\left[f(\xi )\right]
\]
An equivalent definition: \(\xi^{(n)} \xrightarrow [n \to \infty ]{d} \xi\) if the distribution functions of \(\xi^{(n)}\)
\[
F_{\xi ^{(n)}}(\overrightarrow {x}) = \mathbb {P}\left(\xi ^{(n)}_1 \leq x_1, \xi ^{(n)}_2 \leq x_2, \ldots , \xi ^{(n)}_m \leq x_m\right)
\]
converge to the distribution function of \(\xi\) at every continuity point of the latter:
\[
F_{\xi ^{(n)}}(\overrightarrow {x}) \xrightarrow [n \to \infty ]{} F_{\xi }(\overrightarrow {x}), \quad \forall \overrightarrow {x} \text{ such that } F_{\xi } \text{ is continuous at } \overrightarrow {x}.
\]
Theorem 1 (On relations among various modes of convergence) The following implications among modes of convergence hold
The implication \(L^q \Rightarrow L^p\) for \(q \geq p \geq 1\) is Lyapunovβs inequality: \(\left(\mathbb {E}\left[\left|\cdot \right|^p\right]\right)^{1/p} \leq \left(\mathbb {E}\left[\left|\cdot \right|^q\right]\right)^{1/q}\), applied to \(\left\| \xi^{(n)} - \xi \right\|_{p}\).
Theorem 2 (On relations between convergence of vectors and convergence of their components) Let random vectors \[
\xi = \begin{pmatrix} \xi _1 \\ \vdots \\ \xi _m \end{pmatrix}, \qquad \xi ^{(n)} = \begin{pmatrix} \xi ^{(n)}_1 \\ \vdots \\ \xi ^{(n)}_m \end{pmatrix}
\] be given, where \(n \in \mathbb {N}\). Then for every \(i = 1,\ldots ,m\) we have \[
\begin{align} \xi ^{(n)} \xrightarrow [n \to \infty ]{\text{a.s.}} \xi & \iff \xi ^{(n)}_i \xrightarrow [n \to \infty ]{\text{a.s.}} \xi _i, \\ \xi ^{(n)} \xrightarrow [n \to \infty ]{\mathbb {P}} \xi & \iff \xi ^{(n)}_i \xrightarrow [n \to \infty ]{\mathbb {P}} \xi _i, \\ \xi ^{(n)} \xrightarrow [n \to \infty ]{L^{p}} \xi & \iff \xi ^{(n)}_i \xrightarrow [n \to \infty ]{L^{p}} \xi _i, \\ \xi ^{(n)} \xrightarrow [n \to \infty ]{d} \xi & \Longrightarrow \xi ^{(n)}_i \xrightarrow [n \to \infty ]{d} \xi _i. \end{align}
\]
Note that for convergence in distribution the reverse-direction theorem does not hold. Convergence in distribution of random vectors implies componentwise convergence, but the converse is not always true, as the following example shows. Let \(\xi\) and \(\eta\) be independent identically distributed random variables. Let the sequences of random variables \(\left\{ \xi^{(n)}\right\}_{n \in \mathbb {N}}\) and \(\left\{ \eta^{(n)}\right\}_{n \in \mathbb {N}}\) be such that \(\xi^{(n)}=\xi\) and \(\eta^{(n)}=\eta\) for every \(n \in \mathbb {N}\). Then, obviously, \(\xi^{(n)} \stackrel{d}{\rightarrow } \xi\) and \(\eta^{(n)} \stackrel{d}{\rightarrow } \xi\), since \(\xi \stackrel{d}{=} \eta\) by assumption. But vector convergence does not hold: \(\left(\xi^{(n)}, \eta^{(n)}\right) \stackrel{d}{\not\to }(\xi , \xi )\), since \(\left(\xi^{(n)}, \eta^{(n)}\right) \stackrel{d}{\rightarrow }(\xi , \eta )\) and the distributions of the vectors \((\xi , \eta )\) and \((\xi , \xi )\) do not coincide, since all values of the second vector lie almost surely on the line \(y=x\), whereas for the first vector this is far from always the case.
However, if we have 2 componentwise-convergent-in-distribution sequences and the limit of one of them is a constant, then the reverse implication in theorem Theorem 2 does hold, as the following lemma states.
Lemma 1 Let \(\eta^{(1)}, \eta^{(2)}, \ldots\) be a sequence of random vectors of dimension \(k\), converging in distribution to \(\eta\): \[
\eta ^{(n)} \xrightarrow [n \to \infty ]{d} \eta
\] Let \(\zeta^{(1)}, \zeta^{(2)}, \ldots\) be a sequence of random vectors of dimension \(l\), converging in distribution to some constant vector \(c = (c_1, \ldots , c_{l})^T \in \mathbb {R}^{l}\): \[
\zeta ^{(n)} \xrightarrow [n \to \infty ]{d} c.
\] Then the random vector \(\begin{pmatrix} \eta^{(n)} \\ \zeta^{(n)} \end{pmatrix}\) of dimension \(k + l\) converges in distribution to \(\begin{pmatrix} \eta \\ c \end{pmatrix}\): \[
\begin{pmatrix} \eta ^{(n)} \\ \zeta ^{(n)} \end{pmatrix} \xrightarrow [n \to \infty ]{d} \begin{pmatrix} \eta \\ c \end{pmatrix}.
\]
The distribution function of the limiting vector \((\eta , c)\) has the form
\[
F_{(\eta ,c)}(x,y) = \mathbb {P}\left(\eta \leq x, \, c \leq y\right) = F_\eta (x) \cdot \; \mathbb {1}_{y \geq c},
\]
since \(c\) is a nonrandom vector. By the definition of convergence in distribution via distribution functions, it suffices to check the convergence \(F_{(\eta^{(n)}, \zeta^{(n)})}(x,y) \to F_{(\eta ,c)}(x,y)\) at every continuity point of \(F_{(\eta ,c)}\) β that is, at points \((x,y)\) where \(x\) is a continuity point of \(F_\eta\) and \(y\) does not lie on any of the hyperplanes \(\left\{ y_i = c_i\right\}\), \(i = 1,\ldots ,l\). Fix such a point \((x,y)\).
Case \(y > c\) (strictly, componentwise). From the inclusions \(\left\{ \eta^{(n)} \leq x, \, \zeta^{(n)} \leq y\right\} \subset \left\{ \eta^{(n)} \leq x\right\}\) and \(\left\{ \eta^{(n)} \leq x\right\} \setminus \left\{ \zeta^{(n)} \leq y\right\} \subset \left\{ \zeta^{(n)} \not\leq y\right\}\) we obtain the bound
\[
\mathbb {P}\left(\eta ^{(n)} \leq x\right) - \mathbb {P}\left(\zeta ^{(n)} \not\leq y\right) \; \leq \; F_{(\eta ^{(n)}, \zeta ^{(n)})}(x,y) \; \leq \; \mathbb {P}\left(\eta ^{(n)} \leq x\right).
\]
As \(n \to \infty\) both outer terms tend to \(F_\eta (x)\): the left one because \(x\) is a continuity point of \(F_\eta\), and \(\mathbb {P}\left(\zeta^{(n)} \not\leq y\right) \to \mathbb {P}\left(c \not\leq y\right) = 0\) (since \(y > c\)); the right one for the same reason. By the squeeze theorem, \(F_{(\eta^{(n)}, \zeta^{(n)})}(x,y) \to F_\eta (x) = F_{(\eta ,c)}(x,y)\).
Case \(y_i < c_i\) for some \(i\). From \(\left\{ \zeta^{(n)} \leq y\right\} \subset \left\{ \zeta^{(n)}_i \leq y_i\right\}\) we obtain
\[
0 \; \leq \; F_{(\eta ^{(n)}, \zeta ^{(n)})}(x,y) \; \leq \; \mathbb {P}\left(\zeta ^{(n)}_i \leq y_i\right) \xrightarrow [n \to \infty ]{} \mathbb {P}\left(c_i \leq y_i\right) = 0,
\]
since \(y_i < c_i\). Hence, \(F_{(\eta^{(n)}, \zeta^{(n)})}(x,y) \to 0 = F_{(\eta ,c)}(x,y)\).
These two cases exhaust all continuity points of \(F_{(\eta ,c)}\) (at the remaining points either \(y > c\), or \(y_i < c_i\) for at least one \(i\)), which completes the proof.
Prove that convergence in distribution of a sequence of \(m\)-dimensional random vectors \(\xi^{(n)}\) to a constant \(C \in \mathbb {R}^{m}\) implies convergence \(\xi^{(n)} \stackrel{\mathbb {P}}{\rightarrow } C\).
From theorem Theorem 2 it follows that for every \(i \in \{ 1, \ldots , m\}\) we have \(\xi^{(n)}_{i} \stackrel{d}{\rightarrow } C_i\), where \(\xi^{(n)}=\left(\xi^{(n)}_{1}, \ldots , \xi^{(n)}_{m}\right)^T, C=\left(C_{1}, \ldots , C_{m}\right)^T\). We know from the probability course that the latter convergence implies convergence in probability: \(\xi^{(n)}_{i} \stackrel{\mathbb {P}}{\rightarrow } C_i\) for every \(i \in \{ 1, \ldots , m\}\). But then from theorem Theorem 2 we have \(\xi^{(n)} \stackrel{\mathbb {P}}{\rightarrow } C\).
Inheritance of Convergence
The next important result we will need is the theorem on inheritance of convergence.
Theorem 3 (on inheritance of convergence / Mann-Wald / the continuous mapping theorem) Let \(\xi , \xi^{(1)}, \xi^{(2)}, \ldots\) be random vectors of dimension \(m\), and let \(h: \mathbb {R}^{m} \to \mathbb {R}^{s}\) be a function that is continuous almost everywhere with respect to the distribution of the random variable \(\xi\), i.e., there exists a set \(B \in \mathscr {B}\left(\mathbb {R}^{m}\right), \; \mathbb {P}\left(\xi \in B\right) = 1\) such that \(h\) is continuous on \(B\). Then \[
\begin{align} \xi ^{(n)} \xrightarrow [n \to \infty ]{\text{a.s.}} \xi & \Longrightarrow h(\xi ^{(n)}) \xrightarrow [n \to \infty ]{\text{a.s.}} h(\xi ) \\ \xi ^{(n)} \xrightarrow [n \to \infty ]{\mathbb {P}} \xi & \Longrightarrow h(\xi ^{(n)}) \xrightarrow [n \to \infty ]{\mathbb {P}} h(\xi ) \\ \xi ^{(n)} \xrightarrow [n \to \infty ]{d} \xi & \Longrightarrow h(\xi ^{(n)}) \xrightarrow [n \to \infty ]{d} h(\xi ) \end{align}
\]
Corollary 1 Suppose we have a parametric model \(\mathcal{P} = \left\{ \mathbb {P}_\theta , \theta \in \Theta \right\}\) and a sample \(X_1, X_2, \ldots\) from it. Let \(\widehat{\theta }\) be a consistent (strongly consistent) estimator of \(\theta\). Let \(h: \Theta \to h(\Theta )\) be some map that is continuous on \(\Theta\). Then \(h(\widehat{\theta })\) is a consistent (respectively, strongly consistent) estimator of \(h(\theta )\).
Convergence in the Lebesgue space \(L^p\) is inherited in a somewhat more subtle way, if only because \(h(\xi )\) does not always lie in \(L^p\), even if \(\xi\) is in \(L^p\) and \(h\) is everywhere continuous. However, if \(h: \mathbb {R}^{m} \to \mathbb {R}^{s}\) is a Lipschitz function:
\[
\exists C > 0: \; \left\| h(\overrightarrow {x}) - h(\overrightarrow {y})\right\| _{q} \leq C \cdot \left\| \overrightarrow {x} - \overrightarrow {y}\right\| _{p}
\]
\(\forall \overrightarrow {x}, \overrightarrow {y} \in \mathbb {R}^{m}\), then convergence of \(\xi^{(n)}\) to \(\xi\) in \(L^p\) implies convergence in \(L^q\) for \(h(\xi^{(n)})\):
\[
\mathbb {E}\left[\left\| h(\xi ^{(n)}) - h(\xi )\right\| _{q}\right] \leq C \mathbb {E}\left[\left\| \xi ^{(n)} - \xi \right\| _{p}\right] \xrightarrow [n \to \infty ]{} 0
\]
Convergence and Arithmetic Operations
By theorems Theorem 2 and Theorem 3, from the convergence of sequences of random variables \(\xi^{(n)} \stackrel{\mathbb {P}}{\rightarrow } \xi\), \(\eta^{(n)} \stackrel{\mathbb {P}}{\rightarrow } \eta\) we obtain the convergences
\[
\xi ^{(n)}+\eta ^{(n)} \stackrel{\mathbb {P}}{\rightarrow } \xi +\eta , \quad \xi ^{(n)} \eta ^{(n)} \stackrel{\mathbb {P}}{\rightarrow } \xi \eta .
\]
For convergence in distribution, a similar conclusion cannot be drawn, since, as we already know, componentwise convergence in distribution does not imply convergence in distribution of vectors. For example, let \(\xi \sim \mathscr {N}\left(0, 1\right)\) be an arbitrary standard normal random variable, \(\xi_n := \xi \; \forall n \in \mathbb {N}\). Then we can write
\[
\xi _n \xrightarrow [n\to \infty ]{d}\xi , \quad \xi _n \xrightarrow [n\to \infty ]{d} -\xi ,
\]
since \(\left(-\xi \right) \sim \mathscr {N}\left(0, 1\right)\). But
\[
\xi _n + \xi _n = 2\xi _n \stackrel{d}{\not\to } \xi - \xi = 0,
\]
since \(2\xi_n = 2\xi \sim \mathscr {N}\left(0, 4\right)\).
However, if one of the sequences converges in distribution to a constant, then the following statement holds.
Theorem 4 (Slutskyβs theorem) Let \(\xi_n \xrightarrow [n \to \infty ]{d} \xi\) and \(\eta_n \xrightarrow {d} c \in \mathbb {R}\) be random variables. Then \[
\begin{align} \xi _n + \eta _n \xrightarrow [n \to \infty ]{d} \xi + c \\ \xi _n \eta _n \xrightarrow [n \to \infty ]{d} \xi c \end{align}
\]
Prove Slutskyβs theorem.
Use lemma Lemma 1 and theorem Theorem 3 on inheritance of convergence in distribution.
Inheritance of Asymptotic Normality, One-Dimensional Case
We are ready to move on directly to the question of inheritance of asymptotic normality. A consequence of theorems Theorem 3 and Theorem 4 is
Theorem 5 (The Delta Method) Let \(\xi^{(n)} \stackrel{d}{\rightarrow } \xi\) be random vectors of dimension \(m\), and let \(h(x): \mathbb {R}^{m} \to \mathbb {R}\) be a function differentiable at the point \(\theta \in \mathbb {R}^{m}\). Let \(b_{n} \rightarrow 0\), \(b_{n} \neq 0\). Then
\[
\frac{h(\theta + \xi ^{(n)}b_n) - h(\theta )}{b_n} \xrightarrow [n \to \infty ]{d} \left\langle \xi , \nabla h|_{\theta } \right\rangle ,
\]
where \(\nabla h|_{\theta }\) is the gradient of the function \(h(x)\), taken at the point \(\theta\), and the right-hand side is a random variable (the dot product of the vector \(\xi\) with the vector \(\nabla h|_{\theta }\)).
In particular, if \(\xi^{(n)} \stackrel{d}{\rightarrow } \xi\) are random variables (\(m=1\)), and \(h(x): \mathbb {R} \to \mathbb {R}\) is a function differentiable at the point \(\theta \in \mathbb {R}\), then
\[
\frac{h(\theta + \xi ^{(n)}b_n) - h(\theta )}{b_n} \; \xrightarrow [n \to \infty ]{d} \; h'(\theta ) \cdot \xi .
\]
Theorem 6 (on inheritance of asymptotic normality) Let \(\widehat{\theta } = \widehat{\theta }(X)\) be an asymptotically normal estimator of \(\theta\) with asymptotic variance \(\sigma^2(\theta )\), and let \(h(\theta )\) be a differentiable function on \(\Theta \subset \mathbb {R}\), whose derivative is finite and nonzero at every point \(\theta \in \Theta\). Then \(h(\widehat{\theta })\) is an asymptotically normal estimator of \(h(\theta )\) with asymptotic variance \(\sigma^{2}(\theta ) \cdot \left(h^{\prime }(\theta )\right)^{2}\).
We are ready to move on to answering the question about the asymptotic normality of \(\widehat{\sigma }_1\).
Let \(X_{1}, \ldots , X_{n}\) be a sample from \(\mathscr {N}\left(0, \sigma^2\right), \sigma > 0\). Find for which parameter the estimator \(\widehat{\sigma }_1 = \sqrt{\overline{X^2}}\) is asymptotically normal, and find the asymptotic variance.
Let us apply the CLT to \(X_n^2, \; n \in \mathbb {N}\). Recall that \(\mathbb {E}\left[X_1^2\right] = \sigma^2, \; \operatorname {Var}\left[X_1^2\right] = \mathbb {E}\left[X_1^4\right] - \left(\mathbb {E}\left[X_1^2\right]\right)^2 = 3\sigma^4 - \sigma^4 = 2\sigma^4\). Let us apply the CLT:
\[
\sqrt{n}\left(\overline{X^2} - \sigma ^2\right) \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, 2\sigma ^4\right)
\]
Thus, \(\overline{X^2}\) is an asymptotically normal estimator of \(\sigma^2\) with asymptotic variance \(2\sigma^4\).
Let us use the theorem on inheritance of asymptotic normality. We have
\[
\begin{align} h\left(\theta \right) & = \sqrt{\theta }, \quad h'(\theta ) = \frac{1}{2\sqrt{\theta }}, \quad h'(\sigma ^2) = \frac{1}{2\sigma } \end{align}
\]
\(h\) is differentiable on \((0, +\infty )\), and its derivative does not vanish. Consequently, \(\sqrt{\overline{X^2}}\) is an asymptotically normal estimator of \(\sigma\) with asymptotic variance \(2\sigma^4 \cdot \left(\frac{1}{2\sigma }\right)^2 = \frac{1}{2}\sigma^2\):
\[
\sqrt{n}\left(\sqrt{\overline{X^2}} - \sigma \right) \xrightarrow [n\to \infty ]{d} \mathscr {N}\left(0, \frac{1}{2}\sigma ^2\right)
\]
Parameter \(\sigma\). Asymptotic variance \(\frac{1}{2}\sigma^2\).
Let us consider a few more examples.
Let \(X_1, \ldots , X_n\) be a sample from a distribution with nonzero expectation \(a\) and variance \(\sigma^{2}\). Is the statistic \(1/\overline{X}\) an asymptotically normal estimator of some parameter? If so, what is the asymptotic variance?
By the central limit theorem, \(\overline{X}\) is an asymptotically normal estimator of \(a\) with asymptotic variance \(\sigma^2\):
\[
\sqrt{n} \left(\overline{X} - a\right) \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, \sigma ^2\right)
\]
Next, let us use the theorem on inheritance of asymptotic normality. We have \(h(y) = \frac{1}{y}\). This function is differentiable on \(\mathbb {R} \setminus \left\{ 0\right\}\). Here \(h'(y) = -\frac{1}{y^2}\). Consequently, \(\frac{1}{\overline{X}}\) is an asymptotically normal estimator of \(\frac{1}{a}\) with asymptotic variance \(\frac{\sigma^2}{a^4}\):
\[
\sqrt{n}\left(\frac{1}{\overline{X}} - \frac{1}{a}\right) \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, \frac{\sigma ^2}{a^4}\right)
\]
Let \(X_{1}, \ldots , X_{n}\) be a sample from \(U[1+\theta , \theta (1+\theta )]\), the uniform distribution on the interval \([1+\theta , \theta (1+\theta )], \theta >1\). Find an asymptotically normal estimator of the parameter \(\theta\) and its asymptotic variance.
Note that \(\mathbb {E}\left[X_1\right] = \frac{(1+\theta )^2}{2} =: m(\theta )\). By the central limit theorem, \(\overline{X}\) is an asymptotically normal estimator of the parameter \(m(\theta )\) with asymptotic variance \(\sigma^{2}(\theta )=\frac{\left(\theta^{2}-1\right)^{2}}{12}\). Note that \(m(\theta )\) takes values in the set \((2, \infty )\). Set \(h(x)=\sqrt{2 x}-1\) (a function differentiable on \((2, \infty )\)). Then \(h'(x) = \frac{1}{\sqrt{2x}}\), whence \(\left(h'(x)\right)^2 = \frac{1}{2x}\), and, substituting \(x = m(\theta ) = \frac{(1+\theta )^2}{2}\):
\[
\left(\left.h^{\prime }\right|_{m(\theta )}\right)^{2} = \frac{1}{2m(\theta )} = \frac{1}{(1+\theta )^2}.
\]
By the theorem on inheritance of asymptotic normality, \(h(\overline{X})\) is an asymptotically normal estimator of \(\theta\) with asymptotic variance
\[
\sigma ^{2}(\theta )\left(\left.h^{\prime }\right|_{m(\theta )}\right)^{2}=\frac{\left(\theta ^{2}-1\right)^{2}}{12} \cdot \frac{1}{(1+\theta )^2} = \frac{(\theta -1)^2(\theta +1)^2}{12(1+\theta )^2} = \frac{(\theta -1)^{2}}{12}
\]
Estimator \(\sqrt{2\overline{X}} - 1\), asymptotic variance \(\frac{(\theta - 1)^2}{12}.\)
Let \(X_1, \dots , X_n\) be a sample from \(\operatorname {Exp}\left(\alpha \right), \alpha > 0\). Find constants \(a(\alpha ), \sigma^2(\alpha )\) such that \[
\begin{align} \sqrt{n} (\overline{X} \sin (\overline{X}) - a(\alpha )) \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, \sigma ^2(\alpha )\right) \end{align}
\]
A sample \(X_{1}, \ldots , X_{n}\) is given from the exponential law with parameter \(\lambda >0\). For which quantity \(h(\lambda )\) is the statistic \(\bar{X} \ln \bar{X}\) an asymptotically normal estimator? Compute the asymptotic variance of this estimator.
Let \(X_{1}, \ldots , X_{n}\) be a sample from the exponential distribution with parameter \(\theta\). Show that for every \(k \in \mathbb {N}\) the statistic \(\sqrt[k]{k ! / \overline{X^{k}}}\) is an asymptotically normal estimator of the parameter \(\theta\). Find its asymptotic variance.
All the estimators constructed above were asymptotically normal at rate \(\sqrt{n}\). The following problem shows that this is far from always true: for the sample maximum \(X_{(n)}\), both the rate of convergence and the limiting distribution turn out to be quite different, so the delta method in its current form is not applicable here.
Let \(X_{1}, \ldots , X_{n}\) be a sample from the distribution \(U[0, \theta ]\) (the uniform distribution on the interval \([0,\theta ]\)). Find a number \(\delta >0\) and a nondegenerate distribution law \(\mathbb {P}_{\theta }\) (i.e., not the distribution law of a constant) such that \(n^{\delta }\left(X_{(n)}-\theta \right) \stackrel{d}{\longrightarrow } \xi \sim \mathbb {P}_\theta\) as \(n \rightarrow \infty\).
Construct an asymptotically normal estimator of \(b\) from a sample \(X_{1}, \ldots , X_{n}\) from the distribution \[
\begin{align} \mathbb {P}\left(X_1 = 1\right) & = 1-a-a^{2}-6b, \\ \mathbb {P}\left(X_1 = 2\right) & = 5 b, \\ \mathbb {P}\left(X_1 = 5\right) & = a+a^{2}+b, \end{align}
\] where \(a, b\) are unknown positive parameters.
Choose a function \(g\) such that \(\mathbb {E}\left[g(X_1)\right] = b\) for all \(a, b\); for example, a quadratic polynomial with roots \(1\) and \(5\).
A sample \(X_{1}, \ldots , X_{n}\) is given, \(X_{1}=\xi +\eta , \xi \sim U(0, \theta ), \eta \sim\) \(U(\theta , 2 \theta )\) β independent random variables, where \(\theta >0\) is an unknown parameter. Find an asymptotically normal estimator of \(\theta\) and its asymptotic variance.
Let us consider an example of applying the delta method that turns out to be useful in machine learning.
Let \(X_{1}, \ldots , X_{n}\) be a sample from \(U[0,1]\), that is, \((X_1, \ldots , X_n)^T\) is a random point from the cube \([0,1]^n\) with the uniform distribution. Consider the statistic \[
N = N_n := \sqrt{X_1^2 + \ldots + X_n^2}
\] β the \(2\)-norm of the vector, i.e., the Euclidean distance from zero to this random point. Find the distribution of \(N_n\) for large \(n\).
We have
\[
\begin{align} \mathbb {E}\left[X_1^2\right] & = \int _0^1 x^2\, dx = \frac{1}{3}, \\ \mathbb {E}\left[X_1^4\right] & = \int _0^1 x^4\, dx = \frac{1}{5}, \\ \operatorname {Var}\left[X_1^2\right] & = \mathbb {E}\left[X_1^4\right] - \mathbb {E}\left[X_1^2\right]^2 = \frac{1}{5} - \frac{1}{9} = \frac{4}{45}. \end{align}
\]
By the central limit theorem, applied to the quantities \(X_1^2, \ldots , X_n^2\),
\[
\sqrt{n}\left(\frac{N_n^2}{n} - \frac{1}{3}\right) \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, \frac{4}{45}\right).
\]
Now let us apply the delta method to the function \(h(\theta ) = \sqrt{\theta }\) at the point \(\theta = \frac{1}{3}\), since \(h'(\theta ) = \frac{1}{2\sqrt\theta }\). Let us compute the asymptotic variance: \(h'\left(\frac13\right) = \frac{1}{2\sqrt{1/3}} = \frac{\sqrt3}{2}\), whence \(\left(h'\left(\frac13\right)\right)^2 = \frac34\), and
\[
\frac{4}{45} \cdot \left(h'\left(\tfrac 13\right)\right)^2 = \frac{4}{45}\cdot \frac34 = \frac{1}{15}.
\]
Consequently,
\[
\sqrt{n}\left(\frac{N_n}{\sqrt{n}} - \sqrt{\frac{1}{3}}\right) \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, \frac{1}{15}\right).
\]
Note that on the left, after multiplying by \(\sqrt{n}\), we get the expression \(N_n - \sqrt{n/3}\). That is, approximately (for large \(n\))
\[
\operatorname {Law}(N_n) \approx \mathscr {N}\left(\sqrt{n/3}, \frac{1}{15}\right),
\]
in other words, \(N_n\) is approximately normally distributed with expectation \(\sqrt{n/3}\) and standard deviation \(\sqrt{1/15} \approx 0.26\).
This yields a curious geometric picture: a random point in the hypercube \([0,1]^n\), for large \(n\), lies practically on the surface of a sphere centered at zero with radius \(\sqrt{n/3}\). Meanwhile, the standard deviation of the distance is almost independent of \(n\) and is about \(0.26\). For large \(n\), this deviation turns out to be exceptionally small compared to the mean distance \(\sqrt{n/3}\), which grows without bound.
Conclusion for machine learning. In high-dimensional spaces, almost all points of a sample turn out to lie at comparable, practically identical distances from each other and from the origin. This phenomenon is one facet of the curse of dimensionality: methods based on the notion of distance (for example, the \(k\)-nearest-neighbors method or clustering by the Euclidean metric) lose their meaning, since distances between points cease to be informative β βcloseβ and βfarβ points are practically indistinguishable from one another.
Let \((X_1, \ldots , X_n)^T\) and \((Y_1, \ldots , Y_n)^T\) be 2 independent random points from the hypercube \([0,1]^n\) with the uniform distribution. Consider the statistic \[
D = D_n := \sqrt{(X_1 - Y_1)^2 + \ldots + (X_n- Y_n)^2}
\] β the Euclidean distance between them. Find the distribution of \(D_n\) for large \(n\).
Limit Theorems for Random Vectors
For random vectors, limit theorems analogous to the corresponding theorems for random variables continue to hold: the law of large numbers, the strong law of large numbers, the central limit theorem. Let us state them. Going forward, for a random vector \(\xi\) we will write \(\mathbb {E}\left[\xi \right]\), meaning the vector made up of the expectations of the components of the vector \(\xi\).
Theorem 7 (the multivariate analogue of the law of large numbers) Let \(\left\{ \xi^{(n)}\right\}_{n \in \mathbb {N}}-\) be independent random vectors, with the variance of each component of each random vector bounded by a constant \(C\). Then \[
\frac{\xi ^{(1)}+\ldots +\xi ^{(n)}-\mathbb {E}\left[\xi ^{(1)}+\ldots +\xi ^{(n)}\right]}{n^{0.5+\delta }} \xrightarrow [n \to \infty ]{\mathbb {P}} 0,
\] where \(\delta\) is an arbitrary positive number.
Theorem 8 (the multivariate analogue of the strong law of large numbers) Let \(\left\{ \xi^{(n)}\right\}_{n \in \mathbb {N}}-\) be independent identically distributed random vectors, with the expectations of all components finite and equal to \(\mathbb {E}\left[\xi^{(1)}\right] = a\). Then \[
\frac{\xi ^{(1)}+\ldots +\xi ^{(n)}}{n} \xrightarrow [n \to \infty ]{\text{a.s.}} a
\]
Theorem 9 (the multivariate analogue of the central limit theorem) Let \(\left\{ \xi^{(n)}\right\}_{n \in \mathbb {N}}-\) be independent identically distributed random vectors, \(\mathbb {E}\left[\xi^{(1)}\right] = a\), and let the covariance matrix of the random vector \(\xi^{(1)}\) be \(\operatorname {Var}\left[\xi^{(1)}\right] = \Sigma\). Then \[
\sqrt{n}\left(\frac{\xi ^{(1)}+\ldots +\xi ^{(n)}}{n}-a\right) \stackrel{d}{\rightarrow } \eta \sim \mathcal{N}(0, \Sigma ),
\] where \(\mathcal{N}(0, \Sigma )\) is the multivariate normal distribution (a Gaussian vector).
Inheritance of Asymptotic Normality, Multivariate Case
We are now almost ready to answer the question about the asymptotic normality of the estimator \(\widehat{\sigma }_2\) introduced at the beginning of the chapter. The last step is to state the multivariate version of the theorem on inheritance of asymptotic normality.
Theorem 10 (on inheritance of asymptotic normality, multivariate version) Let \(\widehat{\theta }(X)\) be an asymptotically normal estimator of a \(k\)-dimensional parameter \(\theta \in \Theta \subset \mathbb {R}^{k}\) with asymptotic covariance matrix \(\Sigma (\theta )\), and let \(h: \Theta \to \mathbb {R}\) be a differentiable function on \(\Theta \subset \mathbb {R}^{k}\) (the gradient \(\nabla h\) is finite and nonzero at every point \(\theta \in \Theta )\). Then \(h\left(\widehat{\theta }(X)\right)\) is an asymptotically normal estimator of \(h(\theta )\) with asymptotic variance \[
\left(\nabla h|_{\theta }\right)^T \cdot \Sigma \cdot \nabla h|_{\theta }
\]
The proof is analogous to the one-dimensional case.
Let \(X_{1}, \ldots , X_{n}\) be a sample from \(\mathscr {N}\left(0, \sigma^2\right)\). Find for which parameter the estimator \(\widehat{\sigma }_2 = \sqrt{\frac{\overline{X^4}}{3\overline{X^2}}}\) is asymptotically normal, and find the asymptotic variance.
Let us apply the multivariate CLT to \(\xi^{(n)} := (X_n^2, X_n^4)^T, \; n \in \mathbb {N}\). Let us find the expectation and covariance matrix of \(\xi^{(n)}\):
\[
\begin{align} \mathbb {E}\left[X_1^2\right] & = \sigma ^2, \quad \mathbb {E}\left[X_1^4\right] = 3\sigma ^4, \quad \mathbb {E}\left[X_1^6\right] = 15\sigma ^6, \quad \mathbb {E}\left[X_1^8\right] = 105\sigma ^8\\ \operatorname {Var}\left[X_1^2\right] & = \mathbb {E}\left[X_1^4\right] - \left(\mathbb {E}\left[X_1^2\right]\right)^2 = 2\sigma ^4, \quad \operatorname {Var}\left[X_1^4\right] = \mathbb {E}\left[X_1^8\right] - \left(\mathbb {E}\left[X_1^4\right]\right)^2 = 96\sigma ^8, \\ \operatorname {Cov}\left[ X_1^2, X_1^4 \right] & = \mathbb {E}\left[X_1^6\right] - \mathbb {E}\left[X_1^2\right]\mathbb {E}\left[X_1^4\right] = 12 \sigma ^6\\ \mathbb {E}\left[\begin{pmatrix} X_1^2 \\ X_1^4 \end{pmatrix}\right] & = \begin{pmatrix} \sigma ^2 \\ 3\sigma ^4 \end{pmatrix}, \qquad \operatorname {Var}\left[\begin{pmatrix} X_1^2 \\ X_1^4 \end{pmatrix}\right] = \Sigma = \begin{pmatrix} 2 \sigma ^{4} & 12 \sigma ^{6} \\ 12 \sigma ^{6} & 96 \sigma ^{8} \end{pmatrix} \end{align}
\]
Let us apply the CLT:
\[
\sqrt{n}\left(\begin{pmatrix} \overline{X^2} \\ \overline{X^4} \end{pmatrix} - \begin{pmatrix} \sigma ^2 \\ 3\sigma ^4 \end{pmatrix}\right) \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, \Sigma \right)
\]
Thus, \(\begin{pmatrix} \overline{X^2} \\ \overline{X^4} \end{pmatrix}\) is an asymptotically normal estimator of \(\theta := \begin{pmatrix} \sigma^2 \\ 3\sigma^4 \end{pmatrix}\) with asymptotic covariance matrix \(\Sigma\).
Let us use the multivariate version of the theorem on inheritance of asymptotic normality. We have
\[
\begin{align} h\left((x,y)^T\right) & = \sqrt{\frac{y}{3x}}, \quad \nabla h = \left(-\frac{1}{2}\sqrt{\frac{y}{3x^3}}, \frac{1}{2}\sqrt{\frac{1}{3xy}}\right)^T, \quad \nabla h|_{\theta } = \left(-\frac{1}{2\sigma }, \frac{1}{6\sigma ^3}\right)^T \end{align}
\]
Let us separately compute \(\left(\nabla h|_{\theta }\right)^T \cdot \Sigma\):
\[
\begin{align} \left(\nabla h|_{\theta }\right)^T \cdot \Sigma & = \begin{pmatrix} -\dfrac {1}{2\sigma } & \dfrac {1}{6\sigma ^3} \end{pmatrix} \begin{pmatrix} 2\sigma ^4 & 12\sigma ^6 \\ 12 \sigma ^6 & 96\sigma ^8\end{pmatrix} \\ & = \begin{pmatrix} -\dfrac {1}{2\sigma }\cdot 2\sigma ^4 + \dfrac {1}{6\sigma ^3}\cdot 12\sigma ^6 & -\dfrac {1}{2\sigma }\cdot 12\sigma ^6 + \dfrac {1}{6\sigma ^3}\cdot 96\sigma ^8 \end{pmatrix} = \begin{pmatrix} \sigma ^3 & 10\sigma ^5 \end{pmatrix} \end{align}
\]
Then
\[
\begin{align} \left(\nabla h|_{\theta } \right)^T \cdot \Sigma \cdot \nabla h|_{\theta } & = \begin{pmatrix} \sigma ^3 & 10 \sigma ^5 \end{pmatrix}\begin{pmatrix} -1/2\sigma \\ 1/6\sigma ^3 \end{pmatrix} = -\frac{1}{2}\sigma ^2 + \frac{10}{6}\sigma ^2 = \left(\frac{10}{6} - \frac{1}{2}\right) \sigma ^2 = \frac{7}{6}\sigma ^2 \end{align}
\]
Consequently, \(\sqrt{\frac{\overline{X^4}}{3\overline{X^2}}}\) is an asymptotically normal estimator of \(h(\theta ) = \sigma\) with asymptotic variance \(\frac{7}{6}\sigma^2\):
\[
\sqrt{n}\left(\sqrt{\frac{\overline{X^4}}{3\overline{X^2}}} - \sigma \right) \xrightarrow [n\to \infty ]{d} \mathscr {N}\left(0, \frac{7}{6}\sigma ^2\right)
\]
Parameter \(\sigma\). Asymptotic variance \(\frac{7}{6}\sigma^2\).
A collection of independent identically distributed random variables \(X_{1}, \ldots , X_{n}\) with distribution \(\mathcal{N}\left(0, \sigma^{2}\right)\) is given. Consider the statistics \(Y=\frac{1}{n} \sum_{i=1}^{n}\left|X_{i}\right|, Z=\frac{1}{n} \sum_{i=1}^{n} X_{i}^{2}\) and \(T=\sqrt{\frac{2}{\pi }} \frac{Z}{Y}\). Prove that the statistic \(T\) is an asymptotically normal estimator of \(\sigma\), and find its asymptotic variance.
If \(\xi \sim \mathscr {N}\left(0, \sigma^2\right)\), then \[
\mathbb {E}\left[\left|\xi \right|^p\right] = \sigma ^p (p-1)!!\begin{cases} \sqrt{\frac{2}{\pi }}, & p \text{ -- odd}\\ 1, & p \text{ -- even} \end{cases}
\]
Let \(X_{1}, \ldots , X_{n}\) be independent random variables having the Laplace distribution with parameter \(\sigma > 0\), i.e., with density \[
p(x)=\frac{1}{2 \sigma } e^{-\frac{|x|}{\sigma }} .
\] Consider \(Y=\frac{1}{n} \sum_{i=1}^{n}\left|X_{i}\right|, Z=\frac{1}{n} \sum_{i=1}^{n} X_{i}^{2}\). Prove that the statistic \(T := \frac{Z^2}{4Y^3}\) is an asymptotically normal estimator of \(\sigma\), and find its asymptotic variance.
A sample \(X_{1}, \ldots , X_{n}\) is given from the distribution \(U[0, \theta ]\), where \(\theta\) is an unknown parameter. Find an asymptotically normal estimator of \(\theta\) as a function of \(\overline{X^2}/\overline{X^3}\). Find its asymptotic variance.