Goodness-of-Fit Tests

WarningDraft

This text is still being edited: it may contain typos, inaccuracies, and locally unpolished passages. If you find something, please let the author know (contact information on the course page).

The next class of hypotheses that we consider in this text concerns whether an unknown distribution coincides with a known one. In other words, let \(X\) be an observation with unknown distribution \(\mathbb {P}\), whose distribution function equals \(F\). We pose the hypotheses

\[ H_0: \; \mathbb {P} = \mathbb {P}_0, \qquad H_1: \; \mathbb {P} \neq \mathbb {P}_0 \]

for some given distribution \(\mathbb {P}_0\). The alternative distributions are left entirely unspecified; a priori only the type of distribution (e.g. discrete or absolutely continuous) and the set of possible values of the observation \(X\) may be known. Tests for such hypotheses are called goodness-of-fit tests. The best known among them are the Kolmogorov test and the chi-squared test, which we discuss in this chapter. The first is based on comparing the theoretical and empirical distribution functions, and the second on comparing the theoretical and empirical densities.

For goodness-of-fit tests it is customary to use ones that have the property of consistency.

Definition 1 If \(X=\left(X_{1}, \ldots , X_{n}\right)\) is a sample, then a test \(S_{n}\) for the simple hypothesis \(H_{0}: \mathbb {P} = \mathbb {P}_0\) against \(H_1: \mathbb {P} \neq \mathbb {P}_0\) (strictly speaking, a sequence of tests) is called consistent if, for every \(\mathbb {P} \neq \mathbb {P}_0\), the power of the test tends to one: \[ \beta (\mathbb {P}, S_n) = \mathbb {P}\left(X \in S_n\right) \xrightarrow [n \to \infty ]{} 1, \quad \forall \mathbb {P} \neq \mathbb {P}_0 \] Equivalently: the probability of a type II error \(\mathbb {P}\left(X \not\in S_n\right)\) (\(\mathbb {P} \neq \mathbb {P}_0\)) tends to zero, whatever the alternative distribution \(\mathbb {P} \neq \mathbb {P}_0\) may be.

1 Comparing the theoretical and empirical distribution functions

Let \(X=\left(X_{1}, \ldots , X_{n}\right)\) be a sample from an unknown distribution on \(\mathbb {R}\) with continuous distribution function \(F\). We construct a test for the hypothesis \(H_{0}: \mathbb {P}=\mathbb {P}_{0}\). If \(F_0\) is the distribution function of \(\mathbb {P}_0\), then \(H_0\) can be equivalently restated as \(H_0: F = F_0\).

Definition 2 The empirical distribution is the uniform distribution on the set \(\left\{ X_1, \ldots , X_n\right\}\). That is, it is the distribution that concentrates mass \(\frac{1}{n}\) at each point \(X_i\):

\[ \mathbb {P}_n^* := \sum _{i=1}^n \frac{1}{n}\delta _{X_i}, \]

where \(\delta_{a}\) is the Dirac measure equal to unit mass at the point \(a\). Notation: \(\mathbb {P}_n^*\). We have

\[ \mathbb {P}_n^*(A) = \frac{1}{n}\sum _{i=1}^n \; \mathbb {1}_{X_i \in A} = \overline{\; \mathbb {1}_{X \in A}}, \; A \in \mathscr {B}\left(\mathbb {R}\right). \]

Note that \(\mathbb {P}_n^*\) is an estimator of the true distribution. It is random, like any estimator, since it depends on the sample.

The empirical distribution function of the sample \(X\) is the distribution function of the empirical distribution:

\[ F_{n}^{*}(x) := \mathbb {P}_n^*\left((-\infty , x]\right) = \frac{\sum _{i=1}^{n}\; \mathbb {1}_{X_i \leq x}}{n} = \overline{\; \mathbb {1}_{X \leq x}} \]

This is a nondecreasing step function that jumps by \(\frac{1}{n}\) at each sample element \(X_i\). It is also a random function: for each fixed \(x\), the value \(F^*(x)\) depends on the sample, i.e. it is a random variable.

ProblemProblem 1

Prove that, under the hypothesis \(H_{0}: F=F_{0}\), for every \(x \in \mathbb {R}\), \(F_n^*(x)\) is a strongly consistent estimator of \(F_0(x)\).

Theorem 1 (Glivenko-Cantelli) Suppose the hypothesis \(H_0: F = F_0\) holds, i.e. the true distribution is given by the distribution function \(F_0\). Then \(F_n^*\) converges a.s. to \(F_0\) in the supremum metric: \[ \sup _{x \in \mathbb {R}}\left|F_n^*(x) - F_{0}(x) \right| \xrightarrow [n \to \infty ]{\text{a.s.}} 0 \]

1.1 The Kolmogorov goodness-of-fit test

The Kolmogorov statistic is the distance between the functions \(F_n^*\) and \(F_0\) in the supremum metric:

\[ D_n := \sup _{x \in \mathbb {R}}\left|F_n^*(x) - F_{0}(x) \right|. \tag{1}\]

The theoretical distribution function \(F_0(x)\) and the empirical distribution function \(F_n^*(x)\) for a sample of size \(n=12\). The Kolmogorov statistic \(D_n=\sup_x|F_n^*(x)-F_0(x)|\) is marked on the graph.

ExampleExample 1

Prove that if \(H_0\) holds, then the distribution of \(D_n\) depends only on \(n\). In particular, it does not depend on \(F_0\).

SolutionSolution

If \(H_0\) holds, then with probability \(1\) every sample element falls in the set \(Q_0((0,1))\), where \(Q_0\) is the quantile function of the distribution \(P_0\). The whole sample falls in \(Q^n_0([0,1])\) with probability \(1\). Hence,

\[ D_n \stackrel{\text{a.s.}}{=} \sup _{x \in Q_0((0,1))}\left|F_n^*(x) - F_{0}(x) \right| \tag{2}\]

Next, apply to the sample \(X_1, \ldots, X_n\) the probability integral transform: \(Y_i := F_0(X_i), \; i \in \left\{ 1,\ldots , n\right\}\). Since \(F_0\) is continuous, \(\mathbb {P}_0\) has no points of nonzero mass, \((0,1) \subset F_0(\mathbb {R})\), and \(F_0(\cdot )\) is a one-to-one map between \(Q_0((0,1)) \subset \mathbb {R}\) and \((0,1)\): \(F_0^{-1} = Q_0\). So \(Y_1, \ldots , Y_n\) is a sample from \(\operatorname {U}[0,1]\).

Note that one-to-one transformations of the argument cannot change the value of the supremum in Equation 1, Equation 2. Hence,

\[ D_n \stackrel{\text{a.s.}}{=} \sup _{y \in (0,1)}\left|F_n^*(Q_0(y)) - y\right|. \]

Moreover, \(F_n^*(Q_0(y))\) is, a.s., the empirical distribution function for the sample \(Y_1, \ldots , Y_n\):

\[ F_n^*(Q_0(y)) = \frac{1}{n}\sum _{i=1}^n \; \mathbb {1}_{X_i \leq Q_0(y)} \stackrel{\text{a.s.}}{=} \frac{1}{n}\sum _{i=1}^n \; \mathbb {1}_{F_0(X_i) \leq y} = \frac{1}{n}\sum _{i=1}^n \; \mathbb {1}_{Y_i \leq y} \]

Thus,

\[ D_n \stackrel{\text{a.s.}}{=} \sup _{y \in (0,1)}\left|\tilde{F}_n^*(y) - y \right|, \]

where \(\tilde{F}_n^*\) is the empirical distribution function for the sample \(Y_1, \ldots , Y_n\) from \(\operatorname {U}[0,1]\).

The distribution of \(D_n\) depends only on \(n\), but for finite \(n\) it is given by overly cumbersome formulas. The limiting distribution of the statistic \(D_n\) looks somewhat simpler.

Theorem 2 (Kolmogorov) Suppose the hypothesis \(H_{0}\) holds, i.e. the distribution \(\mathbb {P}_0\) with distribution function \(F_0\) is the true distribution of the sample \(\overrightarrow {X}\). Then the asymptotic distribution of the statistic \(D_n\), scaled by \(\sqrt{n}\), is a certain known distribution not depending on \(\mathbb {P}_0, F_0\), called the Kolmogorov distribution (notation: \(\operatorname {K}\)):

\[ \sqrt{n}D_n \xrightarrow [n\to \infty ]{d_0} \operatorname {K} \]

The Kolmogorov distribution \(\operatorname {K}\) can be defined via the distribution function \(F_{\operatorname {K}}\):

\[ F_{\operatorname {K}}(t) = \sum _{j=-\infty }^{+\infty }(-1)^{j} e^{-2 j^{2} t^{2}} \cdot \; \mathbb {1}_{t \geq 0} \]

Corollary 1 Under the conditions of Kolmogorov’s theorem, \[ \mathbb {P}_{0}\left(\sqrt{n}D_n \leq t\right) \xrightarrow [n \to \infty ]{} F_{\operatorname {K}}(t), \quad \forall t > 0 \]

The Kolmogorov test of significance level \(\alpha\) is written as follows: \(\left\{ \sqrt{n} D_{n}> Q_{\operatorname {K}}(1-\alpha )\right\}\), where \(Q_{\operatorname {K}}(\cdot )\) is the quantile function of the Kolmogorov distribution. The Kolmogorov test is asymptotic, since we use not the exact distribution of \(D_n\) for each \(n\), but only the limiting distribution. The Kolmogorov test is customarily applied when \(n \geq 20\).

The density of the Kolmogorov distribution (the limiting distribution of the statistic \(\sqrt n D_n\)).

ExampleExample 2

Prove that

\[ D_{n}=\max _{1 \leq k \leq n} \max \left\{ \frac{k}{n}-F\left(X_{(k)}\right), F\left(X_{(k)}\right)-\frac{k-1}{n}\right\} . \]

SolutionSolution

Indeed, for any \(k \in \{ 1, \ldots , n-1\}\), on the interval \(\left[X_{(k)}, X_{(k+1)}\right)\) the function \(F_{n}^{*}\) is constant and equal to \(\frac{k}{n}\), while \(F(x)\) is nondecreasing, so on this interval any value of \(\left|F_{n}^{*}(x)-F(x)\right|\) does not exceed \(\max \left\{ \frac{k}{n}-F\left(X_{(k)}\right), F\left(X_{(k+1)}\right)-\frac{k}{n}\right\}\) (note that the second term has the form \(F(X_{(j)}) - \frac{j-1}{n}\) with \(j = k+1\)). Furthermore, on the interval \(\left(-\infty , X_{(1)}\right)\) the function \(F_{n}^{*}\) equals 0, so on this interval \(\left|F_{n}^{*}(x)-F(x)\right| \leq F\left(X_{(1)}\right)\). Finally, on the interval \(\left[X_{(n)}, \infty \right)\) the function \(F_{n}^{*}\) equals 1, so on this interval \(\left|F_{n}^{*}(x)-F(x)\right| \leq\) \(1-F\left(X_{(n)}\right)\). Since every value on the right-hand side of the claimed equality is attained by the function \(\left|F_{n}^{*}(x)-F(x)\right|\) (as one-sided limits at the points \(X_{(k)}\)), the equality is proved.

ProblemProblem 2

Prove the consistency of the Kolmogorov test.

ProblemProblem 3

We have a sample \(X_{1}, X_{2}, X_{3}\) of size 3. To test the hypothesis that the sample is drawn from the uniform distribution on \([0,1]\), the following version of the Kolmogorov test is used: the hypothesis of uniformity is rejected if \[ \sup _{y \in [0,1]}\left|F_{3}^{*}(y)-y\right|>1 / 3 . \] What is the significance level (size) of this test?

1.2 The Cramér-von Mises-Smirnov test

The Cramér-von Mises-Smirnov statistic is the squared distance between the functions \(F_n^*\) and \(F_0\) in the space \(L^{2}\left(\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right), \mathbb {P}_0\right)\):

\[ \omega ^2 := \int _{\mathbb {R}} \left|F_n^*(t) - F_0(t)\right|^2 \mathbb {P}_0(dt) = \int _{-\infty }^\infty \left|F_n^*(t) - F_0(t)\right|^2 dF_0(t). \]

The last integral is a Lebesgue-Stieltjes integral.

As with the distribution of the Kolmogorov statistic, the distribution of the statistic \(\omega^2\) does not depend on \(F_0\) when \(H_0\) holds.

ProblemProblem 4

Prove that if \(H_0\) holds, then \[ \omega ^2 \stackrel{d}{=} \int _{0}^1 \left|\tilde{F}_n^*(y) - y\right|^2 dy, \] where \(\tilde{F}_n^*(y)\) is the empirical distribution function for a sample from \(\operatorname {U}[0,1]\). That is, the distribution of \(\omega^2\) depends only on \(n\). In particular, it does not depend on \(F_0\).

As with \(D_n\), the distribution of \(\omega^2\) for finite \(n\) is given by overly cumbersome formulas, but the limiting distribution looks more or less simple.

Theorem 3 (Smirnov) Suppose the hypothesis \(H_{0}\) holds, i.e. the distribution \(\mathbb {P}_0\) with distribution function \(F_0\) is the true distribution of the sample \(X\). Then the asymptotic distribution of the statistic \(\omega^2\), scaled by \(n\), is a certain known distribution not depending on \(\mathbb {P}_0, F_0\), called the omega-squared distribution (notation: \(\operatorname {OS}\)):

\[ n\omega ^2 \xrightarrow [n\to \infty ]{d_0} \operatorname {OS} \]

The distribution \(\operatorname {OS}\) can be defined via its characteristic function:

\[ \varphi _{\operatorname {OS}}(t) =\prod _{k=1}^\infty \left(1 - \frac{2it}{\pi ^2 k^2}\right)^{-\frac{1}{2}} \]

The Cramér-von Mises-Smirnov test of significance level \(\alpha\) is written as follows: \(\left\{ n \omega^2>Q_{\operatorname {OS}}(1-\alpha )\right\}\), where \(Q_{\operatorname {OS}}(\cdot )\) is the quantile function of the omega-squared distribution. The Cramér-von Mises-Smirnov test, like the Kolmogorov test, is asymptotic, since we use not the exact distribution of \(\omega^2\) for each \(n\), but only the limiting distribution.

ProblemProblem 5

Prove the representation \[ \omega ^2 = \frac{1}{12 n^{2}} + \frac{1}{n} \sum _{k=1}^{n} \left(F_0\left(X_{(k)}\right) - \frac{2k-1}{2n}\right)^2. \] This representation is often used in practice to compute the value of the statistic \(\omega^{2}\).

ProblemProblem 6

Prove the consistency of the Cramér-von Mises-Smirnov test.

2 Comparing the theoretical and empirical densities: the chi-squared test

2.1 A sample from a Bernoulli scheme with known outcomes

Let \(X_{1}, \ldots , X_{n}\) be a sample from a Bernoulli scheme with \(m \geq 2\) known outcomes \(A = \left\{ a_1, \ldots , a_m\right\}\). The probability mass function (a generalized density with respect to the counting measure on \(A\)):

\[ p_j := \mathbb {P}\left(X_1 = a_j\right), \; j = \overline{1,m} \]

We know the outcomes \(a_1, \ldots , a_m\), but the probabilities \(p_1, \ldots , p_m\) are unknown. Such a distribution is also called categorical and is denoted

\[ \operatorname {Cat}(A, \overrightarrow {p}), \qquad \overrightarrow {p} = \left(p_1, p_2, \ldots , p_m\right)^T \]

Let us estimate the unknown distribution by the empirical distribution. In that case the estimator of \(p_j\) equals the sample frequency of hitting \(a_j\):

\[ \widehat{p_j} = \overline{\; \mathbb {1}_{X = a_j}}, \qquad \widehat{\overrightarrow {p}} = \left(\widehat{p}_1, \widehat{p}_2, \ldots , \widehat{p}_m\right)^T \]

ExampleExample 3

Prove that \(\widehat{\overrightarrow {p}}\) is an asymptotically normal estimator of \(\overrightarrow {p}\). Find its asymptotic covariance matrix.

SolutionSolution

Let

\[ \xi ^{(i)} := \begin{pmatrix} \; \mathbb {1}_{X_i = a_1} \\ \; \mathbb {1}_{X_i = a_2} \\ \vdots \\ \; \mathbb {1}_{X_i = a_m} \end{pmatrix} \]

Then \(\xi^{(1)}, \xi^{(2)}, \ldots\) is a sequence of independent identically distributed random vectors, with

\[ \frac{1}{n}\sum _{i=1}^n \xi ^{(i)} = \widehat{\overrightarrow {p}} \]

Next,

\[ \begin{align} \mathbb {E}\left[\xi ^{(1)}\right] & = \begin{pmatrix} \mathbb {P}\left(X_i = a_1\right) \\ \mathbb {P}\left(X_i = a_2\right) \\ \vdots \\ \mathbb {P}\left(X_i = a_m\right) \end{pmatrix} = \begin{pmatrix} p_1 \\ p_2 \\ \vdots \\ p_m \end{pmatrix} = \overrightarrow {p}, \\ \operatorname {Var}\left[\xi ^{(1)}\right] & = \mathbb {E}\left[\xi ^{(1)} \left(\xi ^{(1)}\right)^T\right] - \mathbb {E}\left[\xi ^{(1)}\right]\left(\mathbb {E}\left[\xi ^{(1)}\right]\right)^T = \\ & = \begin{pmatrix} \mathbb {P}\left(X_1 = a_1\right) & \mathbb {P}\left(X_1 = a_1, X_1 = a_2\right) & \ldots & \mathbb {P}\left(X_1 = a_1, X_1 = a_m\right) \\ \mathbb {P}\left(X_1 = a_2, X_1 = a_1\right) & \mathbb {P}\left(X_1 = a_2, X_1 = a_2\right) & \ldots & \mathbb {P}\left(X_1 = a_2, X_1 = a_m\right) \\ \vdots & \vdots & \vdots & \vdots \\ \mathbb {P}\left(X_1 = a_m, X_1 = a_1\right) & \mathbb {P}\left(X_1 = a_m, X_1 = a_2\right) & \ldots & \mathbb {P}\left(X_1 = a_m, X_1 = a_m\right) \end{pmatrix} - \mathbb {E}\left[\xi ^{(1)}\right]\left(\mathbb {E}\left[\xi ^{(1)}\right]\right)^T \end{align} \]

Since at any given moment \(X_1\) takes exactly one value, the events \(\left\{ X_1=a_i\right\}\) and \(\left\{ X_1=a_j\right\}\) are disjoint for \(i \neq j\) and coincide for \(i=j\); hence \(\mathbb {P}\left(X_1=a_i, X_1=a_j\right) = 0\) for \(i \neq j\) and \(=p_i\) for \(i=j\), i.e. the matrix of joint probabilities is diagonal:

\[ \begin{align} \operatorname {Var}\left[\xi ^{(1)}\right] & = \begin{pmatrix} p_1 & 0 & 0 & \ldots & 0 \\ 0 & p_2 & 0 & \ldots & 0 \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & 0 & \ldots & p_m \end{pmatrix} - \begin{pmatrix} p^2_1 & p_1p_2 & p_1p_3 & \ldots & p_1p_m \\ p_2p_1 & p^2_2 & p_2p_3 & \ldots & p_2p_m \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ p_mp_1 & p_mp_2 & p_mp_3 & \ldots & p_m^2 \end{pmatrix} = \\ & = \begin{pmatrix} p_1(1 - p_1) & -p_1p_2 & -p_1p_3 & \ldots & -p_1p_m \\ -p_2p_1 & p_2(1 - p_2) & -p_2p_3 & \ldots & -p_2p_m \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ -p_mp_1 & -p_mp_2 & -p_mp_3 & \ldots & p_m(1 - p_m) \end{pmatrix} \end{align} \]

By the CLT we have

\[ \sqrt{n}\left(\frac{1}{n}\sum _{i=1}^n \xi ^{(i)} - \overrightarrow {p}\right) = \sqrt{n}\left(\widehat{\overrightarrow {p}} - \overrightarrow {p}\right) \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, \operatorname {Var}\left[\xi ^{(1)}\right]\right) \]

2.1.1 Simple hypothesis

Let \(A\) be a fixed set of outcomes, and consider the following parametric model

\[ \mathcal{P} = \left\{ \operatorname {Cat}(A, \overrightarrow {p}): \; \overrightarrow {p}\in [0,1]^m, \sum _{i=1}^m p_i = 1\right\} . \]

Let \(\overrightarrow {p}^0 \in [0,1]^m\) be a fixed vector such that \(\sum_{i=1}^m p_i = 1\). Consider the parametric hypotheses

\[ H_0: \mathbb {P} = \mathbb {P}_0 = \operatorname {Cat}(A, \overrightarrow {p}^0), \qquad H_1: \mathbb {P} \in \mathcal{P}\setminus \left\{ \mathbb {P}_0\right\} = \left\{ \operatorname {Cat}(A, \overrightarrow {p}): \overrightarrow {p} \neq \overrightarrow {p}^0\right\} \]

Equivalently:

\[ H_0: \overrightarrow {p} = \overrightarrow {p}_0, \qquad H_1 : \overrightarrow {p} \neq \overrightarrow {p}_0 \]

Definition 3 The chi-squared statistic is defined as \[ \widehat{\chi ^2} := \sum _{j=1}^{m} \frac{\left(\mu _{j}-n p_{j}^{0}\right)^{2}}{n p_{j}^{0}} = \sum _{j=1}^{m} n\frac{\left(\widehat{p_j} - p_{j}^{0}\right)^{2}}{p_{j}^{0}} = \left\| \eta _n\right\| _{2}^2, \] where \[ \mu _{j}=\sum _{i=1}^{n} \; \mathbb {1}_{X_i = a_j}, \qquad \eta _n = \sqrt{n} \begin{pmatrix} \frac{\widehat{p_1} - p_{1}^{0}}{\sqrt{p_{1}^{0}}} \\ \vdots \\ \frac{\widehat{p_m} - p_{m}^{0}}{\sqrt{p_{m}^{0}}} \end{pmatrix} = \sqrt{n}B \left(\widehat{\overrightarrow {p}} - \overrightarrow {p}^0\right), \qquad B = \begin{pmatrix} \frac{1}{\sqrt{p_1^0}} & 0 & 0 & \ldots & 0 & 0 \\ 0 & \frac{1}{\sqrt{p_2^0}} & 0 & \ldots & 0 & 0 \\ 0 & 0 & \frac{1}{\sqrt{p_3^0}} & \ldots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & 0 & \ldots & \frac{1}{\sqrt{p_{m-1}^0}} & 0 \\ 0 & 0 & 0 & \ldots & 0 & \frac{1}{\sqrt{p_m^0}} \\ \end{pmatrix} \]

Theorem 4 (Pearson) Under the hypothesis \(H_{0}\), the statistic \(\widehat{\chi^2}\) converges in distribution to the chi-squared distribution with \(m-1\) degrees of freedom: \[ \widehat{\chi ^2} \xrightarrow [n \to \infty ]{d_0} \chi _{m-1}^2 \]

ExampleExample 4

Prove Pearson’s theorem.

SolutionSolution

We proved above that \(\sqrt{n}\left(\widehat{\overrightarrow {p}} - \overrightarrow {p}^0\right) \xrightarrow [n \to \infty ]{d_0} \eta \sim \mathscr {N}\left(0, \operatorname {Var}\left[\xi^{(1)}\right]\right)\). Hence \(\eta_n \xrightarrow [n \to \infty ]{d_0} \eta \sim \mathscr {N}\left(0, \Sigma \right)\), where \(\Sigma := B\cdot \operatorname {Var}\left[\xi^{(1)}\right] \cdot B^T\). Since \(B\) is diagonal with entries \(1/\sqrt{p_i^0}\), the \((i,j)\)-th entry of \(\Sigma\) equals the \((i,j)\)-th entry of \(\operatorname {Var}\left[\xi^{(1)}\right]\) divided by \(\sqrt{p_i^0p_j^0}\): for \(i=j\) we get \(p_i^0(1-p_i^0)/p_i^0 = 1-p_i^0\), and for \(i \neq j\) we get \(-p_i^0p_j^0/\sqrt{p_i^0p_j^0} = -\sqrt{p_i^0p_j^0}\), i.e.

\[ \Sigma = \begin{pmatrix} 1 - p^0_1 & -\sqrt{p^0_1p^0_2} & -\sqrt{p^0_1p^0_3} & \ldots & -\sqrt{p^0_1p^0_m} \\ -\sqrt{p^0_2p^0_1} & 1 - p^0_2 & -\sqrt{p^0_2p^0_3} & \ldots & -\sqrt{p^0_2p^0_m} \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ -\sqrt{p^0_mp^0_1} & -\sqrt{p^0_mp^0_2} & -\sqrt{p^0_mp^0_3} & \ldots & 1 - p^0_m \end{pmatrix} \]

By the theorem on preservation of convergence in distribution we obtain

\[ \left\| \eta _n\right\| _{2}^2 \xrightarrow [n \to \infty ]{d} \left\| \eta \right\| _{2}^2 \]

That is, we simply need to understand how \(\left\| \eta \right\|_{2}^2\) is distributed, where \(\eta \sim \mathscr {N}\left(0, \Sigma \right)\).

Let us study the matrix \(\Sigma\). The matrix \(\Sigma\) is symmetric, so its singular value decomposition has the form \(\Sigma = U \Lambda U^T\), where \(\Lambda\) is a matrix of dimension \(\operatorname {rk}(\Sigma ) \times \operatorname {rk}(\Sigma )\) with the nonzero real eigenvalues of \(\Sigma\) on the diagonal, and \(U\) is a matrix of dimension \(m \times \operatorname {rk}(\Sigma )\) with orthonormal column vectors.

How do we find the eigenvalues \(\lambda_1, \ldots , \lambda_m\) of the matrix \(\Sigma\)? Notice an interesting property of \(\Sigma\): \(\Sigma^2 = \Sigma\) (this property is not hard to show; we leave the proof as an exercise), i.e. \(\Sigma\) is idempotent. At the same time \(\Sigma^2 = U \Lambda U^TU \Lambda U^T = U \Lambda^2 U^T\). Hence the eigenvalues of \(\Sigma\) can only equal \(1\) or \(0\). Moreover, it is known from linear algebra that

\[ \lambda _1 + \ldots + \lambda _m = \operatorname {tr}(\Sigma ) = m - \sum _{i=1}^m p_i^0 = m-1 \]

Thus, \(\lambda_1 = \ldots = \lambda_{m - 1} = 1, \; \lambda_m = 0\).

Next, by the properties of Gaussian vectors, \(\eta = U \Lambda^{1/2} \zeta\), where \(\Lambda^{1/2} = I_{m-1}^{1/2} = I_{m-1}\), and \(\zeta \sim \mathscr {N}\left(0, I_{m-1}\right)\) is a vector of \(m-1\) independent standard normal random variables. Then

\[ \left\| \eta \right\| _{2}^2 = \eta ^T \eta = \zeta ^T U^T U \zeta = \zeta ^T \zeta = \left\| \zeta \right\| _{2}^2 \sim \chi ^2_{m-1} \]

Pearson’s chi-squared test is written as follows: \(\left\{ \widehat{\chi^2}> Q_{\chi^2_{m-1}}(1-\alpha )\right\}\), where \(Q_{\chi^2_{m-1}}(\cdot )\) is the quantile function of \(\chi_{m-1}^{2}\). The chi-squared test is asymptotic; it is customarily applied when \(n \geq 50\) and \(\mu_{j} \geq 5\) for all \(j \in \{ 1, \ldots , m\}\).

ProblemProblem 7

Prove the consistency of the chi-squared test.

ExampleExample 5

According to statistics gathered at a psychiatric hospital over a year, the number of patients admitted to the intensive care unit had the following distribution by day of the week: \[ \text{Mon} - 36, \text{Tue} - 53, \text{Wed} - 35, \text{Thu} - 26, \text{Fri} - 30, \text{Sat} - 44, \text{Sun} - 28. \] Are these data consistent with the hypothesis that admission does not depend on the day of the week, at significance levels \(0.05\) and \(0.01\)?

SolutionSolution

Set \(m=7, p_{j}^{0}=\frac{1}{7}, j \in \{ 1, \ldots , m\}\). The total number of observations equals

\[ n=36+53+35+26+30+44+28=252 \]

Thus \(n p_{j}^{0}=36\) for all \(j \in \{ 1, \ldots , m\}\). The deviations of the observed frequencies from the expected value, by day (Mon, Tue, Wed, Thu, Fri, Sat, Sun), equal

\[ \mu _j - 36 = 0, \; 17, \; -1, \; -10, \; -6, \; 8, \; -8, \]

and the contribution of each day to the statistic, i.e. the term \((\mu_j-36)^2/36\), equals

\[ 0, \quad \frac{17^2}{36} \approx 8.03, \quad \frac{1^2}{36}\approx 0.03, \quad \frac{10^2}{36}\approx 2.78, \quad \frac{6^2}{36}=1, \quad \frac{8^2}{36}\approx 1.78, \quad \frac{8^2}{36}\approx 1.78. \]

Summing, we get

\[ \widehat{\chi ^2} = 0+8.03+0.03+2.78+1+1.78+1.78=15.4. \]

Observed frequencies of admissions to the unit by day of the week (data from the example above, \(n=252\)) against the values expected under \(H_0\) (uniform distribution over days, \(np_j^0 = 36\) — dashed line). Hover over a bar to see its contribution \((\mu_j-36)^2/36\) to the \(\chi^2\) statistic. The “move patients” slider lets you redistribute observations between two selected days and live-recomputes \(\chi^2\) and the \(p\)-value; the scale below compares the current \(\chi^2\) with the critical value for the selected significance level. Here \(\chi^2 \sim \chi^2_6\); the \(p\)-value uses the Wilson-Hilferty approximation to the \(\chi^2\) distribution function (a normal approximation), not the exact incomplete gamma function.

Move 0 patients from to

\(\chi^2 = \) \(p\)-value \(\approx\) Significance level:

The quantiles of the chi-squared distribution with 6 degrees of freedom are: \(u_{0.95} \approx 12.6, u_{0.99} \approx 16.8\). Thus, at significance level \(0.05\) we reject the hypothesis of uniform admission of patients, while at level \(0.01\) we do not reject it (though in the second case the probability of a type II error, of course, increases substantially, so a high significance level is not advisable when there are not so many observations).

ProblemProblem 8

The digits \(0,1,2, \ldots , 9\) among the first 800 decimal digits of \(\pi\) appeared 74, 92, 83, 79, 80, 73, 77, 75, 76, 91 times respectively. Using the chi-squared test, check the hypothesis that these data are consistent with the uniform distribution on the set \(\left\{ 0,1, \ldots , 9\right\}\) at significance level a) \(5\%\), b) \(50\%\), c) \(80\%\).

ProblemProblem 9

A professional dentist has learned to knock out wisdom teeth with a punch. It is known that he knocked out 52 wisdom teeth on the first attempt, 31 on the second, 3 on the third, and needed 4 or more attempts for the remaining 5 teeth. Test the hypothesis that the dentist knocks out an arbitrary wisdom tooth with probability \(\frac{2}{3}\), at significance level \(0.1\).

2.1.2 Composite hypothesis

Pearson’s chi-squared test can also be applied in the case of composite hypotheses

\[ H_0: \overrightarrow {p} = \overrightarrow {p}^0(\overrightarrow {\theta }), \; \overrightarrow {\theta } \in \Theta \subset \mathbb {R}^{r}, \; r < m-1, \qquad H_1: \overrightarrow {p} \neq \overrightarrow {p}^0(\overrightarrow {\theta }) \]

To test such hypotheses one also uses the statistic \(\widehat{\chi^2} = \widehat{\chi^2}(\overrightarrow {\theta })\), in which \(\overrightarrow {\theta }\) is replaced by the maximum likelihood estimator

\[ \overrightarrow {\theta }^{*} = \operatorname *{arg\, max}_{\overrightarrow {\theta } \in \Theta } \prod _{j=1}^{m}\left(p_{j}^{0}(\overrightarrow {\theta })\right)^{\mu _{j}} . \]

Theorem 5 (Fisher) Suppose the following conditions hold:

1. the hypothesis \(H_{0}\) holds;

2. for every \(\overrightarrow {\theta } \in \Theta\) the equality \(\sum_{j=1}^{m} p_{j}(\overrightarrow {\theta })=1\) holds;

3. there is a positive number \(c\) such that for all \(j \in \{ 1, \ldots , m\}\) and \(\overrightarrow {\theta } \in \Theta\), \(p_{j}(\overrightarrow {\theta }) \geq c\);

4. for all \(j \in \{ 1, \ldots , m\} , k, l \in\) \(\{ 1, \ldots , r\}\) the continuous derivatives \(\frac{\partial p_{j}^{0}(\overrightarrow {\theta })}{\partial \theta_{k}}, \frac{\partial^{2} p_{j}^{0}(\overrightarrow {\theta })}{\partial \theta_{k} \partial \theta_{l}}\) exist;

5. the matrix formed from the partial derivatives \(\frac{\partial p_{j}^{0}(\overrightarrow {\theta })}{\partial \theta_{k}}(j \in \{ 1, \ldots , m\} , k \in\) \(\{ 1, \ldots , r\} )\) has rank \(r\) for all \(\theta \in \Theta\).

Then

\[ \widehat{\chi ^2}\left(\overrightarrow {\theta }^{*}\right) \xrightarrow [n \to \infty ]{d} \chi _{m-1-r}^{2} \]

In the case of composite hypotheses, the chi-squared test is also applied when \(n \geq 50\) and \(\mu_{j} \geq 5\) for all \(j \in \{ 1, \ldots , m\}\).

ExampleExample 6

Test at significance level \(\alpha\) the hypothesis that the measured quantity \(X\) has a binomial distribution with parameters \((2, \theta )\), if, out of \(n=128\) measurements, it took the value \(0, \mu_{1}=n / 4\) times, the value 1 \(\mu_{2}=n / 4\) times, and the value 2 \(\mu_{3}=n / 2\) times.

SolutionSolution

The problem concerns testing the hypothesis \[ H_{0}: p_{1}^{0}(\theta )=(1-\theta )^{2}, p_{2}^{0}(\theta )=2 \theta (1-\theta ), p_{3}^{0}=\theta ^{2}. \] Let us find the maximum likelihood estimator: \[ \frac{\partial }{\partial \theta }\left(\ln \prod _{j=1}^{3}\left(p_{j}^{0}(\theta )\right)^{\mu _{j}}\right) = -\frac{2 \mu _{1}}{1-\theta }+\frac{\mu _{2}}{\theta }-\frac{\mu _{2}}{1-\theta }+\frac{2 \mu _{3}}{\theta }. \] Bringing this to the common denominator \(\theta (1-\theta )\): the numerator equals \(-2\mu_1\theta - \mu_2 \theta + \mu_2(1-\theta ) + 2\mu_3(1-\theta ) = \mu_2 + 2\mu_3 - 2\theta (\mu_1+\mu_2+\mu_3)\), and since \(\mu_1+\mu_2+\mu_3=n\), this equals \(2\mu_3+\mu_2-2n\theta\). So \[ \frac{\partial }{\partial \theta }\left(\ln \prod _{j=1}^{3}\left(p_{j}^{0}(\theta )\right)^{\mu _{j}}\right) = \frac{2 \mu _{3}+\mu _{2}-2 n \theta }{\theta (1-\theta )}. \] Hence \(\theta^{*}=\frac{2 \mu_{3}+\mu_{2}}{2 n}=\frac{2 \cdot \frac{n}{2}+\frac{n}{4}}{2 n} = \frac{5}{8}\). Then \[ p^0_1\left(\tfrac {5}{8}\right) = \frac{9}{64}, \quad p^0_2\left(\tfrac {5}{8}\right) = \frac{30}{64}, \quad p^0_3\left(\tfrac {5}{8}\right) = \frac{25}{64}, \qquad np^0_1 = 18, \quad np^0_2 = 60, \quad np^0_3 = 50, \] and finally we obtain the value of the statistic \[ \widehat{\chi ^2}(\theta ^*) = \sum _{j=1}^{3} \frac{\left(\mu _{j}-n p_{j}^{0}(5/8)\right)^{2}}{n p_{j}^{0}(5/8)}=\frac{(32-18)^{2}}{18}+\frac{(32-60)^{2}}{60}+\frac{(64-50)^{2}}{50} \approx 10.89 + 13.07 + 3.92 = 27.88 \] Here \(m = 3\), \(r = 1\), i.e. under \(H_0\) the statistic is asymptotically distributed as \(\chi^2_{m-1-r} = \chi^2_1\). Since the 0.99-quantile of \(\chi_{1}^{2}\), to four decimal places, equals \(6.6349 \ll 27.88\), the hypothesis should be rejected even at significance level \(0.01\).

ProblemProblem 10

Among 5000 families with three children, there are exactly 1010 families with three boys, 2200 families with two boys and one girl, and 950 families with one boy and two girls (in all other families, all children are girls). Can we, at significance level \(\alpha =0.02\), consider that the number of boys \(\xi\) in a family with three children has the following distribution: \[ \mathbb {P}\left(\xi = 0\right) = \theta , \quad \mathbb {P}\left(\xi = 1\right) = \theta , \quad \mathbb {P}\left(\xi = 2\right) = 2\theta , \quad \mathbb {P}\left(\xi = 3\right) = 1-4 \theta , \qquad \text{ where } \theta \in \left(0, \frac{1}{4}\right). \]

2.2 The general case

The chi-squared test, like the Kolmogorov goodness-of-fit test, is used to test the hypothesis that the distribution from which our sample is drawn coincides with some specific distribution \(\mathbb {P}_0\). Unlike the Kolmogorov test, the chi-squared test does not require heavy computation, but is less “precise.” One can work with the chi-squared test in the case of an arbitrary class of distributions as follows: the range of the sample is partitioned into several intervals, after which the number of sample elements falling into each interval is computed. The resulting values are taken as \(\mu_{j}\). As \(p_{j}^{0}\) one takes the probability that a random variable with distribution \(\mathbb {P}_0\) falls into the \(j\)-th interval. In essence, one compares the histogram of the sample with the theoretical histogram of the distribution \(\mathbb {P}_0\).

On the number of classes, see (Kallenberg et al. 1985).

References

Kallenberg, W. C. M., J. Oosterhoff, and B. F. Schriever. 1985. “The Number of Classes in Chi-Squared Goodness-of-Fit Tests.” Journal of the American Statistical Association 80 (392): 959–68. http://www.jstor.org/stable/2288561.