\(\mathrm{E}[F_{d_1,d_2}] = \dfrac{d_2}{d_2-2}\) =
\(\mathrm{Var}[F_{d_1,d_2}]\) =
Hypothesis Testing in the Linear Gaussian Model
WarningDraft
This note is still being edited: the text may contain typos, inaccuracies, and locally unpolished passages. If you find something, please let the author know (contact information on the course page).
1 Testing a simple hypothesis of the form \(\overrightarrow {\theta } = \overrightarrow {\theta }_0\)
Suppose we have a linear Gaussian model
\[ \overrightarrow {X} = \mathbf{z}\overrightarrow {\theta } + \overrightarrow {\varepsilon }, \qquad \overrightarrow {\theta } \in \mathbb {R}^{\tilde{k}}, \qquad \overrightarrow {\varepsilon } \sim \mathscr {N}\left(\overrightarrow {0}, \sigma ^2 \mathbf{I}_n\right). \]
Recall that if the model has no intercept, i.e.
\[ X_i = \theta _1 z_{i1} + \theta _2 z_{i2} + \ldots + \theta _k z_{ik} + \varepsilon _i, \]
then \(\tilde{k} = k\). If, on the other hand, the regression has an intercept, i.e. the model is
\[ X_i = \theta _0 + \theta _1 z_{i1} + \theta _2 z_{i2} + \ldots + \theta _k z_{ik} + \varepsilon _i, \]
then \(\tilde{k} = k + 1\).
Suppose we are given the following simple parametric null hypothesis against a composite parametric alternative:
\[ H_0: \overrightarrow {\theta } = \overrightarrow {\theta }_0, \qquad H_1: \overrightarrow {\theta } \neq \overrightarrow {\theta }_0, \]
where \(\overrightarrow {\theta }_0 \in \mathbb {R}^{\tilde{k}}\) is an arbitrary fixed vector.
For example, \(\overrightarrow {\theta }_0 = \overrightarrow {0}\): in this case we test \(\overrightarrow {X} = \overrightarrow {\varepsilon }\) against \(\overrightarrow {X} = \mathbf{z}\overrightarrow {\theta } + \overrightarrow {\varepsilon }\), where \(\overrightarrow {\theta } \neq \overrightarrow {0}\). In essence, we are testing the hypothesis that the given regressors \(\mathbf{z}\) genuinely affect the values of the dependent variable \(\overrightarrow {X}\). The null hypothesis in this case is that \(\overrightarrow {X}\) is simply noise, not depending in any way on the regressors.
Geometry of testing the simple hypothesis \(\mathbf z\overrightarrow{\theta}=\overrightarrow{\theta}_0\) (Three.js version): the observation \(X\), its projection \(\widehat X=X_L=\mathbf z\widehat{\overrightarrow\theta}\) onto the subspace \(L\), the hypothesized point \(F=\mathbf z\overrightarrow{\theta}_0\), and the discrepancy vector \(\overrightarrow{FB}=X_L-\mathbf z\overrightarrow{\theta}_0\). The right angle at vertex \(B\) of the triangle \(OFB{\to}A\) illustrates that \(X-X_L\perp L\), and the \(F\)-statistic, when \(H_0\) holds, compares the length of the short discrepancy \(\overrightarrow{FB}\) (lying in \(L\)) with the length of the error \(\overrightarrow{BA}=\widehat{\overrightarrow\varepsilon}\) (perpendicular to \(L\)) — by Cochran’s theorem these vectors are independent. The labels are genuinely rendered LaTeX (KaTeX), anchored to their line/vector and rotated along it; when the scene is rotated with the mouse they stay in place. The mouse wheel zooms.
How can we test \(H_0\) against \(H_1\)? Let us take a closer look at fig. @LinearRegrHypTest:GraphSimpelHyp: if the null hypothesis is true, i.e. the unknown true parameter \(\overrightarrow {\theta }\) equals \(\overrightarrow {\theta }_0\), then its estimator \(\widehat{\overrightarrow {\theta }}\) should not differ much from \(\overrightarrow {\theta }_0\), since
\[ \widehat{\overrightarrow {\theta }} \sim \mathscr {N}\left(\overrightarrow {\theta }_0, \sigma ^2 \left(\mathbf{z}^T \mathbf{z}\right)^{-1}\right). \]
In other words, the vector
\[ \overrightarrow {FB} = \mathbf{z}\widehat{\overrightarrow {\theta }} - \mathbf{z}\overrightarrow {\theta }_0 = \widehat{\overrightarrow {X}} - \mathbb {E}\left[\widehat{\overrightarrow {X}}\right] \]
should not be large compared to the other vectors
\[ \overrightarrow {OA} = \overrightarrow {X}, \qquad \overrightarrow {OB} = \widehat{\overrightarrow {X}}, \qquad \overrightarrow {BA} = \overrightarrow {X} -\widehat{\overrightarrow {X}}. \]
It is convenient to compare \(\mathbf{z}\widehat{\overrightarrow {\theta }} - \mathbf{z}\overrightarrow {\theta }_0\) specifically with the last vector, with the vector \(\overrightarrow {X} -\widehat{\overrightarrow {X}} = \overrightarrow {X}_{L^\perp }\), since by Cochran’s theorem on the orthogonal decomposition of a Gaussian vector, these vectors are independent. By the same theorem we can find (assuming \(H_0\) holds) the distributions of the squared lengths of these vectors. For the vector \(\overrightarrow {FB}\), note that under \(H_0\)
\[ \overrightarrow {FB} = \mathbf{z}\widehat{\overrightarrow {\theta }} - \mathbf{z}\overrightarrow {\theta }_0 = \widehat{\overrightarrow {X}} - \mathbb {E}\left[\widehat{\overrightarrow {X}}\right] = \overrightarrow {X}_L - \mathbb {E}\left[\overrightarrow {X}_L\right], \]
so
\[ \frac{1}{\sigma ^2}\left|\overrightarrow {FB}\right|^2 = \frac{1}{\sigma ^2}\left\| \overrightarrow {X}_L - \mathbb {E}\left[\overrightarrow {X}_L\right]\right\| ^2 \sim \chi ^2_{\tilde{k}}. \]
For the vector \(\overrightarrow {BA} = \overrightarrow {X} -\widehat{\overrightarrow {X}} = \overrightarrow {X}_{L^\perp }\) we have \(\mathbb {E}\left[\overrightarrow {X}_{L^\perp }\right] = \overrightarrow {0}\), so
\[ \frac{1}{\sigma ^2}\left|\overrightarrow {BA}\right|^2 = \frac{1}{\sigma ^2}\left\| \overrightarrow {X}_{L^\perp }\right\| ^2 = \frac{1}{\sigma ^2} \operatorname {RSS} \sim \chi ^2_{n-\tilde{k}}. \]
The density of the Fisher distribution \(\operatorname{F}_{d_1,d_2}\) — the distribution of the ratio \(\frac{\xi_1/d_1}{\xi_2/d_2}\) of independent \(\chi^2_{d_1}, \chi^2_{d_2}\) random variables (see the definition of the F-statistic above). Move the sliders \(d_1, d_2\) — the density updates continuously; the buttons set four characteristic examples. If \(H_0\) holds, the numerator and denominator of the \(F\)-statistic estimate the same variance \(\sigma^2\), and as \(d_1, d_2\) grow, their ratio concentrates around \(1\) (dashed vertical line).
The statistic
\[ F := \frac{\left|\overrightarrow {FB}\right|^2 / \tilde{k}}{\left|\overrightarrow {BA}\right|^2 /( n - \tilde{k})} = \frac{\left\| \mathbf{z}\widehat{\overrightarrow {\theta }} - \mathbf{z}\overrightarrow {\theta }_0\right\| ^2/\tilde{k}}{\left\| \overrightarrow {X} -\widehat{\overrightarrow {X}}\right\| ^2/( n - \tilde{k})} \]
is called the \(F\)-statistic. Explicitly (expanding the squared norms)
\[ F = \frac{\left(\widehat{\overrightarrow {\theta }} - \overrightarrow {\theta }_0\right)^T \mathbf{z}^T \mathbf{z}\left(\widehat{\overrightarrow {\theta }} - \overrightarrow {\theta }_0\right)/\tilde{k}}{\operatorname {RSS}/( n - \tilde{k})}. \]
By definition Definition 1, it follows the Fisher distribution with \(\tilde{k}, n - \tilde{k}\) degrees of freedom: \(F \sim \operatorname {F}_{\tilde{k}, n - \tilde{k}}\).
Test of significance level \(\alpha\) for testing \(H_0\) against \(H_1\): if the \(F\)-statistic is too large — greater than the quantile of level \(1-\alpha\) from the distribution \(\operatorname {F}_{\tilde{k}, n - \tilde{k}}\) — then we reject \(H_0\). That is,
\[ S = \left\{ \overrightarrow {x} \in \mathbb {R}^{n}: F(\overrightarrow {x}) > Q_{\operatorname {F}_{\tilde{k}, n - \tilde{k}}}(1 - \alpha )\right\} . \]
2 General case: testing a composite hypothesis of the form \(\mathbf{w}^T\overrightarrow {\theta } = \overrightarrow {\tau }_0\)
Continuing from the previous section, consider a composite parametric null hypothesis against a composite parametric alternative:
\[ H_0: \mathbf{w}^T \overrightarrow {\theta } = \overrightarrow {\tau }_0, \qquad H_1: \mathbf{w}^T \overrightarrow {\theta } \neq \overrightarrow {\tau }_0, \tag{1}\]
where \(\mathbf{w} \in \mathbb {R}^{\tilde{k} \times m}\) is some fixed matrix of dimension \(\tilde{k} \times m, \; m \leq \tilde{k}\) of full rank: \(\operatorname {rk}(\mathbf{w}) = m\). \(\overrightarrow {\tau }_0 \in \mathbb {R}^{m}\) is some fixed vector.
It is worth noting that hypothesis \(H_0\) from LinearRegrHypTest:ComposHyp is composite only if the matrix \(\mathbf{w}^T\) has fewer rows than columns, i.e. \(m < \tilde{k}\). Note that the simple hypothesis \(\overrightarrow {\theta } = \overrightarrow {\theta }_0\) from the previous section is a special case of LinearRegrHypTest:ComposHyp: one just needs to take \(\mathbf{w} = \mathbf{I}_{\tilde{k}}, \; \overrightarrow {\tau }_0 = \overrightarrow {\theta }_0\).
Geometry of testing the composite hypothesis \(\mathbf w^T\overrightarrow{\theta}=\overrightarrow{\tau_0}\) (Three.js version): the subspace \(L_1\) (condition \(\mathbf w^T\overrightarrow{\theta}=0\)) and its orthogonal complement \(L_2\) in \(L\), the affine subspaces \(\mathcal L_1=L_1+\overrightarrow{l_2}\) and its estimate \(\widehat{\mathcal L}_1=L_1+X_{L_2}\), and the points \(B=X_L=\mathbf z\widehat{\overrightarrow{\theta}}\) and \(F=X_{\mathcal L_1}\) (constrained OLS estimate). The right angles at vertices \(B\) and \(F\) of the triangle \(AFB\) illustrate the Pythagorean theorem. The labels are genuinely rendered LaTeX (KaTeX), anchored to their line/vector and rotated along it; when the scene is rotated with the mouse they stay in place and correctly update their angle. The mouse wheel zooms; the slider \(\tau_0\) below moves the hypothesis.
How can we test \(H_0\) against \(H_1\)? Let us take a closer look at fig. @LinearRegrHypTest:GraphCompositeHyp. Note that the condition \(\mathbf{w}^T \overrightarrow {\theta } = \overrightarrow {0}\) on the vector \(\mathbf{z}\overrightarrow {\theta } \in \mathbb {R}^{n}\) defines some linear subspace \(L_1\) within the subspace
\[ L = \left\{ \mathbf{z}\overrightarrow {\theta }: \; \overrightarrow {\theta } \in \mathbb {R}^{\tilde{k}}\right\} \subset \mathbb {R}^{n}. \]
Since the rank of the matrix \(\mathbf{w}^T\) equals \(m\), i.e. the number of its rows, the condition \(\mathbf{w}^T \overrightarrow {\theta } = \overrightarrow {0}\) imposes \(m\) linear constraints on the \(\tilde{k}\)-dimensional vector \(\overrightarrow {\theta }\). Hence,
\[ \dim (L_1) = \dim (L) - m = \tilde{k}-m. \]
Denote by \(L_2\) the orthogonal complement of \(L_1\) within the subspace \(L\). We have \(\dim (L_2) = \dim (L) - \dim (L_1) = m\).
The condition \(\mathbf{w}^T \overrightarrow {\theta } = \overrightarrow {\tau }_0\) on the vector \(\mathbf{z}\overrightarrow {\theta } \in \mathbb {R}^{n}\) obviously defines some affine subspace \(\mathcal{L}_1\) with linear part \(L_1\). Let \(\overrightarrow {l}_2 := L_2 \cap \mathcal{L}_1\). Note that \(\mathcal{L}_1 = L_1 + \overrightarrow {l}_2\).
Denote \(\widehat{\overrightarrow {\tau }} := \mathbf{w}^T \widehat{\overrightarrow {\theta }}\). Note that
\[ \mathbb {E}\left[\widehat{\overrightarrow {\tau }}\right] = \mathbf{w}^T \mathbb {E}\left[\widehat{\overrightarrow {\theta }}\right] = \mathbf{w}^T \overrightarrow {\theta } \qquad \forall \overrightarrow {\theta } \in \mathbb {R}^{\tilde{k}}, \]
i.e. \(\widehat{\overrightarrow {\tau }}\) is an unbiased estimator of \(\mathbf{w}^T \overrightarrow {\theta }\). Since \(\widehat{\overrightarrow {\tau }}\) is a function of a complete, sufficient statistic, \(\widehat{\overrightarrow {\tau }}\) is the optimal estimator of \(\mathbf{w}^T \overrightarrow {\theta }\).
Denote
\[ \widehat{\mathcal{L}}_1 := \left\{ \mathbf{z}\overrightarrow {\theta }: \; \mathbf{w}^T\overrightarrow {\theta } = \widehat{\overrightarrow {\tau }}\right\} \]
the estimator of the affine subspace \(\mathcal{L}_1\). Note that \(\widehat{\mathcal{L}}_1 = L_1 + \overrightarrow {X}_{L_2}\). Denote
\[ \overrightarrow {X}_{\mathcal{L}_1} = \operatorname *{arg\, min}_{\overrightarrow {x} \in \mathcal{L}_1}\left\| \overrightarrow {X} - \overrightarrow {x}\right\| \]
Note that \(\overrightarrow {X}_{\mathcal{L}_1} = \overrightarrow {X}_{L_1} + \overrightarrow {l}_2\)
If \(H_0\) holds, then the estimator \(\widehat{\overrightarrow {\tau }}\) should be close to \(\overrightarrow {\tau }_0\), and \(\widehat{\mathcal{L}}_1\) should be close to \(\mathcal{L}_1\). The distance between them
\[ \rho (\widehat{\mathcal{L}}_1, \mathcal{L}_1) = \left|\overrightarrow {FB}\right| = \left\| \overrightarrow {X}_L - \overrightarrow {X}_{\mathcal{L}_1}\right\| , \]
as in the previous section, is convenient to compare with \(\left|\overrightarrow {BA}\right| = \left\| \overrightarrow {X} - \overrightarrow {X_L}\right\| = \sqrt{\operatorname {RSS}}\), since by the theorem on the orthogonal decomposition of a Gaussian vector, the vectors
\[ \overrightarrow {FB} = \overrightarrow {X}_L - \overrightarrow {X}_{\mathcal{L}_1} = \overrightarrow {X}_{L_2} - \overrightarrow {l}_2 \qquad \text{and}\qquad \overrightarrow {BA} = \overrightarrow {X} - \overrightarrow {X_L} \]
are independent.
We already know how to find \(\left|\overrightarrow {BA}\right|^2 = \left\| \overrightarrow {X} - \overrightarrow {X_L}\right\|^2 = \operatorname {RSS}\) and how \(\operatorname {RSS}\) is distributed:
\[ \operatorname {RSS} = \left\| \overrightarrow {X} - \mathbf{z}\widehat{\overrightarrow {\theta }}\right\| ^2, \qquad \frac{1}{\sigma ^2}\operatorname {RSS} \sim \chi ^2_{n-\tilde{k}}. \]
Let us express \(\left|\overrightarrow {FB}\right|^2\) in terms of \(\overrightarrow {X}, \mathbf{z}, \mathbf{w}\) and find the distribution of \(\left|\overrightarrow {FB}\right|^2\).
The statistic
\[ F = F(\overrightarrow {X}) := \frac{\left|\overrightarrow {FB}\right|^2 / m}{\left|\overrightarrow {BA}\right|^2 /( n - \tilde{k})} = \frac{\left\| \overrightarrow {X}_{L} - \overrightarrow {X}_{\mathcal{L}_1}\right\| ^2/m}{\left\| \overrightarrow {X} -\widehat{\overrightarrow {X}}\right\| ^2/( n - \tilde{k})} \]
is called the \(F\)-statistic. By theorem Theorem 1, explicitly
\[ F = \frac{\left(\widehat{\overrightarrow {\tau }} - \overrightarrow {\tau }_0\right)^T \mathbf{b}^{-1}\left(\widehat{\overrightarrow {\tau }} - \overrightarrow {\tau }_0\right)/m}{\operatorname {RSS}/( n - \tilde{k})}. \]
If \(H_0\) holds, then, by what was proved above and by definition Definition 1, this statistic is distributed according to the Fisher distribution with \(m, n - \tilde{k}\) degrees of freedom: \(F \sim \operatorname {F}_{m, n - \tilde{k}}\).
Test of significance level \(\alpha\) for testing \(H_0\) against \(H_1\): if the \(F\)-statistic is too large — greater than the quantile of level \(1-\alpha\) from the distribution \(\operatorname {F}_{m, n - \tilde{k}}\) — then we reject \(H_0\). That is,
\[ S = \left\{ \overrightarrow {x} \in \mathbb {R}^{n}: F(\overrightarrow {x}) > Q_{\operatorname {F}_{m, n - \tilde{k}}}(1 - \alpha )\right\} . \]
In some problems computing the statistic \(F\) is difficult because it requires computing the inverse matrix \(\mathbf{b}^{-1}\). In such cases it is convenient to use the Pythagorean theorem in the triangle \(AFB\) (see fig. @LinearRegrHypTest:GraphCompositeHyp). We have
\[ \min _{\overrightarrow {\theta }: \mathbf{w}^T\overrightarrow {\theta } = \overrightarrow {\tau }_0}\left\| \overrightarrow {X} - \mathbf{z}\overrightarrow {\theta }\right\| ^2 = \left\| \overrightarrow {X} - \overrightarrow {X}_{\mathcal{L}_1}\right\| ^2 = \left|\overrightarrow {FA}\right|^2 = \left|\overrightarrow {FB}\right|^2 + \left|\overrightarrow {BA}\right|^2 = \left|\overrightarrow {FB}\right|^2 + \operatorname {RSS} \]
If the quantity
\[ \operatorname {RSS} = \min _{\overrightarrow {\theta } \in \mathbb {R}^{\tilde{k}}}\left\| \overrightarrow {X} - \mathbf{z}\overrightarrow {\theta }\right\| ^2 \]
is called the least squares estimator of the sum of squares, then the quantity
\[ \min _{\overrightarrow {\theta }: \mathbf{w}^T\overrightarrow {\theta } = \overrightarrow {\tau }_0}\left\| \overrightarrow {X} - \mathbf{z}\overrightarrow {\theta }\right\| ^2 \]
is called the constrained least squares estimator. It is often more convenient to compute the constrained least squares estimator, and then express \(\left|\overrightarrow {FB}\right|^2\) as
\[ \left|\overrightarrow {FB}\right|^2 = \min _{\overrightarrow {\theta }: \mathbf{w}^T\overrightarrow {\theta } = \overrightarrow {\tau }_0}\left\| \overrightarrow {X} - \mathbf{z}\overrightarrow {\theta }\right\| ^2 - \operatorname {RSS} \]
3 (*) Classical tests as special cases of the linear model
The material of this chapter (together with the chapter on linear regression) is enough to “read” almost the entire standard set of tests built into statistical packages. For concreteness we will focus on the Excel add-in “Analysis ToolPak”; the tools linked to the linear Gaussian model in it are:
three \(t\)-tests: “Paired two-sample \(t\)-test for means”, “Two-sample \(t\)-test assuming equal variances”, “Two-sample \(t\)-test assuming unequal variances”;
“Two-sample \(z\)-test for means”;
“Two-sample \(F\)-test for variances”;
three variants of analysis of variance: “One-way ANOVA”, “Two-way ANOVA with replication”, “Two-way ANOVA without replication”;
“Regression”.
Almost all of them are special cases of the \(F\)-test from this chapter, applied to a suitable linear model (and the few exceptions — Welch’s test and the \(F\)-test for variances — are assembled from the same “building blocks” of Cochran’s theorem). Let us go through them one by one.
3.1 The \(t\)-form of the \(F\)-test for one-dimensional hypotheses
All the \(t\)-statistics listed above are “square roots” of \(F\)-statistics. The following theorem gives this precise meaning.
Recall (see the chapter on linear regression) that as the number of degrees of freedom \(\nu\) grows, Student’s distribution \(T_\nu\) approaches \(N(0,1)\) — for large samples the difference between the \(t\)-test and the \(z\)-test disappears, and the “heaviness” of the tails of \(T_\nu\) matters primarily for small \(n\).
The density of Student’s distribution \(T_\nu\) compared with \(N(0,1)\): the more degrees of freedom \(\nu\), the thinner the tails and the closer the curve is to the normal one. Move the \(\nu\) slider to see how the curve “flows” into the standard normal; the switch below highlights the two-sided critical tail of level \(\alpha\) and shows how the critical value of \(t\) changes with \(\nu\).
t critical (two-tailed) =
A test of significance level \(\alpha\) based on \(T\) against the two-sided alternative \(H_1: \mathbf{w}^T\overrightarrow {\theta } \neq \tau_0\) coincides with the \(F\)-test:
\[ S = \left\{ \left|T\right| > Q_{T_{n-\tilde{k}}}\left(1 - \tfrac {\alpha }{2}\right)\right\} , \qquad p\text{-value} = \mathbb {P}\left(\left|T_{n - \tilde{k}}\right| > \left|T(\overrightarrow {x})\right|\right). \]
The advantage of the \(t\)-form is that, unlike \(F = T^2\), it “sees the sign” of the deviation, and thus allows testing one-sided alternatives as well, \(H_1: \mathbf{w}^T\overrightarrow {\theta } > \tau_0\) (the test \(\left\{ T > Q_{T_{n-\tilde{k}}}(1 - \alpha )\right\}\)) and \(H_1: \mathbf{w}^T\overrightarrow {\theta } < \tau_0\) (the test \(\left\{ T < Q_{T_{n-\tilde{k}}}(\alpha )\right\}\)). This is exactly why Excel’s output for the \(t\)-tests gives a pair of values: “\(t\) Critical one-tail” and “\(t\) Critical two-tail” (and the corresponding \(p\)-values).
3.2 One-sample \(t\)-test
Given a sample \(X_1, \ldots , X_n\) from \(\mathscr {N}\left(\theta , \sigma^2\right)\) (both parameters unknown), we test \(H_0: \theta = \theta_0\). We have already encountered this model in example Example 1 about the coffee machine:
\[ \overrightarrow {X} = \overrightarrow {1}\cdot \theta + \overrightarrow {\varepsilon }, \qquad \tilde{k} = 1, \quad \mathbf{w} = 1, \quad b = \left(\overrightarrow {1}^T\overrightarrow {1}\right)^{-1} = \frac{1}{n}, \quad \widehat{\tau } = \widehat{\theta } = \overline{X}. \]
Here \(\widehat{\sigma^2} = \operatorname {RSS}/(n-1) = s^2\) is the unbiased sample variance, and theorem Theorem 2 gives the statistic familiar from the chapter on confidence intervals
\[ T = \frac{\overline{X} - \theta _0}{\sqrt{s^2/n}} \sim T_{n-1} \quad \text{under } H_0. \]
In example Example 1 we already saw it in action (and the relation \(F = T^2\)).
In the “Analysis ToolPak” there is no separate button for the one-sample test: it is obtained from the paired test below if a column of constants \(\theta_0\) is supplied as the second “sample”.
3.3 Paired \(t\)-test
The tool “Paired two-sample \(t\)-test for means”. The data are \(n\) pairs \((X_i, Y_i)\): two measurements on the same object (before/after treatment, left/right eye, two instruments on the same part). Within a pair the measurements are dependent, so the two-independent-samples model does not apply. Instead, let us pass to the differences:
\[ D_i := X_i - Y_i = \delta + \varepsilon _i, \qquad \varepsilon _i \sim \mathscr {N}\left(0, \sigma _D^2\right) \text{ independent}, \]
and the hypothesis “there is no effect” becomes \(H_0: \delta = \delta_0\) (usually \(\delta_0 = 0\); Excel allows specifying a “hypothesized mean difference”). This is the one-sample \(t\)-test for \(\overrightarrow {D}\):
\[ T = \frac{\overline{D} - \delta _0}{\sqrt{s_D^2/n}} \sim T_{n-1} \quad \text{under } H_0, \qquad s_D^2 = \frac{1}{n-1}\sum _{i=1}^n \left(D_i - \overline{D}\right)^2. \]
What is the gain of the paired scheme? If each object has its own individual level \(\mu_i\), i.e. \(X_i = \mu_i + \xi_i\), \(Y_i = \mu_i - \delta + \eta_i\), then in the differences \(D_i = \delta + (\xi_i - \eta_i)\) the individual levels cancel out: the between-object spread of \(\mu_i\) does not enter the variance of the statistic.
3.4 Two-sample \(t\)-test: equal variances
The tool “Two-sample \(t\)-test assuming equal variances”. The data are two independent samples with a common unknown variance:
\[ X^{(1)}_1, \ldots , X^{(1)}_{n_1} \sim \mathscr {N}\left(a_1, \sigma ^2\right), \qquad X^{(2)}_1, \ldots , X^{(2)}_{n_2} \sim \mathscr {N}\left(a_2, \sigma ^2\right); \]
we test \(H_0: a_1 - a_2 = \delta_0\) (usually \(\delta_0 = 0\): “the means are equal”).
This is exactly the model from example Example 3 about weighing coins: \(\tilde{k} = 2\), \(\mathbf{w} = (1, -1)^T\), and, as computed in example Example 5,
\[ \widehat{\tau } = \overline{X^{(1)}} - \overline{X^{(2)}}, \qquad b = \frac{1}{n_1} + \frac{1}{n_2}, \qquad \widehat{\sigma ^2} = \frac{\operatorname {RSS}}{n_1 + n_2 - 2} = \frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2} =: s_p^2, \]
where \(s_i^2\) are the unbiased sample variances. The estimator \(s_p^2\) is called the pooled variance: both samples are combined into a common “pool” for estimating \(\sigma^2\). By theorem Theorem 2
\[ T = \frac{\overline{X^{(1)}} - \overline{X^{(2)}} - \delta _0}{s_p\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}} \sim T_{n_1 + n_2 - 2} \quad \text{under } H_0, \]
and the \(F\)-statistic from example Example 5 is its square (for \(\delta_0 = 0\)).
3.4.1 (*) Effect size: Cohen’s \(d\)
The \(p\)-value only answers the question “is there an effect” (more precisely, whether the data are compatible with “no effect”), but says nothing about how large the effect is. Because of this, a small, practically insignificant effect with a large sample yields just as small a \(p\)-value as a large effect with a small sample — statistical significance mixes up the effect size with the sample size. The standard way to separate one from the other is to convert the difference of means into dimensionless units by dividing it by the spread of the data.
By construction, \(\widehat{d}\) is the difference of means measured in “number of standard deviations”: \(\widehat{d} = 1\) means that the means differ by one \(s_p\). Unlike the \(T\)-statistic, \(\widehat{d}\) does not depend explicitly on the sample sizes \(n_1, n_2\) — it is a feature of the data themselves (of the effect size), not of the precision of its measurement. This is exactly why, as the sample grows, the \(T\)-statistic and the associated \(p\)-value can grow arbitrarily, while \(\widehat{d}\) (for the same true effect) stays put.
J. Cohen proposed conventional benchmarks for interpreting \(\left|d\right|\) in the behavioral and social sciences: \(d \approx 0.2\) is a small effect, \(d \approx 0.5\) a medium one, \(d \approx 0.8\) and above a large one. These thresholds are conventional and depend on the subject area — in physics, for example, effects of \(d = 0.2\) are usually of no interest, while in psychology they are considered quite noticeable.
Cohen’s effect size \(\hat d = \dfrac{\overline{X}^{(1)}-\overline{X}^{(2)}}{s_p}\) for the A/B test (old and new checkout form): two Gaussian distributions shifted by \(\hat d\) standard deviations. Move the \(d\) slider to see how the overlap of the distributions changes with the effect size, and how this same quantity relates to intuitive metrics: the overlap fraction (% overlap), the probability of superiority and the NNT. The default value \(d\approx1.33\) is exactly the one from the example in the text.
% overlap (OVL)
probability of superiority
Cohen's U₃
NNT
3.5 Welch’s two-sample \(t\)-test: unequal variances
The tool “Two-sample \(t\)-test assuming unequal variances”. If \(\sigma_1^2 \neq \sigma_2^2\), then homoscedasticity — the key condition of our linear model (\(\operatorname {Var}\left[\overrightarrow {\varepsilon }\right] = \sigma^2\mathbf{I}_n\)) — is violated, and the entire machinery of this chapter stops working. There is no exact analogue of theorem Theorem 2 here (this is the famous Behrens–Fisher problem), but there is a good approximation — the Welch test:
\[ T_W = \frac{\overline{X^{(1)}} - \overline{X^{(2)}} - \delta _0}{\sqrt{\dfrac {s_1^2}{n_1} + \dfrac {s_2^2}{n_2}}} \approx T_{\nu }, \qquad \nu = \frac{\left(\dfrac {s_1^2}{n_1} + \dfrac {s_2^2}{n_2}\right)^2}{\dfrac {(s_1^2/n_1)^2}{n_1 - 1} + \dfrac {(s_2^2/n_2)^2}{n_2 - 1}} \]
(the Welch–Satterthwaite approximation: a number of degrees of freedom \(\nu\) is chosen so that Student’s distribution best approximates the true distribution of the denominator). The number \(\nu\) is usually non-integer, \(\min (n_1, n_2) - 1 \leq \nu \leq n_1 + n_2 - 2\); Excel’s output rounds it to an integer.
Practical remarks. When \(s_1^2 \approx s_2^2\) and \(n_1 \approx n_2\), the statistics and conclusions of the two two-sample tests almost coincide. If, however, the variances differ noticeably (especially with different \(n_i\)), the test with pooled variance can badly misstate the significance level, while Welch’s test remains approximately honest. Hence the modern practical recommendation: if in doubt about equality of variances, use Welch’s test right away.
3.6 \(z\)-tests: known variances
The tool “Two-sample \(z\)-test for means”. If the variances \(\sigma_1^2, \sigma_2^2\) are known (Excel asks you to enter them by hand), then there is no need to estimate \(\sigma^2\), no Studentization is required, and under \(H_0: a_1 - a_2 = \delta_0\) the statistic has an exact normal distribution:
\[ Z = \frac{\overline{X^{(1)}} - \overline{X^{(2)}} - \delta _0}{\sqrt{\dfrac {\sigma _1^2}{n_1} + \dfrac {\sigma _2^2}{n_2}}} \sim \mathscr {N}\left(0, 1\right). \]
Note: here \(\sigma_1^2 \neq \sigma_2^2\) is allowed — the problem of the previous section arose only from estimating the variances. The one-sample variant is analogous: \(Z = \sqrt{n}\left(\overline{X} - \theta_0\right)/\sigma \sim \mathscr {N}\left(0, 1\right)\).
When can the variance be considered known? Either it is specified in the measuring instrument’s specification sheet (calibrated by the manufacturer on a huge amount of material), or the samples are large enough that \(s_i^2\) can be substituted for \(\sigma_i^2\): then the \(z\)-test is asymptotic (by the CLT it, incidentally, does not even require normality of the samples), and the \(t\)-tests for large \(n\) practically coincide with it (\(T_n \xrightarrow [n \to \infty ]{d} \mathscr {N}\left(0, 1\right)\)).
3.7 \(F\)-test for equality of variances
The tool “Two-sample \(F\)-test for variances”. Here the hypothesis is no longer about the mean, but about the variances of two independent normal samples: \(H_0: \sigma_1^2 = \sigma_2^2\). From the chapter on linear regression we know (essentially — from Cochran’s theorem for the constant-only model) that
\[ \frac{(n_1 - 1)s_1^2}{\sigma _1^2} \sim \chi ^2_{n_1 - 1}, \qquad \frac{(n_2 - 1)s_2^2}{\sigma _2^2} \sim \chi ^2_{n_2 - 1}, \]
and these quantities are independent (the samples are independent). By definition Definition 1 of the Fisher distribution, under \(H_0\)
\[ F = \frac{s_1^2}{s_2^2} = \frac{\frac{(n_1-1)s_1^2}{\sigma _1^2}\Big/(n_1 - 1)}{\frac{(n_2-1)s_2^2}{\sigma _2^2}\Big/(n_2 - 1)} \sim \operatorname {F}_{n_1 - 1, n_2 - 1}. \]
The alternative is two-sided (\(\sigma_1^2 \neq \sigma_2^2\)), so we reject if \(F \not\in \left(Q_{\operatorname {F}_{n_1-1,n_2-1}}\left(\frac{\alpha }{2}\right), Q_{\operatorname {F}_{n_1-1,n_2-1}}\left(1 - \frac{\alpha }{2}\right)\right)\). Excel gives a one-sided \(p\)-value \(\mathbb {P}\left(\operatorname {F} > F\right)\) — for a two-sided alternative it needs to be doubled (or, as is often done, the larger variance is put in the numerator in advance).
Two caveats. First, this test is often used to “check equality of variances” in order to choose between the two two-sample \(t\)-tests; such a two-stage procedure distorts the resulting significance level, and it is simpler to just use Welch’s test right away. Second, unlike the \(t\)-tests (fairly robust to non-normality of the samples thanks to the CLT), the \(F\)-test for variances is very sensitive to deviations from normality even for large \(n\).
3.8 Analysis of variance (ANOVA)
“One-way analysis of variance” is exactly problem Problem 8: \(k\) independent groups of observations with a common variance, the homogeneity hypothesis \(H_0: \mu_1 = \ldots = \mu_k\) (\(m = k - 1\) linear constraints), the statistic
\[ F = \frac{\operatorname {SS}_{\text{Treatments}}/(k-1)}{\operatorname {SS}_{\text{Error}}/(N-k)} \sim \operatorname {F}_{k-1, N-k} \quad \text{under } H_0 \]
(between-group spread against within-group spread). Excel outputs a table with \(\operatorname {SS}\), \(\operatorname {df}\), \(\operatorname {MS} = \operatorname {SS}/\operatorname {df}\), \(F\), the \(p\)-value, and the critical value.
Two-way analysis of variance generalizes the picture: observations are classified by two factors at once (for example, crop yield: variety \(\times\) fertilizer). In the balanced model “with replication,” \(r\) observations are made in each cell \((i, j)\):
\[ X_{ijl} = \mu + \alpha _i + \beta _j + \gamma _{ij} + \varepsilon _{ijl}, \qquad i = \overline{1,a}, \quad j = \overline{1,b}, \quad l = \overline{1,r}, \]
where \(\alpha_i\) is the effect of level \(i\) of the first factor, \(\beta_j\) of the second, \(\gamma_{ij}\) their interaction, and for identifiability the constraints \(\sum_i \alpha_i = \sum_j \beta_j = \sum_i \gamma_{ij} = \sum_j \gamma_{ij} = 0\) are imposed. This is again a linear Gaussian model, and the hypotheses “the first factor has no effect” (\(H_0^{A}\): all \(\alpha_i = 0\)), “the second has no effect” (\(H_0^{B}\)), “there is no interaction” (\(H_0^{AB}\)) are linear hypotheses of the form \(\mathbf{w}^T\overrightarrow {\theta } = \overrightarrow {0}\), to which the general theory applies. Let us write out the answer (the derivation is a direct, though cumbersome, application of the scheme of this chapter via the constrained least squares estimator; we leave it to the reader as an exercise of increased effort). In terms of the cell means \(\overline{X}_{ij\cdot }\), the factor-level means \(\overline{X}_{i\cdot \cdot }, \overline{X}_{\cdot j \cdot }\) and the overall mean \(\overline{X}\):
\[ \begin{align} \operatorname {SS}_{A} & = br\sum _{i=1}^a \left(\overline{X}_{i\cdot \cdot } - \overline{X}\right)^2, & \operatorname {SS}_{B} & = ar\sum _{j=1}^b \left(\overline{X}_{\cdot j\cdot } - \overline{X}\right)^2, \\ \operatorname {SS}_{AB} & = r\sum _{i=1}^a\sum _{j=1}^b \left(\overline{X}_{ij\cdot } - \overline{X}_{i\cdot \cdot } - \overline{X}_{\cdot j\cdot } + \overline{X}\right)^2, & \operatorname {SS}_{E} & = \sum _{i,j,l} \left(X_{ijl} - \overline{X}_{ij\cdot }\right)^2, \end{align} \]
and under the respective hypotheses
\[ F_A = \frac{\operatorname {SS}_{A}/(a-1)}{\operatorname {SS}_{E}/(ab(r-1))} \sim \operatorname {F}_{a-1,\, ab(r-1)}, \qquad F_B \sim \operatorname {F}_{b-1,\, ab(r-1)}, \qquad F_{AB} \sim \operatorname {F}_{(a-1)(b-1),\, ab(r-1)} \]
(the statistics \(F_B, F_{AB}\) are built from \(\operatorname {SS}_{B}, \operatorname {SS}_{AB}\) analogously). The tool “without replication” is the case \(r = 1\): the interaction is then indistinguishable from noise (there is one observation per cell), so the model is taken to be additive (\(\gamma_{ij} = 0\)), and the role of \(\operatorname {SS}_{E}\) is played by the residual sum \(\operatorname {SS}_{\text{res}} = \sum_{i,j}\left(X_{ij} - \overline{X}_{i\cdot } - \overline{X}_{\cdot j} + \overline{X}\right)^2\) with \((a-1)(b-1)\) degrees of freedom.
3.9 The “Regression” tool
The last and most substantive tool of the “Analysis ToolPak” fits a model with an intercept \(\overrightarrow {X} = \mathbf{z}\overrightarrow {\theta } + \overrightarrow {\varepsilon }\), \(\tilde{k} = k + 1\), and outputs three blocks, all assembled from the results of the chapters on linear regression and on hypothesis testing. Denote, as usual, \(\left(\mathbf{z}^T\mathbf{z}\right)^{-1} = (a_{i,j})\), \(\widehat{\sigma^2} = \operatorname {RSS}/(n - \tilde{k})\).
Block “Regression Statistics”: the coefficient of determination “\(R\) Square” \(R^2 = \operatorname {ESS}/\operatorname {TSS}\); “Multiple \(R\)” \(= \sqrt{R^2}\) (as we will see in the chapter on correlation coefficients, this is the sample correlation between \(\overrightarrow {X}\) and \(\widehat{\overrightarrow {X}}\)); “Standard Error” \(= \sqrt{\widehat{\sigma^2}}\); “Adjusted \(R\) Square”
\[ R^2_{adj} = 1 - (1 - R^2)\frac{n-1}{n - \tilde{k}} \]
— a version of \(R^2\) penalized for the number of regressors: ordinary \(R^2\) never decreases when any regressor is added, even a useless one, whereas \(R^2_{adj}\) can decrease (unbiased estimators of the residual and response variances are used instead of biased ones).
Block “ANOVA”: rows “Regression” / “Residual” / “Total” with \(\operatorname {df} = k, \; n - k - 1, \; n-1\) and \(\operatorname {SS} = \operatorname {ESS}, \operatorname {RSS}, \operatorname {TSS}\); the statistic
\[ F = \frac{\operatorname {ESS}/k}{\operatorname {RSS}/(n - k - 1)} \sim \operatorname {F}_{k, n-k-1} \quad \text{under } H_0: \theta _1 = \ldots = \theta _k = 0 \]
— the \(F\)-test of overall significance of the regression (“do the regressors explain anything at all”), familiar to us from problem Problem 2 (there also — its expression in terms of \(R^2\)); “Significance \(F\)” is its \(p\)-value.
Table of coefficients, one row per parameter \(\theta_j\) (including the intercept — “\(Y\) Intercept”): the estimator \(\widehat{\theta }_j\); its standard error \(\operatorname {se}_j := \sqrt{a_{j,j}\widehat{\sigma^2}}\); “\(t\) Stat”
\[ t_j = \frac{\widehat{\theta }_j}{\operatorname {se}_j} \sim T_{n - \tilde{k}} \quad \text{under } H_0: \theta _j = 0 \]
— a test of significance of an individual coefficient: this is theorem Theorem 2 with \(\mathbf{w} = \overrightarrow {e}_j\), \(\tau_0 = 0\), \(b = a_{j,j}\); the (two-sided) “\(P\)-value” and the bounds of the confidence interval \(\widehat{\theta }_j \pm Q_{T_{n - \tilde{k}}}\left(1 - \frac{\alpha }{2}\right)\operatorname {se}_j\) at confidence level \(95\%\) (coincides with the interval from the chapter on linear regression).
ExampleExample 16
Run the taxi ride data from example Example 2 through the “Regression” tool and read the output.
SolutionSolution
The tool’s output (all numbers recomputed from the formulas above):
| Coeff. | Std. error | \(t\)-stat. | \(P\)-value | Lower\(95\%\) | Upper\(95\%\) | |
|---|---|---|---|---|---|---|
| \(Y\)-intercept | \(108.27\) | \(16.71\) | \(6.48\) | \(0.0013\) | \(65.31\) | \(151.23\) |
| \(d\) (km) | \(32.08\) | \(1.90\) | \(16.92\) | \(1.0 \cdot 10^{-5}\) | \(27.20\) | \(36.95\) |
Let us read it.
The model explains the data excellently: \(R^2 \approx 0.983\) (and \(R^2_{adj} \approx 0.979\) is not far behind), the typical spread of price around the line — “Standard Error” \(\sqrt{\widehat{\sigma^2}} \approx 21.4\) rubles.
“Significance \(F\)” \(\approx 10^{-5}\): the hypothesis “distance has no effect on price at all” (\(\theta_1 = 0\)) is soundly rejected — as expected. Note: for \(k = 1\) this \(F\)-test is equivalent to the \(t\)-test of the single coefficient, and indeed \(F = 286.25 = 16.92^2 = t_1^2\), and the \(p\)-values coincide.
Looking at the coefficient rows: both are significantly different from zero. The confidence interval for the pickup fee \((65.3, \; 151.2)\) covers the advertised \(100\) rubles, and the interval for the per-kilometer price \((27.2, \; 36.9)\) covers the advertised \(30\) rubles/km. That is, coefficient by coefficient there is no complaint about the fare.
Compare with example Example 2, where the joint hypothesis \((a, b) = (100, 30)\) gave \(p \approx 6.2\%\) — much closer to the boundary. There is no contradiction: the joint \(F\)-test looks at both parameters at once (its “confidence region” is an ellipse in the \((a,b)\) plane, accounting for the negative correlation between the estimators \(\widehat{a}\) and \(\widehat{b}\)), and the point \((100, 30)\) lies close to the boundary of this ellipse, even though it falls freely into the rectangle formed by the two separate intervals. We saw the reverse effect in problem Problem 7 about cubic regression: there each coefficient is insignificant by the \(t\)-test (multicollinearity inflates the standard errors), while the joint \(F\)-test does see the effect. Moral: coefficient-by-coefficient \(t\)-tests and the joint \(F\)-test answer different questions, and one must not substitute one for the other.
3.10 Summary
Let us gather the section’s glossary into one table (\(s^2\), \(s_i^2\), \(s_D^2\) are unbiased sample variances, \(s_p^2\) is the pooled variance, \(\widehat{\sigma^2} = \operatorname {RSS}/(n - \tilde{k})\)):
| Test | \(H_0\) | Statistic | Distribution under\(H_0\) |
|---|---|---|---|
| One-sample\(t\) | \(\theta = \theta _0\) | \(\dfrac {\samplemean {X} - \theta _0}{\sqrt{s^2/n}}\) | \(T_{n-1}\) |
| [2ex] Paired\(t\) | \(\delta = \delta _0\) | \(\dfrac {\samplemean {D} - \delta _0}{\sqrt{s_D^2/n}}\) | \(T_{n-1}\) |
| [2ex] Two-sample\(t\) (equal var.) | \(a_1 - a_2 = \delta _0\) | \(\dfrac {\samplemean {X^{(1)}} - \samplemean {X^{(2)}} - \delta _0}{s_p\sqrt{1/n_1 + 1/n_2}}\) | \(T_{n_1 + n_2 - 2}\) |
| [2ex] Two-sample Welch\(t\) | \(a_1 - a_2 = \delta _0\) | \(\dfrac {\samplemean {X^{(1)}} - \samplemean {X^{(2)}} - \delta _0}{\sqrt{s_1^2/n_1 + s_2^2/n_2}}\) | \(\approx T_{\nu }\) |
| [2ex]\(z\)-test (known variances) | \(a_1 - a_2 = \delta _0\) | \(\dfrac {\samplemean {X^{(1)}} - \samplemean {X^{(2)}} - \delta _0}{\sqrt{\sigma _1^2/n_1 + \sigma _2^2/n_2}}\) | \(\NormalDistribution {0}{1}\) |
| [2ex]\(F\)-test for variances | \(\sigma _1^2 = \sigma _2^2\) | \(s_1^2/s_2^2\) | \(\operatorname {F}_{n_1 - 1, n_2 - 1}\) |
| [2ex] One-way ANOVA | \(\mu _1 = \ldots = \mu _k\) | \(\dfrac {\operatorname {SS}_{\text{Tr}}/(k-1)}{\operatorname {SS}_{E}/(N-k)}\) | \(\operatorname {F}_{k-1, N-k}\) |
| [2ex] Two-way ANOVA | no factor effect | \(\operatorname {MS}_{\text{factor}}/\operatorname {MS}_{E}\) | \(\operatorname {F}\) |
| [2ex] Regression:\(t\) for a coefficient | \(\theta _j = 0\) | \(\estim {\theta }_j / \sqrt{a_{j,j}\estim {\sigma ^2}}\) | \(T_{n - \tilde{k}}\) |
| [2ex] Regression:\(F\) for significance | \(\theta _1 = \ldots = \theta _k = 0\) | \(\dfrac {\operatorname {ESS}/k}{\operatorname {RSS}/(n-k-1)}\) | \(\operatorname {F}_{k, n-k-1}\) |
| [1ex] |
The same tests under the same names live in every statistical package: in python these are scipy.stats.ttest_1samp, ttest_rel (paired), ttest_ind (two-sample; the equal_var flag switches between pooled variance and Welch), f_oneway (one-way analysis of variance), and statsmodels OLS (the full regression output); in R — t.test, var.test, aov, lm/summary.