Hypothesis Testing

Published

18 October 2026

Draft

This note is still being edited: the text may contain typos, inaccuracies, and locally unpolished passages. If you find something, please let the author know (contact information on the course page).

1 Introduction

If, when conducting experiments, a researcher has no expectations about the possible values of the model’s parameters, then the methods of the previous chapters let them answer many of their questions: how to estimate a parameter, what properties such estimators may have, how to construct the best estimator, how to construct a confidence interval for the parameter, and so on. However, in many applications we do not need to estimate the value of the parameter $\theta$ itself.

For example, suppose a potential new cold medicine has come in for study, and we need to assess its effectiveness. Suppose we know that, on average, a patient with a cold who takes no medication recovers thanks to their immune system in $\theta_0 := 6$ days with standard deviation $\sigma_0 := \sqrt{6} \approx 2.5$ days. The pharmaceutical chemists who developed the new drug claim that, when taken, a person recovers on average in $\theta_1 := 4$ days.

Suppose that for the study $100$ people with a cold at an early stage were randomly selected, and they took the new medicine for the duration of their illness (but no more than $13 \approx \theta_0 + 3\sigma_0$ days; if the illness had not subsided within that period, the patient was switched to a treatment of proven effectiveness). In the end, nobody was sick for more than $13$ days, and patients recovered in $4.8$ days on average.

$4.8$ days is less than $6$, but as researchers we need to understand whether this difference is significant — whether it is really an effect of the medicine. Or perhaps this difference of $1.2$ days can simply be attributed to chance, and in reality taking the new medicine has no effect at all.

In the language of mathematical statistics, the problem can be formulated as follows: let $\mathbb {P}_\theta$ be the distribution of the number of days required for recovery when taking the new drug, where $\theta$ is the expectation of this distribution. Let a sample $X_1, \ldots , X_n$ from the distribution $\mathbb {P}_\theta$ be given. We need to use the sample to answer a binary question:

either $\theta = 6$ (taking the medicine has no effect);
or $\theta = 4$ (taking the medicine has the effect described by the chemists who created it).

More generally, one tests $\theta \in \Theta_0$ against $\theta \in \Theta_1$, where $\Theta_0$ must not intersect $\Theta_1$.

Two disjoint assumptions about the true distribution are called (opposing) hypotheses about the distribution and are denoted $H_0$, $H_1$. Statistical hypothesis testing is an investigation attempting to answer the question of whether $H_0$ or $H_1$ is correct.

1.1 Different effects of errors, different hypotheses. Singling out the base hypothesis

When testing statistical hypotheses one can make 2 different kinds of errors: either claiming that hypothesis $H_1: \theta \in \Theta_1$ holds when in fact $H_0: \theta \in \Theta_0$ holds (a type I error), or the reverse (a type II error). As a rule, these errors are not equivalent for the study — one is more dangerous than the other.

For example, in a study of a new drug: at present there is a mass of cold medicines with proven effectiveness, and if the researcher says that the new drug does not work when it actually does, this will not be critical for future patients, since instead of the new medicine they can turn to proven older ones and reliably recover in a short time. But if the researcher claims that a useless drug is effective, and this drug appears on pharmacy shelves, then future patients will be at risk, since they will be buying a useless medicine expecting a quick cure.

Since the errors are not equivalent for the researcher, the hypotheses cannot be equivalent either: we should be inclined to accept the hypothesis whose mistaken rejection would be more critical for us. This hypothesis is usually denoted $H_0$ and called the null (base) hypothesis, and the opposite one, $H_1$, the alternative. In the case of medical studies, as a rule, the null hypothesis is that the new drug (or treatment method) is ineffective, and the alternative is that it is effective.

		Null hypothesis ($H_0$)
		True	False
Decision on the null hypothesis ($H_0$)	Not rejected	Correct decision	Type II error
	Rejected	Type I error	Correct decision

Error table: possible outcomes of a statistical hypothesis test.

1.2 Significance of data, p-value, significance level

Testing statistical hypotheses proceeds as follows: the researcher initially, before the study, considers the null hypothesis $H_0$ to be true. Then they obtain data and examine it, trying to answer the question: is the data critical enough for $H_0$ to reject the null hypothesis $H_0$ in favor of the alternative $H_1$?

How can we numerically assess how critical the data is? Let us consider the example above, about the study of a new drug.

Example 1

Recovery time is most simply modeled by an exponential distribution. Let $X_1, \ldots , X_n, \; n = 100$ be a sample from the exponential distribution $\operatorname {Exp}\left(\frac{1}{\theta }\right), \; \theta \in \left\{ 4,6\right\}$ (i.e. $\mathbb {E}\left[X_1\right] = \theta$). Suppose a specific experiment $\omega_0$ was carried out, with \[ \overline{X}(\omega _0) = \frac{1}{n}\sum _{i=1}^n X_i(\omega _0) = 4.8 \] Let the null hypothesis be $H_0: \theta = \theta_0 = 6$, and the alternative $H_1: \theta = \theta_1 = 4$. Assess how critical the data in experiment $\omega_0$ is, based on the statistic $\overline{X}$.

Solution

Density of the statistic $\overline{X} \sim \Gamma(n, n/\theta_0)$ under $H_0$ ($\theta_0=6$), the observed value $\overline{X}(\omega_0)=4.8$ and the p-value $P(\overline{X}\le 4.8 \mid H_0)$ – the shaded area on the left. Drag the vertical line of the observed value (or move the slider) to see how the p-value changes along with the data; the $n$ and $\theta_0$ sliders change the shape of the density.

x̄(ω₀) = 4.8 n = 100 θ₀ = 6.0

p-value $P(\overline{X}\le\text{x̄}(\omega_0)\mid H_0)$ =

The $p$-value itself is a random variable: from experiment to experiment it changes along with the sample. It is useful to think not of a single value of it, but of its distribution over many repetitions of the experiment.

The distribution of the $p$-value over many repeated experiments (simulation: on each repetition a new sample, and hence a new $\overline{X}$, is generated, and the p-value with respect to $H_0: \theta=\theta_0$ is computed for it). When $H_0$ is true, it is uniform on $[0,1]$ – this is built into the very definition of the p-value. When $H_0$ is false (the true value is $\theta_1 \ne \theta_0$), the distribution shifts toward zero – the more so, the greater the power of the test (larger $n$ or larger difference $|\theta_1-\theta_0|$). Switch the mode and move the sliders to see this dependence.

n = 30

fraction of p ≤ 5%:

Initially we assume that $H_0$ is true, i.e. that $\theta = 6$. We need to understand how critical the data we obtained is — its statistical significance. Let us try to understand this based on the statistic $\overline{X}$.

$\theta$ is the expectation. The closer the sample mean is to $\theta_1$ compared to $\theta_0$, the more critical the data. In our case the parameter in the alternative hypothesis $\theta_1$ is smaller than the parameter $\theta_0$ in the null hypothesis, so the smaller the sample mean, the more critical the data.

Practical meaning: the less time it takes to recover while taking the new drug, the more improbable this is if we assume that the drug does not work.

How can we numerically express how critical the data is? From the properties of the gamma distribution we know that $\overline{X} \sim \Gamma (n, n/\theta )$, i.e. under $H_0$ we have $\overline{X} \sim \Gamma (100, 100/6)$. We have

\[ \mathbb {P}_{\theta _0}\left(\overline{X} \leq 4.8\right) \approx 1.7\% \]

This number is called the p-value of experiment $\omega_0$. What is its meaning? We obtained a value of the statistic $\overline{X}$ during the experiment. The p-value is the probability of obtaining, in a copy of the experiment, a value of the statistic that is the same or more critical, given that hypothesis $H_0$ holds.

The p-value can be used to assess how critical the data is under the assumption of hypothesis $H_0$.

Comparison of hypothetical recovery-time distributions: $H_0$ — $\operatorname{Exp}(1/\theta_0)$, $\theta_0=6$ (the drug has no effect), and $H_1$ — $\operatorname{Exp}(1/\theta_1)$, $\theta_1=4$ (the drug is effective). The dashed lines mark the means $\theta_0$, $\theta_1$ and the observed sample mean $\overline{X}(\omega_0)=4.8$.

Answer

p-value: $1.7\%$.

The data of an experiment is said to have statistical significance if obtaining the same data in an independent copy of the experiment would be extremely improbable, under the assumption that the null hypothesis holds. If the data is statistically significant, the null hypothesis must be rejected in favor of the alternative.

We obtained some p-value during the experiment. How can we use it to determine whether the data is significant, whether we need to reject hypothesis $H_0$? To resolve this question, before the experiment one sets some threshold for the p-value, called the significance level (notation: $\alpha$). The significance level sets the acceptable probability with which we are willing to reject $H_0$ when it is true. That is, the significance level is the acceptable probability of making a type I error. As a rule, the value $\alpha = 5\%$ is taken as the significance level, although in medical studies it may also be $1\%$, $0.1\%$, or $0.01\%$.

Example 2

Under the conditions of example Example 1, test $H_0$ against $H_1$ at significance levels $\alpha_1 = 5\%$, $\alpha_2 = 1\%$.

Solution

(via the p-value). p-value: $1.7\%$.

$1.7\% < 5\%$, hence the data is significant, critical enough under the assumption of $H_0$. The variability of the data cannot be attributed to chance. $H_0$ must be rejected in favor of $H_1$, and $H_1$ accepted;

$1.7\% > 1\%$, hence the data is not significant, not critical enough under the assumption of $H_0$. Its variability can be attributed to chance. $H_0$ should not be rejected.

Alternative solution (via critical values). We know the significance levels. Let $q_1$ and $q_2$ be the values of the statistic $\overline{X}$ whose criticality is exactly equal to $\alpha_1$ and $\alpha_2$:

\[ \mathbb {P}_{\theta _0}\left(\overline{X} \leq q_i\right) = \alpha _i, \quad i \in \left\{ 1,2\right\} \]

Given that the null hypothesis holds, $\overline{X} \sim \Gamma (100,100/6)$. So $q_i$ is the quantile of level $\alpha_i$ of this distribution. Let us find them:

\[ q_1 = 5.05, \quad q_2 = 4.69 \]

These values are called critical values of the statistic (in this case, of the sample mean). If the statistic is more critical than such a value (in this case, smaller), then $H_0$ should be rejected.

$\overline{X}(\omega_0) = 4.8 < 5.05 = q_1$, so at significance level $\alpha_1= 5\%$, $H_0$ should be rejected;

$\overline{X}(\omega_0) = 4.8 > 4.69 = q_2$, so at significance level $\alpha_2= 1\%$, $H_0$ should not be rejected.

Remark. Note that in the second case it is incorrect to say that we accept $H_0$, since we already accepted it initially, before the experiment. It is more correct to say that we did not reject $H_0$.

Answer

At a significance level of $5\%$, $H_0$ should be rejected in favor of $H_1$. At $1\%$, $H_0$ should not be rejected.

Testing statistical hypotheses is similar to a court trial. Practically every judicial system in the modern world follows the principle of presumption of innocence. Before the trial begins, the defendant is considered innocent; this is, one might say, the court’s null hypothesis $H_0$. Then, during the trial, the prosecutor presents evidence to the court indicating guilt. The judge’s task is to examine the evidence, determine whether the evidence is sufficient to reject the base assumption of the defendant’s innocence and accept the hypothesis of guilt, and render a verdict.

In mathematical statistics, the evidence is the sample, and the researcher must determine whether the evidence is critical enough to reject $H_0$ in favor of $H_1$.

Problem 1

Let $X_1, \ldots , X_n, \; n = 225$ be a sample from the normal distribution $\mathscr {N}\left(\theta , 1\right), \; \theta \in \left\{ 5,8\right\}$, where $\overline{X}(\omega_0) = 5.15$. Let the null hypothesis be $H_0: \theta = \theta_0 = 5$, and the alternative $H_1: \theta = \theta_1 = 8$. Find the p-value for experiment $\omega_0$, and test hypothesis $H_0$ against $H_1$ at significance levels $\alpha_1 = 5\%$, $\alpha_2 = 2\%$, $\alpha_3 = 1\%$.

Hint

Normal distribution.

2 General problem setup

2.1 Test, critical region

How can hypothesis testing be formalized in general? Let $\vec{X} = (X_1, \ldots , X_n)^T$ be a sample from an unknown distribution $\mathbb {P} \in \mathcal{P}$. Often we have a parametric model, in which case $\mathcal{P} = \left\{ \mathbb {P}_\theta , \; \theta \in \Theta \right\}$, although in general $\mathcal{P}$ can be any family of distributions.

Hypotheses:

in a parametric model

\[ H_0: \theta \in \Theta _0, \qquad H_1: \theta \in \Theta _1 \]

where $\Theta_0 \subset \Theta$, $\Theta_1 = \Theta_0^{\complement }$
in general

\[ H_0: \mathbb {P} \in \mathcal{P}_0, \qquad H_1: \mathbb {P} \in \mathcal{P}_1 \]

where $\mathcal{P}_0 \subset \mathcal{P}$, $\mathcal{P}_1 = \mathcal{P}_0^{\complement }$.

Going forward, for simplicity, we will mostly formulate definitions for the parametric case. But they are easily carried over to the general nonparametric case as well.

If a hypothesis involves only one parameter (one distribution), it is called simple. Otherwise the hypothesis is called composite. So far, in examples (Example 1) and problems (Problem 1) we tested only a simple null hypothesis against a simple alternative. Examples with composite hypotheses are given below.

The decision to reject or not reject $H_0$ in favor of $H_1$ can, as seen in example Example 1, be made in two ways: via the p-value and via critical values. Let us consider the second way, via critical values. We rejected $H_0$ if the sample mean was less than some value $q$. This method can also be phrased as follows: let

\[ S := \left\{ \vec{x} \in \mathbb {R}^{n} \; : \; \overline{x} \leq q\right\} \]

$S$ is some subset of $\mathbb {R}^{n}$ (consisting of those column vectors whose arithmetic mean of coordinates does not exceed $q$), and we reject $H_0$ if our sample $\vec{X}$ falls into $S$:

\[ \vec{X} \in S \; \Rightarrow \; \text{reject }H_0 \text{ at significance level }\alpha \]

$S$ is called the critical set.

In general it is not always clear which statistic to use when testing a hypothesis. Or, even if a statistic has been chosen, it is not always clear what “criticality” of the statistic means. For example, if in the example above we instead had not

\[ H_0: \theta \in \left\{ 6\right\} \text{ against } H_1: \theta \in \left\{ 4\right\} \]

but

\[ H_0: \theta \in \left\{ 3, 6\right\} \text{ against } H_1: \theta \in \left\{ 2,4,8\right\} \]

Which value of the sample mean is more critical here, and which is less? Here “criticality” cannot be defined.

Computing the p-value and the critical value is impossible without the notion of “criticality.” Therefore, in general, hypothesis testing is formalized precisely through critical sets: we reject $H_0$ if the sample $\vec{X}$ falls into some subset $S \subset \mathcal{X}$ of the space $\mathcal{X}$ of all possible sample values (as a rule, $\mathcal{X} = \mathbb {R}^{n}$). Otherwise we do not reject $H_0$. To reiterate, the region $S \subset \mathcal{X}$ is called the critical region or simply the test.

Definition 1 Let $S \subset \mathcal{X}$ be a test.

The power function of the test $S$ is the function

\[ \beta (\theta , S) := \mathbb {P}_{\theta }(\vec{X} \in S), \quad \theta \in \Theta \]

This is the probability of rejecting $H_0$ if the true parameter is $\theta$.
A quantity $\alpha$ is called a significance level of the test $S$ if $\alpha \geq \beta (\theta , S)$ for every $\theta \in \Theta_{0}$. Note that if $\theta \in \Theta_0$, then the null hypothesis $H_0$ holds. In that case $\beta (\theta , S)$ is the probability of rejecting a true $H_0$, i.e. of making a type I error. The significance level is thus an upper bound for the possible probabilities of type I errors. These probabilities can differ, since in general $\Theta_0$ contains several elements.
The minimal significance level under the null hypothesis

\[ \alpha _{0}=\sup _{\theta \in \Theta _{0}} \beta (\theta , S) \]

is called the size of the test $S$. The size of a test is the exact upper bound for the probabilities of type I errors.

The size of the test is often called the significance level and denoted $\alpha$.

Example 3

Let $X$ be a sample of size 1. Let $H_0: X \sim U[0,1], \; H_1: X \sim \mathrm{Exp}(1)$. Let $\alpha \in (0,1)$ be arbitrary. Construct all possible tests $S$ of significance level (size) $\alpha \in (0,1)$.

Solution

Here $\mathcal{P} = \left\{ U[0,1], \operatorname {Exp}(1)\right\}$, $\mathcal{P}_0 = \left\{ U[0,1]\right\} , \; \mathcal{P}_1 = \left\{ \operatorname {Exp}(1)\right\}$. Denote $\mathbb {P}_0 := U[0,1], \; \mathbb {P}_1 := \operatorname {Exp}(1)$.

Note that here the range of the sample values is $\mathcal{X} = \mathbb {R}_+$. Let $S \subset \mathcal{X}$ be a test. Since $\mathcal{P}_0 = \left\{ \mathbb {P}_0\right\}$ consists of a single distribution, the supremum over $\mathcal{P}_0$ is taken trivially: its significance level is

\[ \alpha = \sup _{\mathbb {P} \in \mathcal{P}_0} \beta (\mathbb {P}, S) = \beta (\mathbb {P}_0, S) = \mathbb {P}_{0}\left(X \in S\right). \]

The density of the distribution $\mathbb {P}_0 = U[0,1]$ equals $\; \mathbb {1}_{X \in [0,1]}$, so

\[ \mathbb {P}_{0}\left(X \in S\right) = \int _S \; \mathbb {1}_{X \in [0,1]}dx = \int _{S \cap [0,1]} dx = \lambda \left(S \cap [0,1]\right), \]

where $\lambda$ is Lebesgue measure.

Answer

$S$ is any set from $\mathscr {B}\left(\mathbb {R}_+\right)$ such that $\lambda (S \cap [0,1]) = \alpha$.

2.2 Power of a test

If the goal is to constrain the probability of a type I error as much as possible, then, generally speaking, it can be made zero by setting the significance level $\alpha$ to zero. At such a significance level we would have to stand by $H_0$ regardless of the data $\vec{X}$, and we would need to take $S = \varnothing$. In this case we would never reject $H_0$, since the sample would never fall into the critical set (as it is empty).

However, in every case where $H_1$ is actually true, we would be standing by $H_0$, making a type II error. The probability of a type II error in this case would be $100\%$.

There is an inverse relationship between the probabilities of type I and type II errors: the smaller one probability, the larger the other, and vice versa. There is no “best” test that would simultaneously minimize both type I and type II errors.

In practice, when searching for the “best” test, one first fixes the significance level (the probability of making a type I error), and then among all tests with the given significance level one searches for the best one from the point of view of the type II error.

Definition 2 The upper bound for the possible values of type II errors is denoted by the letter $\beta$. The notation coincides with that of the power function, but this usually causes no confusion.

Recall that a type II error occurs when $H_1$ is true but the researcher does not reject $H_0$, i.e. when $\vec{X} \not\in S$:

\[ \begin{align} \beta & := \sup _{\theta \in \Theta _{1}} \mathbb {P}_{\theta }\left(\vec{X} \not\in S\right) = \sup _{\theta \in \Theta _{1}} \left[ 1 - \mathbb {P}_{\theta }\left(\vec{X} \in S\right)\right] = \\ & = \sup _{\theta \in \Theta _{1}} \left[ 1 - \beta (\theta , S)\right] \end{align} \]

The quantity $1-\beta$ is called the power of the statistical test. Power is the ability of a test to detect an effect (hypothesis $H_1$), if that effect really exists. From a statistical point of view, it is the lower bound of the probability of correctly rejecting the null hypothesis:

\[ 1 - \beta = \inf _{\theta \in \Theta _{1}} \mathbb {P}_{\theta }\left(\vec{X} \in S\right) = \inf _{\theta \in \Theta _{1}} \beta (\theta , S) \quad \left(1 - \beta = \inf _{\mathbb {P} \in \mathcal{P}_{1}} \beta (\mathbb {P}, S) \text{ in general}\right) \]

Error table		Null hypothesis ($H_0$)
		True	False
Decision on the null hypothesis ($H_0$)	Not rejected	Correct decision True Negative (TN) $1-\alpha$	Type II error False Negative (FN) $\beta$
	Rejected	Type I error False Positive (FP) $\alpha$	Correct decision True Positive (TP) $1-\beta$

Error table showing the probabilities of type I and type II errors ($\alpha$, $\beta$) and the power of the test ($1-\beta$).

We test $H_0:\theta=0$ against $H_1:\theta=\theta_1$ using the statistic $\overline X$ for a sample of size $n$ from $\mathcal N(\theta,1)$, i.e. $\overline X\sim\mathcal N(\theta,1/n)$. Below is the full version of the $\alpha$/$\beta$/power/$n$/effect relationship, where any of the five parameters can be computed from the others: choose the “solve for” mode, set the remaining sliders, and the chosen parameter will be computed by bisection. The switch below toggles between a one-sided and a two-sided alternative.

solve for:

power 1−β

θ₁

Type I error (α)

Type II error (β)

Power

Example 4

Under the conditions of example Example 3, find the power of each test $S$. Find the most powerful test of significance level $\alpha \in (0,1)$.

Hint

Here $\mathcal{P}_1 = \left\{ \mathbb {P}_1\right\}$ contains only one distribution. The power computation simplifies: \[ \beta = \mathbb {P}_{1}\left(X \not\in S\right), \qquad 1 - \beta = \mathbb {P}_{1}\left(X \in S\right) \]

Solution

Let us compute the power of $S$, i.e. the probability of correctly rejecting $H_0$. The density of the distribution $\mathbb {P}_1 = \operatorname {Exp}(1)$ equals $e^{-x}$ on $\mathbb {R}_+$, so

\[ \beta (\mathbb {P}_1, S) = \mathbb {P}_{1}\left(X \in S\right) = \int _S e^{-x} dx. \]

The sets $[0,1]$ and $(1, +\infty )$ form a partition of $\mathcal{X} = \mathbb {R}_+$, so the integral over $S$ splits into 2 independent terms:

\[ \int _S e^{-x} dx = \int _{S \cap [0,1]}e^{-x} dx + \int _{S \cap (1, +\infty )}e^{-x} dx. \]

Note that the values of these 2 integrals do not depend on each other. Moreover, on $S \cap (1, +\infty )$ we have no restrictions at all. Hence, to maximize the power, we need to take $S$ such that $S \cap (1, +\infty ) = (1, +\infty )$.

Next, consider the integral $\int_{S \cap [0,1]}e^{-x} dx$. $e^{-x}$ is strictly decreasing, so, taking into account the constraint $\lambda (S \cap [0,1]) = \alpha$, the test $S$ must be taken such that $S \cap [0,1] = [0, \alpha ]$.

$H_0\!:X\sim U[0,1]$ against $H_1\!:X\sim\operatorname{Exp}(1)$. Most powerful test of significance level $\alpha$: $S=[0,\alpha]\cup(1,+\infty)$ (shown for $\alpha=0.3$). The violet region is the area under the density of $\operatorname{Exp}(1)$ over $S$, i.e. the power $1-\beta$ of the most powerful test.

Answer

$[0, \alpha ] \cup (1, +\infty )$.

Problem 2

Let $X$ be a sample of size 1 from a distribution $\mathbb {P}$. Let

$H_0: X \sim U[1,4], \; H_1: X \sim \mathscr {N}\left(3, 1\right)$;
$H_0: X \sim \mathscr {N}\left(0, 1\right), \; H_1: X \sim \mathscr {N}\left(1, 1\right)$. Let $\alpha \in (0,1)$ be arbitrary. Find the most powerful test of significance level $\alpha$.

Hint

Use the Neyman-Pearson lemma: the critical region should consist of the points where the likelihood ratio $f_1(x)/f_0(x)$ is largest.

Recall, a hypothesis is called simple if it involves only one parameter or one distribution. So far we have only dealt with testing simple null hypotheses against simple alternatives. In the next problem one must test a simple null hypothesis against a composite alternative.

Problem 3

Let $X$ be a sample of size 1 from a distribution $\mathbb {P}$. Let

$H_0: X \sim U[-3,3], \; H_1: \mathbb {P} \in \left\{ \operatorname {TR}[-2,0], \operatorname {TR}[0,2]\right\}$, where $\operatorname {TR}[a,b]$ is the triangular distribution with density

\[ p(t) = \begin{cases} 0, & x < a \\ \frac{2(x-a)}{(b-a)\left(\frac{b+a}{2} - a\right)}, & x \in [a, \frac{a+b}{2}), \\ \frac{2(b-x)}{(b-a)\left(b - \frac{b+a}{2}\right)}, & x \in [\frac{a+b}{2}, b), \\ 0, & x \geq b \end{cases} \]

$H_0: X \sim U[-3,3], \; H_1: \mathbb {P} \in \left\{ \mathscr {N}\left(-1, 1\right), \mathscr {N}\left(1, 1\right)\right\}$. Let $\alpha \in (0,1)$ be arbitrary. Find the most powerful test of significance level $\alpha$.

2.3 Uniformly most powerful tests

Suppose we have a composite alternative $H_1$: $\left|\Theta_1\right| > 1$. Recall that in this situation the power at different parameters $\theta \in \Theta_1$ from $H_1$ can turn out to be different. The power of the test itself is then defined as the infimum of these values. Tests of the same significance level can be compared based on this infimum. The best test is the one with the largest power.

In some cases one can find a test with a given significance level whose power is not merely the largest, but is uniformly over $\theta \in \Theta_1$ the largest.

Definition 3 If $S$ and $R$ are two tests of significance level (size) $\alpha$, then $S$ is uniformly more powerful than $R$ if

\[ \beta (\theta , S) \geq \beta (\theta , R) \quad \forall \theta \in \Theta _1 \]

This is equivalent to the condition on the probabilities of type II errors

\[ 1 - \beta (\theta , S) \leq 1 - \beta (\theta , R) \quad \forall \theta \in \Theta _1 \]

In other words, $S$ is uniformly more powerful than $R$ if, at every value of the true parameter from hypothesis $S$, the probability of a type II error for $H_1$ is smaller. Put differently: the probability of a type II error of test $S$ is smaller than the probability of a type II error of test $R$ uniformly over $\theta \in \Theta_1$.

A test $S$ is called a uniformly most powerful test (UMP test) of significance level $\alpha$ if it is uniformly more powerful than any other test of significance level $\alpha$.

Note that in the case of a simple alternative, speaking of a UMP test is meaningless.

Example 5

Let $\vec{X} = \left(X_{1}, \ldots , X_{n}\right)^T$ be a sample from the uniform distribution on the interval $[0, \theta ], \theta >0$. Find a UMP test for testing the hypothesis $H_{0}: \theta =\theta_{0}$ against the alternative $H_{1}: \theta <\theta_{0}$, if one exists.

Hint

Consider tests of the form $S_c = \left\{ \vec{x} \in \mathbb {R}^{n} \; : \; x_{(n)} \leq c \theta_0\right\}$. That is, we reject $H_0$ when the sample maximum is less than or equal to $c\theta_0$.

Solution

Let us find, among the tests $\left\{ S_c\right\}$, the test $S_{c_0}$ of the required significance level $\alpha$. We have

\[ \begin{align} \alpha & = \mathbb {P}_{\theta _0}\left(\vec{X} \in S_{c_0}\right) = \mathbb {P}_{\theta _0}\left(X_{(n)} \leq c_0\theta _0\right) = \left(\frac{c_0\theta _0}{\theta _0}\right)^n = \\ & = c_0^n \end{align} \]

Hence $c_{0}=\sqrt[n]{\alpha }$. The test:

\[ S_{c_0} = \left\{ \vec{x} \in \mathbb {R}_+^n: x_{(n)} \leq \sqrt[n]{\alpha } \theta _0\right\} \]

Let us prove that $S_{c_0}$ is uniformly more powerful than all other tests of significance level $\alpha$. Let $R$ be an arbitrary test of significance level $\alpha$.

Let the true parameter satisfy $\theta \leq c_0 \theta_0$. The power of $S_{c_0}$ equals 1, i.e. is no less than the power of any other test.
Let the true parameter satisfy $\theta \in (c_0\theta_0, \theta_0)$. The power of $S_{c_0}$:

\[ \mathbb {P}_{\theta }\left(\vec{X} \in S_{c_0}\right) = \mathbb {P}_{\theta }\left(X_{(n)} \leq c_0 \theta _0\right) = \alpha \left(\frac{\theta _0}{\theta }\right)^n \]

The power of an arbitrary test $R \subset \mathbb {R}_+^n$ of significance level $\alpha$:

\[ \begin{align} \mathbb {P}_{\theta }\left(\vec{X} \in R\right) & = \int _{[0,\theta ]^n} \frac{1}{\theta ^n} \; \mathbb {1}_{R}(\vec{x}) d\vec{x} =\frac{\theta _{0}^{n}}{\theta ^{n}} \int _{\left[0, \theta \right]^{n}} \frac{1}{\theta _{0}^{n}} \; \mathbb {1}_{R}(\vec{x}) d\vec{x} \leq \\ & \leq \frac{\theta _{0}^{n}}{\theta ^{n}} \int _{\left[0, \theta _0\right]^{n}} \frac{1}{\theta _{0}^{n}} \; \mathbb {1}_{R}(\vec{x}) d\vec{x} = \frac{\theta _{0}^{n}}{\theta ^{n}} \mathbb {P}_{\theta _0}\left(\vec{X} \in R\right) \leq \\ & \leq \alpha \frac{\theta _{0}^{n}}{\theta ^{n}} = \mathbb {P}_{\theta }\left(\vec{X} \in S_{c_0}\right) \end{align} \]

Proved.

The power function of the UMP test $S_{c_0} = \{x_{(n)} \leq c_0\theta_0\}$, $c_0 = \alpha^{1/n}$, for testing $H_0: \theta=\theta_0$ against $H_1: \theta<\theta_0$ from a sample from $U[0,\theta]$ ($\theta_0=1$): $\beta(\theta, S_{c_0}) = \min\!\big(1, (c_0\theta_0/\theta)^n\big)$. Move the sliders $n$ for the two curves and the shared $\alpha$ — both curves are recomputed. For $\theta \leq c_0\theta_0$ the test rejects $H_0$ with certainty (power $=1$); at the point $\theta=\theta_0$ the power equals exactly $\alpha$, regardless of $n$; the larger $n$, the sharper the drop of the curve near $\theta_0$.

n₁ = 10 n₂ = 30 α = 0.05

c₀(n₁) = c₀(n₂) =

Answer

$\left\{ \vec{x} \in \mathbb {R}_+^n: x_{(n)} \leq \sqrt[n]{\alpha } \theta_0\right\}$

Problem 4

$X_1, \dots , X_n$ is a sample from $U[0,\theta ], \; \theta > 0$. Construct a UMP test of significance level $\alpha$ for testing $H_0: \theta = \theta_0$ against $H_1: \theta \neq \theta_0$.

If $S$ is a uniformly most powerful test, then it is the most powerful. However, unlike the most powerful test, a UMP test does not always exist.

Problem 5

Prove that under the conditions of problem Problem 3, a UMP test does not exist, in either part a) or part b).

Problem 6

$X_1, \dots , X_n$ is a sample from $\mathrm{Bern}(\theta )$. Prove that there is no UMP test of any significance level $\alpha$ for testing the hypothesis $H_0: \theta = \theta_0 \in (0,1)$ against $H_1: \theta \neq \theta_0$.

3 Constructing the most powerful tests via the likelihood ratio

Let $\left\{ \mathbb {P}_{\theta }, \theta \in \Theta \right\}$ be a dominated family of distributions with densities $\left\{ f_\theta \right\}$, where the support of $\mathbb {P}_\theta$ does not depend on the parameter. The likelihood ratio is the ratio of the likelihood at the null parameter to the likelihood at the alternative parameter:

\[ \frac{L(\theta _0)}{L(\theta _1)} = \frac{L(\theta _0, \vec{X})}{L(\theta _1, \vec{X})} \]

3.1 Simple hypotheses

Let a simple hypothesis against a simple alternative be given:

\[ H_0: \theta = \theta _0, \qquad H_1: \theta = \theta _1. \]

Consider the likelihood ratio. If it is too small — less than some threshold $c$ — then the parameter $\theta_1$ is noticeably more plausible than $\theta_0$, and we should reject $H_0$. It turns out that a test constructed this way will be the most powerful.

Lemma 1 (Neyman-Pearson) Suppose there exists a number $c>0$ such that \[ \mathbb {P}_{\theta _{0}}\left(\frac{L(\theta _0, \vec{X})}{L(\theta _1, \vec{X})} < c\right) = \alpha . \] In other words, \[ \mathbb {P}_{\theta _0}\left(\vec{X} \in S\right) = \alpha . \] where \[ S := \left\{ \vec{x} \in \mathcal{X}: \frac{L(\theta _0, \vec{x})}{L(\theta _1, \vec{x})} < c\right\} \] Then the test $S \subset \mathcal{X}$ is a test of significance level $\alpha$ for testing $H_0$ against $H_1$ (this is obvious). Moreover, it is the most powerful test with this significance level (not obvious — this is the discovery of Neyman and Pearson).

Example 6

Let $X_{1}, \ldots , X_{n}$ be a sample from the distribution $\operatorname {Exp}(1/\theta )$ ($\mathbb {E}\left[X_1\right] = \theta$). Construct the most powerful test for testing the hypothesis $H_{0}: \theta =\theta_{0} \in (0,+\infty )$ against the alternative $H_{1}: \theta =\theta_{1} \in (0,+\infty )$, if $\theta_{0} > \theta_{1}$.

Solution

Note that $\theta_0 > \theta_1 \iff \frac{1}{\theta_0} < \frac{1}{\theta_1}$.

Density:

\[ f_{\theta }(x) = \frac{1}{\theta }e^{-\frac{1}{\theta }x} \cdot \; \mathbb {1}_{\mathbb {R}_+}(x) \]

Likelihood:

\[ \begin{align} L(\theta ) & = L(\theta , X_1,\ldots ,X_n) = \prod _{i=1}^{n}f_{\theta }(X_i) = \\ & = \frac{1}{\theta ^n} \cdot e^{-\frac{1}{\theta }n\overline{X}} \cdot \; \mathbb {1}_{\mathbb {R}_+}(X_{(1)}) \end{align} \]

Likelihood ratio:

\[ \frac{L(\theta _0)}{L(\theta _1)} = \left(\frac{\theta _1}{\theta _0}\right)^{n} \mathrm{exp}\left(n\overline{X}\left(\frac{1}{\theta _1} - \frac{1}{\theta _0}\right)\right) \]

By the Neyman-Pearson lemma, the most powerful test consists of the points where the likelihood ratio is less than $c$:

\[ S = \left\{ \vec{X} \in \mathbb {R}_+^n: \mathrm{exp}\left(n\overline{x}\left(\frac{1}{\theta _1} - \frac{1}{\theta _0}\right)\right) \left(\frac{\theta _1}{\theta _0}\right)^{n} < c\right\} . \]

Let us take the logarithm of the inequality inside the braces and move the term not depending on $\vec{x}$ to the right:

\[ n\overline{x}\left(\frac{1}{\theta _1} - \frac{1}{\theta _0}\right) < \ln c - n\ln \left(\frac{\theta _1}{\theta _0}\right) = \ln \left(c \cdot \left(\frac{\theta _0}{\theta _1}\right)^{n}\right). \]

Since $\theta_0 > \theta_1$, the coefficient $\frac{1}{\theta_1} - \frac{1}{\theta_0}$ is positive, and dividing by it does not change the direction of the inequality:

\[ S = \left\{ \vec{X} \in \mathbb {R}_+^n: n \overline{x} \; < \frac{1}{\frac{1}{\theta _1} - \frac{1}{\theta _0}} \ln \left(c \cdot \left(\frac{\theta _0}{\theta _1}\right)^{n} \right)\right\} . \]

For the test $S$ to have significance level $\alpha$, we require that $\mathbb {P}_{\theta_0}\left(\vec{X} \in S\right) = \alpha$:

\[ \alpha = \mathbb {P}_{\theta _0}\left(\vec{X} \in S\right) = \mathbb {P}_{\theta _0}\left(\overline{X} < \underbrace{\frac{1}{n}\frac{1}{\frac{1}{\theta _1} - \frac{1}{\theta _0}} \ln \left(c \cdot \left(\frac{\theta _0}{\theta _1}\right)^{n} \right)}_{ =: \tilde{c}}\right) \]

Recall that $\overline{X} \sim \Gamma \left(n, \frac{n}{\theta }\right)$. Hence $\tilde{c} = Q_{\Gamma \left(n, \frac{n}{\theta_0}\right)}(\alpha )$, where $Q_{\Gamma \left(n, n/\theta_0\right)}(\cdot )$ is the quantile function of the distribution $\Gamma \left(n, \frac{n}{\theta_0}\right)$.

From $\tilde{c}$ one could derive $c$, but this is unnecessary — the answer can be written directly in terms of $\tilde{c}$:

\[ S = \left\{ \vec{x} \in \mathbb {R}_+^n \; : \; \overline{x} < Q_{\Gamma \left(n, n/\theta _0\right)}(\alpha )\right\} . \]

Answer

$S = \left\{ \vec{x} \in \mathbb {R}_+^n: \overline{x} < Q_{\Gamma \left(n, n/\theta_0\right)}(\alpha )\right\}$.

*Remark.** It turns out that the tests based on the sample mean, which we used in examples Example 1, Example 2, were the most powerful tests of the corresponding significance level.

Problem 7

$X_1, \dots , X_n$ is a sample from $N(\theta , 1)$. Construct the most powerful test of significance level $\alpha$ for testing $H_0: \theta = \theta_0$ against $H_1: \theta = \theta_1$, if

$\theta_0 > \theta_1$;
$\theta_0 < \theta_1$. Compare the answer in part b) with the answer in part b) of problem Problem 2.

Each part is worth 0.5 points.

Problem 8

$X_1, \dots , X_n$ is a sample from $N(0, \theta )$. Construct the most powerful test of significance level $\alpha$ for testing $H_0: \theta = \theta_0$ against $H_1: \theta = \theta_1$, and find its power.

3.2 Composite hypotheses: constructing a UMP test

Suppose the model has a sufficient statistic $T(\vec{X})$. Then, by the Neyman-Fisher factorization criterion, we have

\[ L(\theta , \vec{X}) = \psi (T(\vec{X}), \theta )h(\vec{X}). \]

In that case the likelihood ratio takes the form

\[ \frac{L(\theta ', \vec{X})}{L(\theta '', \vec{X})} = \frac{\psi (T(\vec{X}), \theta ')}{\psi (T(\vec{X}), \theta '')} \]

Definition 4 The family $\left\{ \mathbb {P}_{\theta }, \theta \in \Theta \right\}$ has a nondecreasing likelihood ratio in the statistic $T(\vec{X})$ if the likelihood ratio

\[ \frac{L(\theta ', \vec{X})}{L(\theta '', \vec{X})} = \frac{\psi (T(\vec{X}), \theta ')}{\psi (T(\vec{X}), \theta '')} =: L_{\theta ',\theta ''}(T(\vec{X})) \]

is a nondecreasing function of $T$ for all $\theta '' < \theta '$ belonging to $\Theta$.

A nonincreasing likelihood ratio is defined analogously. In both cases the model is said to have a monotone likelihood ratio.

Theorem 1 Suppose a simple or composite null hypothesis is given against a composite alternative of the form

$H_0: \theta = \theta_0 \; (\text{or } \theta \geq \theta_0, \text{ it does not matter}), \qquad H_1: \theta < \theta_0$;
$H_0: \theta = \theta_0 \; (\text{or } \theta \leq \theta_0, \text{ it does not matter}), \qquad H_1: \theta > \theta_0$;

Let $\left\{ \mathbb {P}_{\theta }, \theta \in \Theta \right\}$ be a family with a nondecreasing likelihood ratio in the statistic $T(\vec{X})$. Let $\alpha \in (0,1)$. Suppose there exists $c$ such that

\[ \begin{array}{cc} \text{case 1)} & \mathbb {P}_{\theta _0}\left(T(\vec{X}) \leq c\right) = \alpha \\ \text{case 2)} & \mathbb {P}_{\theta _0}\left(T(\vec{X}) \geq c\right) = \alpha \end{array}.\qquad \text{Then the test} \quad \begin{array}{cc} \text{case 1)} & \left\{ T \leq c\right\} \\ \text{case 2)} & \left\{ T \geq c\right\} \end{array} \]

is a test of significance level $\alpha$ for testing $H_0$ against $H_1$ (this is obvious). Moreover, it is a UMP test with this significance level (not obvious).

Example 7

Let $X_1, \ldots , X_n$ be a sample from $\operatorname {Pois}\left(\theta \right)$. Construct a UMP test of significance level $\alpha$ for testing $H_0: \theta = \theta_0$ against $H_1: \theta > \theta_0$.

Solution

Likelihood function: \[ L(\theta , x) = \prod _{i=1}^n \frac{\theta ^{x_i}}{x_i!}e^{-\theta } = \frac{\theta ^{\sum _{i=1}^n x_i}}{\prod _{i=1}^n x_i!}e^{-n\theta }. \] For $\theta ' > \theta ''$ the likelihood ratio \[ \frac{L(\theta ', x)}{L(\theta '', x)} = \left(\frac{\theta '}{\theta ''}\right)^{\sum _{i=1}^n x_i} e^{-n(\theta ' - \theta '')} \] is nondecreasing in the statistic $T(\vec{X}) = \sum_{i=1}^n X_i$ (since $\theta '/\theta '' > 1$). Hence the family $\left\{ \operatorname {Pois}\left(\theta \right), \theta > 0\right\}$ has a nondecreasing likelihood ratio in $T$, and by theorem Theorem 1 (case 2) we can apply the theorem on the monotone likelihood ratio: the UMP test has the form $S = \left\{ T(\vec{X}) \geq c\right\}$, where $c$ is found from the condition \[ \mathbb {P}_{\theta _0}\left(T(\vec{X}) \geq c\right) = \alpha . \] It remains to note that under $H_0$ (i.e. $\theta = \theta_0$) a sum of independent Poisson variables is again Poisson: $T(\vec{X}) = \sum_{i=1}^n X_i \sim \operatorname {Pois}\left(n\theta_0\right)$, so $c$ is (the nearest integer providing the required significance level) the quantile of level $1-\alpha$ of the distribution $\operatorname {Pois}\left(n\theta_0\right)$.

If the likelihood ratio is nonincreasing in the statistic $T(\vec{X})$, then it is nondecreasing in the statistic $\left(-T(\vec{X})\right)$. That is, in this case theorem Theorem 1 can also be used.

Problem 9

$X_1, \dots , X_n$ is a sample from $\mathrm{Exp}(\theta )$. Construct a UMP test for testing $H_0: \theta = \theta_0$ against a) $H_1: \theta > \theta_0$ b) $H_1: \theta < \theta_0$.

Each part is worth 0.5 points.

Problem 10

$X_1, \dots , X_n$ is a sample from $N(\theta , 1)$. Construct a UMP test for testing

$H_0: \theta \geq \theta_0$ against $H_1: \theta < \theta_0$;
$H_0: \theta \leq \theta_0$ against $H_1: \theta > \theta_0$.

Each part is worth 0.5 points.

Error table		Null hypothesis (\(H_0\))
		True	False
Decision on the null hypothesis (\(H_0\))	Not rejected	Correct decision True Negative (TN) \(1-\alpha\)	Type II error False Negative (FN) \(\beta\)
	Rejected	Type I error False Positive (FP) \(\alpha\)	Correct decision True Positive (TP) \(1-\beta\)