Bayesian Estimation

WarningDraft

This lecture note is still being edited: the text may contain typos, inaccuracies, and locally underdeveloped passages. If you find something, please let the author know (contact information on the course page).

1 Frequentist and Bayesian approaches to parameter estimation

Up to this point, when estimating a parameter \(\theta\) we assumed that it is an unknown but constant quantity from some set \(\Theta\). This is the frequentist approach to parametric estimation in statistics. Under this approach we assumed no initial distinctions between possible values of the parameter from the set \(\Theta\). In practice, a researcher often has some information about the parameter before the experiment is even carried out.

Imagine we have a coin, and we want to estimate how fair it is, i.e. we want to estimate the probability of heads. To obtain an estimate of the probability, an experiment is conducted: the researcher tosses the coin several times and then analyzes the results. In the language of mathematical statistics, we can say that we have a sample \(X_1, \ldots , X_n\) from a Bernoulli distribution \(\operatorname {Ber}(\theta )\) with unknown parameter \(0 < \theta < 1\). In the frequentist approach we would treat \(\theta \in \Theta = (0,1)\), i.e. before the experiment we consider it equally possible that \(\theta\) is \(0.5\) or \(0.00001\) — there is no distinction between possible parameter values for us before the experiment. But it is clear that if we are holding an ordinary coin received as change in a store, minted by some government purely to serve as legal tender rather than as a tool in the hands of a swindler, then its probability of heads \(\theta\) is approximately \(0.5\) and cannot deviate far from this value. In this case we have some assumption, some knowledge about the parameter prior to the experiment. Such knowledge is called prior (from Latin a priori, literally “from what precedes”). Given this framing, the goal of the study is to update this knowledge, to revise it in light of the experiment’s results — in philosophical terms, to obtain posterior knowledge (from Latin a posteriori, literally “from what follows”) out of prior knowledge.

How can we mathematically formalize prior and posterior information about a parameter? We can think of \(\theta\) as a random variable defined on the same space as the sample \(X\), with some predetermined distribution \(\operatorname {Law}\left(\theta \right) = \mathbb {Q}\) on the parameter set \(\Theta\), corresponding to our prior knowledge and called the prior distribution. If the sample \(X\) takes values in \(\mathcal{X}\) (usually \(\mathcal{X}\) is \(\mathbb {Z}_+^n\) or \(\mathbb {R}^{n}\)), and \(\theta\) takes values in \(\Theta\), then \((X,\theta )\) can be viewed as a random element with values in the space

\[ (\mathcal{X} \times \Theta , \mathscr {B}\left(\mathcal{X}\right) \otimes \mathscr {B}\left(\Theta \right), \tilde{\mathbb {P}}), \]

with \(\mathbb {Q}(A) = \tilde{\mathbb {P}}(\mathcal{X}, A)\).

In the coin example, it would be natural to take as the prior distribution \(\mathbb {Q}\) some distribution on \([0,1]\) that is continuous and symmetric about \(0.5\) (with expectation \(0.5\), respectively). The most convenient distribution of this kind is the beta distribution \(B(\alpha , \alpha ), \; \alpha > 0\)¹. In this case \(\operatorname {Var}\left[\theta \right] = \frac{1}{4(2\alpha + 1)}\), and the more strongly we are initially confident that \(\theta\) deviates only slightly from \(\frac{1}{2}\), the larger a prior value of \(\alpha\) we can take.

If \(\theta\) is a random variable, then under this approach the sample \(X = (X_1, \ldots , X_n)\) from the distribution \(\mathbb {P}_\theta = \operatorname {Ber}(\theta )\) is a collection of random variables that are conditionally independent given \(\theta\), with conditional distribution given \(\theta\) equal to \(\mathbb {P}_\theta = \operatorname {Ber}(\theta )\):

\[ \operatorname {Law}\left(X \mid \theta \right) = \mathbb {P}_\theta \otimes \ldots \otimes \mathbb {P}_\theta = \left(\mathbb {P}_\theta \right)^{\otimes n} = \left(\operatorname {Ber}(\theta )\right)^{\otimes {n}} \]

As in the frequentist approach, for brevity it is convenient to denote the distribution \(\left(\mathbb {P}_\theta \right)^{\otimes n}\) simply as \(\mathbb {P}_\theta\).

The goal of the study is to update the knowledge about the distribution of the parameter \(\theta\) using the sample \(X\). In mathematical terms this means passing from the prior, unconditional distribution \(\operatorname {Law}\left(\theta \right) = \mathbb {Q}\) of the parameter \(\theta\) to a distribution that accounts for the sample, to the conditional distribution \(\operatorname {Law}\left(\theta \mid X\right)\). The distribution \(\operatorname {Law}\left(\theta \mid X\right)\) is called the posterior distribution.

2 How to obtain the posterior distribution

2.1 Bayes’ formula for densities

How do we find the posterior distribution \(\operatorname {Law}\left(\theta \mid X\right)\)? As a rule, the distribution of \(\theta\) on \(\Theta\) and the conditional distribution \(\mathbb {P}_\theta\) of the sample \(X\) on \(\mathcal{X}\) given \(\theta\) are absolutely continuous (dominated) with respect to some measures \(\mu_\Theta , \mu_{\mathcal{X}}\) on \(\Theta\) and \(\mathcal{X}\), and thanks to this they possess densities \(q(t)\), \(p_{X | \theta }(x|t)\). The two most common choices of dominating measure and the space on which they are defined are: Lebesgue measure \(\lambda\) on \(\mathbb {R}\) (\(\lambda^n\) on \(\mathbb {R}^{n}\)), and counting measure \(\#\) on \(\mathbb {Z}\) (\(\#^n\) on \(\mathbb {Z}^n\)). In this case the joint distribution of \((X,\theta )\) on the space \(\mathcal{X} \otimes \Theta\) is dominated by the product measure \(\mu_{\mathcal{X}} \otimes \mu_\Theta\), and the problem of finding \(\operatorname {Law}\left(\theta \mid X\right)\) reduces to the problem of finding the conditional density \(p_{\theta | X}(t | x)\). Recall that the conditional density equals the joint density divided by the density of the component being conditioned on. The latter is called the marginal density of the corresponding component (from the English “margin” — “edge”). We have

\[ p_{\theta | X}(t | x) = \frac{p_{(X, \theta )}(x, t)}{p_X(x)} \]

At the same time, by the same definition,

\[ p_{(X, \theta )}(x, t) = p_{X | \theta }(x|t) p_{\theta }(t) = p_{X | \theta }(x|t) q(t). \]

The marginal density can be computed by integrating the joint density over the remaining components:

\[ p_X(x) = \int _{\Theta } p_{(X, \theta )}(x, t) \mu _\Theta (dt). \]

Putting this together, we arrive at the formula

\[ p_{\theta | X}(t | x) = \frac{p_{X | \theta }(x|t) q(t)}{\int _{\Theta } p_{X | \theta }(x|t) q(t) \mu _\Theta (dt) }, \]

which is the analog of Bayes’ formula for densities. Note that, just as in the frequentist approach, the (conditional) density of the sample \(p_{X|\theta }(x|t)\) is called the likelihood function.

ExampleExample 1

Let \(X_1, \ldots , X_n\) be a sample from \(\operatorname {Ber}(\theta )\), where the prior distribution of \(\theta\) is \(\operatorname {Law}\left(\theta \right) = B(\alpha , \alpha ), \; \alpha > 0\). Find the posterior distribution of \(\theta\).

SolutionSolution

In this case \(\mathcal{X} = \left\{ 0,1\right\}^n, \mu_{\mathcal{X}} = \#^n, \; \Theta = [0,1], \mu_\Theta = \lambda\). Prior density:
\[ q(t) = \frac{1}{B(\alpha , \alpha )}t^{\alpha - 1}(1-t)^{\alpha - 1}. \] Likelihood function:
\[ p_{X|\theta }(x|t) = \prod _{i=1}^n p_{X_i|\theta }(x_i | t) = \prod _{i=1}^n t^{x_i}(1-t)^{1-x_i} = t^{n\overline{x}}(1-t)^{n-n\overline{x}} \] Let’s find the joint distribution of the sample and the parameter \((X,\theta )\) on the space \(\left\{ 0,1\right\}^n \times [0,1]\). Since it is dominated by the product measure \(\#^n \otimes \lambda\), let’s find their joint density with respect to this product:
\[ p_{X,\theta }(x, t) = p_{X|\theta }(x|t) \cdot q(t) = \frac{1}{B(\alpha , \alpha )} t^{\alpha + n\overline{x} - 1} (1-t)^{\alpha + n(1-\overline{x}) - 1} \] Let’s find the marginal (unconditional) density of the sample \(X\):
\[ p_X(x) = \int _{\Theta } p_{(X, \theta )}(x, t) \mu _\Theta (dt) = \int _0^1 \frac{1}{B(\alpha , \alpha )} t^{\alpha + n\overline{x} - 1} (1-t)^{\alpha + n(1-\overline{x}) - 1} dt = \frac{B(\alpha + n\overline{x}, \alpha + n(1-\overline{x}))}{B(\alpha , \alpha )} \] Finally, let’s find the posterior (conditional) density of \(\theta\) given \(X\):
\[ p_{\theta | X}(t | x) = \frac{p_{X,\theta }(x, t)}{p_X(x)} = \frac{1}{B(\alpha + n\overline{x}, \alpha + n(1-\overline{x}))}t^{\alpha + n\overline{x} - 1} (1-t)^{\alpha + n(1-\overline{x}) - 1} \] Thus, the conditional distribution \(\theta \mid X\) is also a beta distribution, just with different parameters:
\[ \operatorname {Law}\left(\theta \mid X\right) = B(\alpha + n\overline{X}, \alpha + n(1 - \overline{X})) \]

AnswerAnswer

\(B(\alpha + n\overline{X}, \alpha + n(1 - \overline{X}))\)

2.2 Using proportionality

If the support of the density and the dominating measure are known, then the shape of the density \(p(t)\) is determined only by those factors that involve \(t\). The remaining factors merely play the role of normalizing constants. For example, the gamma distribution \(\Gamma (\alpha , \lambda )\) is defined on \(\mathbb {R}_+\), is dominated by Lebesgue measure, and has density

\[ p_{\Gamma (\alpha , \lambda )}(t) = \frac{\lambda ^\alpha }{\Gamma (\alpha )}t^{\alpha - 1}e^{-\lambda t}. \]

Given the known support and dominating measure, this density is specified purely by \(t^{\alpha - 1}e^{-\lambda t}\). The factor \(\frac{\lambda^\alpha }{\Gamma (\alpha )}\) normalizes the density, uniquely determined so that \(\int_0^\infty p_{\Gamma (\alpha , \lambda )}(t) dt = 1\):

\[ \frac{\lambda ^\alpha }{\Gamma (\alpha )} = \frac{1}{\int _0^\infty t^{\alpha - 1}e^{-\lambda t} dt} \]

In example Example 1, for the posterior density \(p_{\theta | X}(t | x)\), the role of the normalizing constant was played by the factor \(\frac{1}{B(\alpha + n\overline{x}, \alpha + n(1-\overline{x}))}\). We knew that the posterior distribution \(\operatorname {Law}\left(\theta \mid X\right)\) is defined on the same set and dominated by the same measure as the prior distribution \(\operatorname {Law}\left(\theta \right)\): on \([0,1]\) with Lebesgue measure. To recognize that \(p_{\theta | X}(t | x)\) is the density of a beta distribution with parameters \(\alpha + n\overline{X}, \alpha + n(1 - \overline{X})\), in principle we only needed to compute \(t^{\alpha + n\overline{x} - 1} (1-t)^{\alpha + n(1-\overline{x}) - 1}\). For this reason, when finding the posterior distribution it is convenient to use the proportionality sign “\(\propto\)”:

\[ f_1(t) \; \propto \; f_2(t) \stackrel{\text{def}}{\iff } \; \exists c \in \mathbb {R}\setminus \left\{ 0\right\} : f_1(t) = c f_2(t) \]

The densities of the gamma distribution and the beta distribution can be rewritten as follows:

\[ p_{\Gamma (\alpha , \lambda )}(t) \; \propto \; t^{\alpha - 1}e^{-\lambda t}, \qquad p_{B(\alpha , \beta )}(t) \; \propto \; t^{\alpha - 1}(1-t)^{\beta - 1} \]

Bayes’ formula can be rewritten as:

\[ p_{\theta | X}(t | x) = \frac{p_{X | \theta }(x|t) q(t)}{\int _{\Theta } p_{X | \theta }(x|t) q(t) \mu _\Theta (dt) } \; \propto \; p_{X | \theta }(x|t) q(t), \]

since the denominator \(\int_{\Theta } p_{X | \theta }(x|t) q(t) \mu_\Theta (dt)\) does not depend on \(t\) and plays the role of a normalizing constant.

ExampleExample @expl-Bayes:CoinExample, short solution

Let \(X_1, \ldots , X_n\) be a sample from \(\operatorname {Ber}(\theta )\), where the prior distribution of \(\theta\) is \(\operatorname {Law}\left(\theta \right) = B(\alpha , \alpha ), \; \alpha > 0\). Find the posterior distribution of \(\theta\).

SolutionSolution

We have

\[ \begin{align} q(t) \; & \propto \; t^{\alpha - 1}(1-t)^{\alpha - 1} \\ p_{X|\theta }(x|t) \; & \propto \; t^{n\overline{x}}(1-t)^{n-n\overline{x}} \\ p_{\theta | X}(t | x) \; & \propto \; p_{X|\theta }(x|t) q(t) \; \propto \; t^{\alpha + n\overline{x} - 1}(1-t)^{\alpha + n-n\overline{x} - 1} \end{align} \]

The support of the distribution of \(\theta \mid X\) is the same as that of \(\theta\), namely \([0,1]\). The dominating measure is unchanged — Lebesgue measure. Consequently, \(\operatorname {Law}\left(\theta \mid X\right) = B(\alpha + n\overline{X}, \alpha + n(1 - \overline{X}))\).

AnswerAnswer

\(B(\alpha + n\overline{X}, \alpha + n(1 - \overline{X}))\)

Bayesian updating in the Bernoulli model: the prior distribution \(B(\alpha_0,\beta_0)\) (gray dashed line) turns, after \(n = 5, 20, 100\) observations, into the posteriors \(B(\alpha_0 + k, \beta_0 + n - k)\), where \(k\) is the accumulated number of successes along a single simulated path with true \(\theta\) (black vertical dashed line). Move the sliders to change the true \(\theta\) and the prior parameters, click “Regenerate data” for a new sample of tosses, or “Animate toss by toss” to watch the density flow smoothly from the prior to the posterior as observations arrive.

True \(\theta\) 0.70 Prior \(\alpha_0\) 3 Prior \(\beta_0\) 3

It is useful to see the geometry of a single step of Bayesian updating: the posterior density is obtained by “multiplying” the prior density by the likelihood (followed by normalization), rather than by simply replacing one curve with another.

The geometry of a single step of Bayesian updating for \(X_1,\ldots,X_n\sim\operatorname{Ber}(\theta)\) with a conjugate \(\operatorname{Beta}(\alpha,\alpha)\) prior (as in the example above): the posterior density is proportional to the product of the prior density and the likelihood, \(p(\theta\mid x)\propto q(\theta)\,L(\theta;x)\). Move the parameter \(\alpha\) of the prior and the number of observed successes/trials to see how the likelihood “pulls” the prior distribution toward the posterior.

α (prior) = 2.0 successes k = 7 trials n = 10

posterior distribution:

ProblemProblem 1

Let \(X_1, \dots , X_n\) be a sample from the normal distribution with parameters \((\theta ,1)\). Find the posterior distribution of the parameter \(\theta\), if the prior distribution of \(\theta\) is \(\mathscr {N}\left(b, \sigma^2\right)\).

ProblemProblem 2

Let \(X_1, \ldots , X_n\) be a sample from the distribution \(U[0,\theta ]\). Find the posterior distribution of the parameter \(\theta\), if \(\theta\) has prior distribution

1. \(U[0,1]\);

2. with density \(q(t) = \frac{1}{t^2} \; \mathbb {1}_{t \geq 1}\).

3 Estimators and their comparison in the Bayesian approach

3.1 Estimation in the Bayesian approach

Suppose we need to estimate the parameter \(\theta\). If the study has not yet taken place and we currently have only the prior (unconditional) distribution of \(\theta\), then the most logical estimate of \(\theta\) is obtained by taking the expectation \(\mathbb {E}\left[\theta \right] = \int_\Theta t \mathbb {Q}(dt)\) with respect to the prior distribution (assuming the expectation exists). Acting analogously after the experiment has been carried out and the posterior distribution obtained, it seems most reasonable to estimate \(\theta\) by the (conditional) expectation with respect to the posterior, conditional distribution \(\operatorname {Law}\left(\theta | X\right)\):

\[ \theta ^* = \theta ^*(X) := \mathbb {E}\left[\theta \mid X\right] = \int _\Theta t \; \cdot \operatorname {Law}\left(\theta | X\right)(dt) \]

Such an estimator is called a Bayesian estimator.

If the distributions of \(\theta\) and \(X\) are dominated and \(q(t)\), \(p_{X|\theta }(x|t)\) are the density of \(\theta\) and the conditional density (likelihood function) of \(X\), then the Bayesian estimator can be found in 2 steps:

1. find the function \(\varphi (x) := \mathbb {E}\left[\theta \mid X = x\right] = \int_\Theta t \cdot p_{\theta | X}(t|x) \mu_\Theta (dt)\);

2. substitute \(X\) into \(\varphi\): \(\theta^* = \mathbb {E}\left[\theta \mid X\right] = \varphi (X)\).

We discussed above how to find the posterior density \(p_{\theta | X}(t|x)\).

ExampleExample 3

Let \(X_1, \ldots , X_n\) be a sample from \(\operatorname {Ber}(\theta )\), where the prior distribution of \(\theta\) is \(\operatorname {Law}\left(\theta \right) = B(\alpha , \alpha ), \; \alpha > 0\). Find the Bayesian estimator of \(\theta\).

SolutionSolution

We have

\[ \begin{align} p_{\theta | X}(t | x) & = \frac{1}{B(\alpha + n\overline{x}, \alpha + n(1-\overline{x}))}t^{\alpha + n\overline{x} - 1} (1-t)^{\alpha + n(1-\overline{x}) - 1} \\ \varphi (x) & = \mathbb {E}\left[\theta \mid X = x\right] = \int _0^1 t \frac{1}{B(\alpha + n\overline{x}, \alpha + n(1-\overline{x}))}t^{\alpha + n\overline{x} - 1} (1-t)^{\alpha + n(1-\overline{x}) - 1} dt = \\ & = \frac{B(\alpha + n\overline{x} + 1, \alpha + n(1-\overline{x}))}{B(\alpha + n\overline{x}, \alpha + n(1-\overline{x}))} \end{align} \]

Let’s use the property of the beta function \(B(a+1,b) = \frac{a}{a+b}B(a,b)\):

\[ \varphi (x) = \frac{\alpha + n\overline{x}}{2 \alpha + n} \]

In total,

\[ \mathbb {E}\left[\theta \mid X\right] = \varphi (X) = \frac{\alpha + n\overline{X}}{2\alpha + n} \]

Note that the problem could have been solved more simply. We know that \(\theta \mid X \sim B(\alpha + n\overline{X}, \alpha + n(1 - \overline{X}))\). And we know that the expectation of a beta distribution \(B(\gamma_1, \gamma_2)\) equals \(\frac{\gamma_1}{\gamma_1 + \gamma_2}\). Hence

\[ \mathbb {E}\left[\theta \mid X\right] = \frac{\alpha + n\overline{X}}{2\alpha + n} \]

AnswerAnswer

\(\frac{\alpha + n\overline{X}}{2\alpha + n}\).

ProblemProblem 3

Let \(X_1, \dots , X_n\) be a sample from the normal distribution with parameters \((\theta ,1)\). Find the Bayesian estimator of the parameter \(\theta\), if the prior distribution of \(\theta\) is \(\mathscr {N}\left(b, \sigma^2\right)\).

ProblemProblem 4

Let \(X_1, \ldots , X_n\) be a sample from the distribution \(U[0,\theta ]\). Find the Bayesian estimator of the parameter \(\theta\), if \(\theta\) has prior distribution

1. \(U[0,1]\);

2. with density \(q(t) = \frac{1}{t^2} \; \mathbb {1}_{t \geq 1}\).

3.2 Consistency of estimators

As in the frequentist approach, in the Bayesian approach an estimator \(\widehat{\theta }\) is a consistent estimator of \(\theta\) if \(\widehat{\theta } \xrightarrow [n \to \infty ]{\mathbb {P}} \theta\). The difference is that here \(\theta\) is not a constant but a random variable. Convergence in probability to a random variable is not equivalent to convergence in distribution. Consistency cannot be reduced to convergence in distribution. One can only use the absence of convergence in distribution to prove the absence of consistency.

The conditional law of large numbers can help prove consistency.

Theorem 1 (Conditional law of large numbers) Let \(X_1, X_2, \dots\) be a sequence of random variables that are conditionally i.i.d. given a random variable \(\theta\). Let \(\mu (\theta ) := \mathbb {E}\left[X_i | \theta \right], \; \sigma^2(\theta ) := \operatorname {Var}\left[X_i | \theta \right]\), and suppose \(\sigma^2(\theta )\) is integrable, i.e. \(\mathbb {E}\left[\sigma^2(\theta )\right] < \infty\). Then
\[ \begin{align} \overline{X} \xrightarrow [n \to \infty ]{\mathbb {P}} \mu (\theta ) \end{align} \]

ExampleExample 4

Investigate the estimator from example Example 3 for consistency.

SolutionSolution

We have \(\mathbb {E}\left[X_1 \mid \theta \right] = \theta , \; \sigma^2(\theta ) = \operatorname {Var}\left[X_1 \mid \theta \right] = \theta (1-\theta )\). Moreover

\[ \mathbb {E}\left[\sigma ^2(\theta )\right] = \mathbb {E}\left[\theta (1-\theta )\right] = \frac{1}{B(\alpha ,\alpha )}\int _0^1 t(1-t) t^{\alpha - 1}(1-t)^{\alpha - 1}dt = \frac{B(\alpha + 1, \alpha + 1)}{B(\alpha ,\alpha )} < \infty \]

Consequently, by the conditional LLN, \(\overline{X} \xrightarrow [n \to \infty ]{\tilde{\mathbb {P}}} \theta\).

Further, \(\frac{\alpha }{n} \xrightarrow [n \to \infty ]{} 0\), so by the theorem on inheritance of convergences,

\[ \theta ^* = \mathbb {E}\left[\theta \mid X\right] = \frac{\frac{\alpha }{n} + \overline{X}}{2\frac{\alpha }{n} + 1} \xrightarrow [n \to \infty ]{\tilde{\mathbb {P}}} \theta \]

Trajectories of the Bayesian estimates \(\theta^*_\alpha(n) = \dfrac{\alpha + k_n}{2\alpha + n}\) under priors \(B(\alpha,\alpha)\) of varying “strength” (\(\alpha = 1, 10, 100\), thin lines), together with the trajectory \(\overline{X}_n = k_n/n\) (formally the case \(\alpha = 0\), gray line) along a single realization of a sample from \(\operatorname{Ber}(\theta)\). Move the \(\alpha\) slider to plot and highlight the trajectory \(\theta^*_\alpha(n)\) for an arbitrary (not just a “reference”) parameter value, the \(\theta\) slider to change the true parameter value, and click “Regenerate sample” for a new realization. Dashed horizontal lines mark the true \(\theta\) and the prior’s center \(0.5\).

True \(\theta\) 0.70 \(\alpha\) 25

AnswerAnswer

Consistent.

ProblemProblem 5

Let \(X_1, \dots , X_n \sim \mathscr {N}\left(\theta , 1\right), \; \theta \sim \mathscr {N}\left(b, \sigma^2\right)\). Check the Bayesian estimator \(\theta^*\) of the parameter \(\theta\) for consistency

ProblemProblem 6

Let \(X_1, \ldots , X_n\) be a sample from the distribution \(U[0,\theta ]\). Check the Bayesian estimator \(\theta^*\) of the parameter \(\theta\) for consistency, if \(\theta\) has prior distribution

1. \(U[0,1]\);

2. with density \(q(t) = \frac{1}{t^2} \; \mathbb {1}_{t \geq 1}\).

*Note.** Each part is worth 2 points.

ProblemProblem 7

Prove the conditional law of large numbers.

3.3 Comparing estimators

Let’s discuss comparing estimators in the Bayesian approach. We have already discussed comparing estimators and used for this the uniform approach with a quadratic loss function. Under this approach, one estimator is considered better than another if the variance of the first is no greater than the variance of the second for every value of the parameter. Of course, if we possess some prior knowledge about the value of the parameter, we can relax the “requirement of being best” by comparing only the averaged values of the variances.

Definition 1 Let \(\theta^{*} = \theta^{*}(X), \widehat{\theta } = \widehat{\theta }(X)\) be two estimators of a random parameter \(\theta\). We say that the estimator \(\theta^{*}\) is better than the estimator \(\widehat{\theta }\) in the Bayesian approach with loss function \(g(\cdot , \cdot )\), if

\[ \int _{\mathcal{X} \times \Theta } g\left(\theta ^*(x), t\right) \cdot \tilde{\mathbb {P}}(dx, dt) = \mathbb {E}\left[g(\theta ^*, \theta )\right] < \mathbb {E}\left[g(\widehat{\theta }, \theta )\right] = \int _{\mathcal{X} \times \Theta } g\left(\widehat{\theta }(x), t\right) \cdot \tilde{\mathbb {P}}(dx, dt) \tag{1}\]

Note that the expectation is taken in the space \((\mathcal{X} \times \Theta , \mathscr {B}\left(\mathcal{X}\right) \otimes \mathscr {B}\left(\Theta \right), \tilde{\mathbb {P}})\) — the space in which the sample and the parameter \((X,\theta )\) are defined. Given that \(\tilde{\mathbb {P}}(dx, dt)\) is the product of the prior distribution \(\mathbb {Q}\) and the distribution of the sample \(\mathbb {P}_\theta\):

\[ \tilde{\mathbb {P}}(dx,dt) = \mathbb {Q}(dt) \cdot \mathbb {P}_t(dx), \]

formula Equation 1 can be rewritten as

\[ \int _\Theta \mathbb {E}_{t}\left[g(\theta ^*, t)\right] \mathbb {Q}(dt) < \int _\Theta \mathbb {E}_{t}\left[g(\widehat{\theta }, t)\right] \mathbb {Q}(dt), \tag{2}\]

where \(\mathbb {E}_{t}\left[\cdot \right] = \mathbb {E}\left[\cdot \mid \theta = t\right]\) is the expectation taken with respect to the (conditional) distribution of the sample \(\operatorname {Law}\left(X\mid \theta = t\right) = \mathbb {P}_t\).

The following theorem is a direct consequence of properties of conditional expectation.

Theorem 2 The Bayesian estimator is the best estimator of \(\theta\) in the Bayesian approach with quadratic loss function.

Mn-proofProof

Let \(\theta^*(x) := \mathbb {E}\left[\theta \mid X = x\right]\) be the Bayesian estimator, and let \(\widehat{\theta }\) be an arbitrary other estimator. By the formula of total expectation (first averaging over \(X\) for fixed \(\theta\), which is exactly \(\mathbb {E}_{t}\left[\cdot \right]\) in Equation 2, and then over \(\mathbb Q(dt)\); or, equivalently, first averaging over \(\theta\) for fixed \(X\), then over the distribution of \(X\)):
\[ \mathbb {E}\left[\left(\widehat{\theta }(X) - \theta \right)^2\right] = \mathbb {E}\left[\mathbb {E}\left[\left(\widehat{\theta }(X) - \theta \right)^2 \mid X\right]\right]. \] Fix \(x\) and consider the function \(c \mapsto \mathbb {E}\left[(c-\theta )^2 \mid X = x\right]\) of the constant \(c\). Expanding the square by linearity of conditional expectation and recalling that \(\theta^*(x) = \mathbb {E}\left[\theta \mid X=x\right]\):
\[ \mathbb {E}\left[(c-\theta )^2 \mid X=x\right] = c^2 - 2c\, \mathbb {E}\left[\theta \mid X=x\right] + \mathbb {E}\left[\theta ^2 \mid X=x\right] = c^2 - 2c\, \theta ^*(x) + \mathbb {E}\left[\theta ^2 \mid X=x\right], \] and completing the square on the right-hand side with respect to \(c\):
\[ \mathbb {E}\left[(c-\theta )^2 \mid X=x\right] = \left(c - \theta ^*(x)\right)^2 + \mathbb {E}\left[\theta ^2\mid X=x\right] - \left(\theta ^*(x)\right)^2, \] which, as a function of \(c\), is minimized exactly at \(c = \theta^*(x)\). Hence, for any \(x\) we have \(\mathbb {E}\left[\left(\theta^*(X) - \theta \right)^2 \mid X=x\right] \leq \mathbb {E}\left[\left(\widehat{\theta }(X) - \theta \right)^2 \mid X=x\right]\), and after averaging over \(X\) we get \(\mathbb {E}\left[\left(\theta^*(X)-\theta \right)^2\right] \leq \mathbb {E}\left[\left(\widehat{\theta }(X)-\theta \right)^2\right]\), i.e. \(\theta^*\) is no worse than any other estimator \(\widehat{\theta }\) in the Bayesian approach with quadratic loss function.

ExampleExample 5

Find the Bayesian estimator of the parameter \(\theta\) from the sample \(X = (X_{1}, \ldots , X_{n})\) from the uniform distribution \(U[\theta , \theta +1]\), if the prior distribution of \(\theta\) is \(\mathbb {Q} = \operatorname {Pois}(1)\). Is the resulting estimator a consistent estimator of the parameter \(\theta\)?

SolutionSolution

We have \(q(t) = \frac{1}{t!}e^{-1}\). The likelihood function is the conditional density of \(X\) given \(\theta\):
\[ p_{X \mid \theta }(x \mid t) = \prod _{i=1}^n \; \mathbb {1}_{x_i \in [t, t + 1]} = \; \mathbb {1}_{t \leq x_{(1)}, \; x_{(n)} \leq t + 1} \] Let \(X=\left(X_{1}, \ldots , X_{n}\right)\). Since \(\mathbb {P}\left(0 \leq X_{(n)}-X_{(1)} < 1\right) = 1\), we have
\[ p_{\theta | X}(t | x) \; \propto \; p_{X \mid \theta }(x \mid t) q(t) = \; \mathbb {1}_{t \leq x_{(1)}, \; x_{(n)} \leq t + 1}\frac{1}{t!}. \] An integer \(t\) satisfies the condition \(t \leq x_{(1)} \leq x_{(n)} \leq t+1\) if and only if \(t \leq x_{(1)} < t+1\) (the left inequality is the same, and the right one is equivalent to \(x_{(n)} \leq t+1\) precisely because \(x_{(n)} - x_{(1)} < 1\)). Such an integer \(t\) is unique and equals \([x_{(1)}]\), so
\[ p_{\theta | X}(t | x) \; \propto \; \; \mathbb {1}_{[x_{(1)}] = t}. \] Now let’s find the Bayesian estimator:
\[ \theta ^{*}= \mathbb {E}\left[\theta \mid X\right] = \sum _{t=0}^{\infty } t \; \mathbb {1}_{[ X_{(1)}] =t} = [ X_{(1)}] \] Let’s check the resulting estimator for consistency. Of course, \(\mathbb {P}\left(\left[ X_{(1)}\right]=\theta \right)=1\). Therefore the Bayesian estimator is strongly consistent.

ProblemProblem 8

Let \(X_1, \ldots , X_n\) be a sample from the normal distribution with parameters \((\theta ,1)\). Find the Bayesian estimator of the parameter \(\theta\), if the prior distribution of \(\theta\) is \(\operatorname {Ber}(p)\). Will the resulting estimator be a consistent estimator of the parameter \(\theta\)?

4 Conjugate distributions

In statistical problems it is often assumed that the prior distribution \(\mathbb {Q} = \operatorname {Law}\left(\theta \right)\) and the posterior distribution \(\operatorname {Law}\left(\theta \mid X\right)\) belong to the same family of distributions. In this case the family of distributions to which \(\mathbb {Q}\) belongs is called the conjugate prior distribution for the family of sample distributions \(\mathcal{P} = \left\{ \mathbb {P}_\theta \right\}\). Of course, for an arbitrary family of sample distributions \(\mathcal{P}\), the family of all possible distributions on \(\Theta\) is conjugate. However, in some models \(\mathcal{P}\) the family of conjugate prior distributions can be significantly narrowed.

Consider example Example 1. In it

\[ \operatorname {Law}\left(\theta \right) = B(\alpha , \alpha ), \qquad X_1, \ldots , X_n \text{ -- a sample from } \operatorname {Ber}(\theta ) \quad \; \Rightarrow \; \quad \operatorname {Law}\left(\theta \mid X\right) = B\left(\alpha + n\overline{X}, \alpha + n(1-\overline{X})\right). \]

It is clear that in general

\[ \operatorname {Law}\left(\theta \right) = B(\alpha , \beta ), \qquad X_1, \ldots , X_n \text{ -- a sample from } \operatorname {Ber}(\theta ) \quad \; \Rightarrow \; \quad \operatorname {Law}\left(\theta \mid X\right) = B\left(\alpha + n\overline{X}, \beta + n(1-\overline{X})\right). \]

The beta distribution \(B(\alpha , \beta )\) is the conjugate prior distribution for \(\operatorname {Ber}(\theta )\).

ExampleExample 6

Find the conjugate prior distribution for the sample distribution \(\operatorname {Exp}(\theta )\).

SolutionSolution

Note in advance that the parameter \(\theta\) must be nonnegative, i.e. the support of the conjugate prior distribution must not exceed \(\mathbb {R}_+ = [0 , +\infty )\).

We have \(X_i \mid \theta = t \; \sim \operatorname {Exp}(t), \; i \in \left\{ 1,\ldots , n\right\}\). The likelihood function on \(\mathbb {R}^{n}_+\):

\[ p_{X|\theta }(x|t) = \prod _{i=1}^n te^{-tx_i} \; \mathbb {1}_{x_i \geq 0} = t^n e^{-tn\overline{x}} \]

We require that \(\operatorname {Law}\left(\theta \right)\) and \(\operatorname {Law}\left(\theta \mid X\right)\) belong to the same class, i.e. that the prior \(q(t)\) and posterior \(p_{\theta | X}(t | x)\) densities have an analogous form. The posterior density:

\[ p_{\theta | X}(t | x) \; \propto \; p_{X|\theta }(x|t) q(t) = t^n e^{-tn\overline{x}}q(t) \]

The prior and posterior densities are connected by multiplication by the power and exponential functions \(t^n e^{-tn\overline{x}}\), up to proportionality. Note that if \(q(t)\) is proportional to the same functions, a product of a power and an exponential function, then \(p_{\theta | X}(t | x)\) will be of the same form:

\[ q(t) \; \propto \; t^{\alpha - 1}e^{-\lambda t} \quad \; \Rightarrow \; \quad p_{\theta | X}(t | x) \; \propto \; t^{n + \alpha - 1}e^{-(\lambda + n\overline{x}) t} \]

A distribution whose density is proportional to \(t^{\alpha - 1}e^{-\lambda t}\), with support \(\mathbb {R}_+\), is, for example, the gamma distribution \(\Gamma (\alpha , \lambda )\).

AnswerAnswer

Conjugate prior distribution: \(\operatorname {Law}\left(\theta \right) = \Gamma (\alpha , \lambda )\). Posterior distribution: \(\operatorname {Law}\left(\theta \mid X\right) = \Gamma (n + \alpha , \lambda + n\overline{X})\).

ExampleExample 7

Find the Bayesian estimator of the parameter \(\theta\) from the sample \(X_{1}, \ldots , X_{n}\) from the exponential distribution with parameter \(\theta\), having a conjugate prior distribution.

SolutionSolution

Posterior distribution of the parameter \(\theta\): \(\operatorname {Law}\left(\theta \mid X\right) = \Gamma (n + \alpha , \lambda + n\overline{X})\). It is known that if \(\xi \sim \Gamma (\beta_1, \beta_2)\), then \(\mathbb {E}\left[\xi \right] = \frac{\beta_1}{\beta_2}\). Then the Bayesian estimator of the parameter \(\theta\) equals
\[ \theta ^* = \mathbb {E}\left[\theta \mid X\right] = \frac{n + \alpha }{\lambda + n\overline{X}} \]

ProblemProblem 9

From the sample \(X_1, \ldots , X_n\) from a Poisson distribution with parameter \(\theta\), find

1. the conjugate prior distribution for the parameter \(\theta\);

2. the best estimator in the Bayesian approach with quadratic loss function, taking as the prior the distribution found in the previous part.

*Note.** Each part is worth 0.5 points.

ProblemProblem 10

Let \(X_1, \dots , X_n\) be a sample from the distribution

1. \(\mathscr {N}\left(\theta , 1\right)\);

2. \(\mathscr {N}\left(0, \theta \right)\);

3. \(\operatorname {Bin}(m,\theta )\), \(m \in \mathbb {N}\) — known. Choose a conjugate distribution for the parameter \(\theta\) and find the Bayesian estimator.

*Note.** Parts a), c) are each worth 0.5 points. Part b) is worth 1 point.

ProblemProblem 11

Check the estimator from example Example 7 for consistency.

*Note.** This problem is worth 3 points.

Footnotes

1. Recall that if \(\xi \sim B(\alpha , \beta ), \; \alpha , \beta > 0\), then the density equals
   \[ \begin{align} p_\xi (x) & = \frac{1}{B(\alpha , \beta )} x^{\alpha - 1}(1-x)^{\beta - 1} \; \mathbb {1}_{x \in [0,1]}, \qquad B(\alpha , \beta ) = \int _0^1 t^{\alpha - 1}(1-t)^{\beta - 1}dt = \frac{\Gamma (\alpha )\Gamma (\beta )}{\Gamma (\alpha + \beta )}, \\ \mathbb {E}\left[\xi \right] & = \frac{\alpha }{\alpha + \beta }, \qquad \operatorname {Var}\left[\xi \right] =\frac{\alpha \beta }{(\alpha + \beta )^2 (\alpha + \beta + 1)}. \end{align} \] If \(\beta = \alpha\), then \(\mathbb {E}\left[\xi \right] = \frac{1}{2}, \; \operatorname {Var}\left[\xi \right] = \frac{1}{4(2\alpha + 1)}\). Note that \(B(1, 1) = U[0,1]\).