This text is still being edited: it may contain typos, inaccuracies, and locally unpolished spots. If you find something, please let the author know (contact details on the course page).
The conditional expectation from topic 11 is the conditional mean; but often a full answer is needed: what is the conditional distribution of \(\xi\) given \(\eta\)? All two-stage models are built this way. A classic example (Klenke [@klenke2020probability]): a coin is βcorruptedβ at random, so that the probability of heads \(X\) is uniform on \([0,1]\), and then, given \(X = x\), it is tossed \(n\) times independently. Intuition immediately says: \(\mathbb {P}\left(Y_1 = \ldots = Y_n = 1 \mid X = x\right)\) βshould beβ \(x^n\) β but the event \(\left\{ X = x\right\}\) has probability zero, and the first-course definition cannot make sense of this expression. Topic 11 provided a workaround β \(\mathbb {P}\left(A \mid \eta \right) := \mathbb {E}\left[ \; \mathbb {1}_{A} \mid \eta \right]\) β but a βpiecemealβ one, with its own null set for each event \(A\). In this topic we will build the full-fledged object β the regular conditional distribution: for each value of the condition, an actual probability measure. Along the way we will get the language of transition kernels (the stochastic processes course will use it to define Markov processes), the computational technique of conditional densities, the famous Borel-Kolmogorov paradox, and β as a bridge to statistics β sufficient statistics. We rely on Shiryaev [@shiryaev2016probability] (Ch. 2, Β§7) and Klenke [@klenke2020probability] (Β§8.3).
Transition kernels
Definition 1 Let \((E_1, \mathcal{E}_1)\) and \((E_2, \mathcal{E}_2)\) be measurable spaces. A function \(K\colon E_1 \times \mathcal{E}_2 \to [0, 1]\) is called a transition (Markov, stochastic) kernel from \(E_1\) to \(E_2\) if:
for each fixed \(y \in E_1\), the map \(B \mapsto K(y, B)\) is a probability measure on \((E_2, \mathcal{E}_2)\);
for each fixed \(B \in \mathcal{E}_2\), the map \(y \mapsto K(y, B)\) is measurable.
A kernel is a βmeasurable family of measuresβ indexed by points of the first space: a machine that, given an input \(y\), outputs a distribution. Examples (Klenke, Example 8.27):
discrete spaces: a kernel is a matrix \(K_{ij} \geq 0\) with unit row sums β the transition probability matrix of a Markov chain (the stochastic processes course);
\(K(x, \cdot ) = \operatorname {Pois}(x)\) β a kernel from \([0, \infty )\) to \(\mathbb {Z}^{+}\): βrandom intensity \(\to\) random countβ (the insurance model below);
\(K(x, \cdot ) = \mathbb {P}_{x + \zeta }\) β the convolution kernel of a random shift: βcurrent point \(\to\) point after a random stepβ.
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/120/03_kernel_fig.tex.
Regular conditional distributions
Why is the βpiecemealβ definition of \(\mathbb {P}\left(\xi \in B \mid \eta \right)\) from topic 11 not enough? Each version is defined up to a null set, its own for every Borel set \(B\); there are uncountably many Borel sets, and the union of all the exceptional sets can cover all of \(\Omega\) (Shiryaev, Β§7.5). What we want instead is that, for (almost) every \(y\), the collection of numbers \(\mathbb {P}\left(\xi \in B \mid \eta = y\right)\), \(B \in \mathscr {B}\left(\mathbb {R}\right)\), be a consistent probability measure β then we can work with it like an ordinary distribution: integrate, compute moments, substitute. This is exactly regularity.
Definition 2 Let \(\xi\) be a random element taking values in \((E, \mathcal{E})\), and let \(\eta\) be a random variable. A regular conditional distribution of \(\xi\) with respect to \(\eta\) is a transition kernel \(K\) from \(\mathbb {R}\) to \(E\) such that \[
\mathbb {P}\left(\xi \in B, \; \eta \in C\right) = \int _C K(y, B) \, \mathbb {P}_\eta (dy) \qquad \text{for all } B \in \mathcal{E}, \; C \in \mathscr {B}\left(\mathbb {R}\right);
\] we write \(K(y, B) = \mathbb {P}\left(\xi \in B \mid \eta = y\right)\). (Equivalently: for each \(B\), the function \(K(\eta (\omega ), B)\) is a version of \(\mathbb {P}\left(\xi \in B \mid \eta \right)(\omega )\).) A regular conditional distribution with respect to a \(\sigma\)-algebra \(\mathcal{G}\) is defined analogously: a kernel \(\mathbb {P}\left(\omega ; B\right)\) from \((\Omega , \mathcal{G})\) to \((E, \mathcal{E})\).
Theorem 1 (Existence; Shiryaev, Theorem 4; Klenke, Theorems 8.29 and 8.37) If \(\xi\) takes values in \(\mathbb {R}\) β or, more generally, in a Borel space (measurably isomorphic to a Borel subset of the line; such are all Polish spaces: \(\mathbb {R}^{d}\), \(\mathbb {R}^{\infty }\), \(C[0,1]\), etc.) β then a regular conditional distribution exists and is unique up to a \(\mathbb {P}_\eta\)-null set of values \(y\).
The idea of the proof for \(\mathbb {R}\) is instructive β it is an βassembly of a measure from countably many partsβ. For each rational \(r\), take some version \(F_r(\omega ) = \mathbb {P}\left(\xi \leq r \mid \mathcal{G}\right)(\omega )\). Monotonicity and the limiting properties of conditional expectation (topic 11) are violated each time only on a null set; there are countably many rational numbers, so all the defects β non-monotonicity, absence of a right limit, incorrect limits at \(\pm \infty\) β fit inside a single null set \(N\). Outside \(N\), the function \(F(\omega ; x) := \inf_{r > x, \, r \in \mathbb {Q}} F_r(\omega )\) is an honest distribution function; on \(N\), substitute any fixed CDF. Every CDF defines a measure (the correspondence theorem, topic 1!), and agreement with the βpiecemealβ conditional probabilities extends from rays to all Borel sets by Dynkinβs lemma. The general case reduces to \(\mathbb {R}\) by transporting along a measurable isomorphism.
The main computational reward of regularity is that conditional expectations are computed by integration against the kernel, at once for all functions:
Theorem 2 (Substitution; Shiryaev, Theorem 3; Klenke, Theorem 8.38) Let \(K\) be a regular conditional distribution of \(\xi\) with respect to \(\eta\), and let \(\mathbb {E}\left[\left|g(\xi )\right|\right] < \infty\). Then \[
\mathbb {E}\left[ g(\xi ) \mid \eta = y \right] = \int _E g(x)\, K(y, dx) \qquad \text{for } \mathbb {P}_\eta \text{-a.e.\ } y.
\]
The standard machine: for \(g = \; \mathbb {1}_{B}\) the statement is the definition Definition 2; then linearity (simple \(g\)), conditional monotone convergence (nonnegative \(g\); here we use that \(K(y, \cdot )\) is a measure, so the limit can be brought inside the integral against it), and the difference \(g = g^+ - g^-\).
Altogether, this is the recipe for computing any conditional characteristic: find the conditional distribution β then all averages, variances, and probabilities are computed from it by ordinary integration, exactly as in the unconditional world.
Conditional densities
Theorem 3 (Conditional density; Shiryaev, Example 2; Klenke, Example 8.31) Let the vector \((\xi , \eta )\) have joint density \(p(x, y)\), and let \(p_\eta (y) = \int p(x,y)\, dx\). Then the regular conditional distribution of \(\xi\) with respect to \(\eta\) has density \[
p_{\xi \mid \eta }(x \mid y) = \frac{p(x, y)}{p_\eta (y)} \qquad (\text{for } p_\eta (y) > 0; \text{ on the null set --- any fixed density}),
\] and, correspondingly, \(\mathbb {E}\left[ g(\xi ) \mid \eta = y \right] = \int g(x)\, p_{\xi \mid \eta }(x \mid y)\, dx\); in particular, the regression function \[
m(y) = \mathbb {E}\left[ \xi \mid \eta = y \right] = \int _{\mathbb {R}} x \, p_{\xi \mid \eta }(x \mid y)\, dx.
\]
We check definition Definition 2 for \(K(y, B) = \int_B p_{\xi |\eta }(x\mid y)\, dx\) by Fubiniβs theorem: \[
\int _C \left[\int _B p_{\xi \mid \eta }(x \mid y)\, dx\right]\, p_\eta (y)\, dy = \iint _{B \times C} p(x, y)\, dx\, dy = \mathbb {P}\left(\xi \in B, \eta \in C\right).
\] Measurability in \(y\) is also Fubini; normalization \(\int p_{\xi |\eta }\, dx = 1\) for \(p_\eta (y) > 0\) is obvious.
Geometry of the formula: the conditional density is a normalized slice of the joint density along the line \(\eta = y\). Along the way we obtain continuous versions of the law of total probability and Bayesβ formula:
\[
p_\xi (x) = \int _{\mathbb {R}} p_{\xi \mid \eta }(x \mid y) \, p_\eta (y) \, dy, \qquad p_{\eta \mid \xi }(y \mid x) = \frac{p_{\xi \mid \eta }(x \mid y)\, p_\eta (y)}{\int _{\mathbb {R}} p_{\xi \mid \eta }(x \mid u)\, p_\eta (u)\, du}.
\]
And two useful extreme cases: if \(\xi \; \perp \! \! \! \perp \; \eta\), then the conditional distribution coincides with the unconditional one (\(K(y, \cdot ) = \mathbb {P}_\xi\)); and the conditional distribution of a sum given one term is a shift: \(\mathbb {P}\left(\xi + \eta \in \cdot \mid \xi = x\right) = \mathbb {P}_{x + \eta }\) (the convolution kernel; Klenke, Example 8.33).
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/120/01_conditional_slice_fig.tex.
\(X \sim \operatorname {U}[0,1]\); given \(X = x\), the variables \(Y_1, \ldots , Y_n\) are independent and \(\operatorname {Bern}(x)\)-distributed. Write out the model formally and find \(\mathbb {P}\left(Y_1 = \ldots = Y_n = 1\right)\).
Formally, the model is exactly the specification of a transition kernel \(K(x, \cdot ) = \left(\operatorname {Bern}(x)\right)^{\otimes n}\) from \([0,1]\) to \(\left\{ 0,1\right\}^n\), together with the requirement that it be a regular conditional distribution of the vector \(Y\) with respect to \(X\): the intuitive expression β\(\mathbb {P}\left(Y_1 = \ldots = Y_n = 1 \mid X = x\right) = x^n\)β becomes a theorem about the kernel. Then by definition Definition 2 (taking \(C = [0,1]\)): \[
\mathbb {P}\left(Y_1 = \ldots = Y_n = 1\right) = \int _0^1 x^n \, dx = \frac{1}{n+1}.
\] Notice: the two-stage description (βfirst a random parameter, then an experiment given the parameterβ) is a construction of a joint distribution from a marginal and a kernel. This is how all hierarchical models are built: Bayesian statistics, hidden Markov chains, generative models.
The Borel-Kolmogorov paradox
An essential warning: the conditional distribution given an event of probability zero depends on which quantityβs value we regard that event as.
Let \(X, Y \sim \mathscr {N}\left(0, 1\right)\) be independent. Find the conditional distribution of \(X\) given (a) \(Y - X = 0\); (b) \(Y/X = 1\). Both conditions define the same event \(\left\{ X = Y\right\}\) (up to the null event \(X = 0\)). Do the answers coincide?
- The pair \((X, \, Y - X)\) is Gaussian; by the theorem on the conditional distribution of a Gaussian vector (topic 10), the conditional distribution of \(X\) given \(Y - X = 0\) is \(\mathscr {N}\left(0, \frac12\right)\): density \(\propto e^{-x^2}\). (b) Pass to the pair \((X, \, W)\), \(W = Y/X\). The joint density via the Jacobian (\(y = wx\), \(\left|\partial y / \partial w\right| = \left|x\right|\)):
\[
p_{X, W}(x, w) = p_{X,Y}(x, wx)\left|x\right| = \frac{1}{2\pi }e^{-\frac{x^2(1 + w^2)}{2}}\left|x\right|,
\]
and the conditional density of \(X\) given \(W = 1\) is proportional to \(\left|x\right| e^{-x^2}\) β a different distribution (with zero density at zero!).
There is no contradiction: βthe condition on the event \(\left\{ X = Y\right\}\)β is not itself well-defined β what is well-defined is a condition on the value of a random variable, and the kernels for \(\eta_1 = Y - X\) and \(\eta_2 = Y/X\) are different objects, each consistent with the joint distribution in its own way. Intuitively: \(\left\{ X = Y\right\}\) as the limit of strips \(\left\{ \left|Y - X\right| < \varepsilon \right\}\) and as the limit of wedges \(\left\{ \left|Y/X - 1\right| < \varepsilon \right\}\) are different limits: the wedge near zero shrinks faster, pushing mass away from zero (the factor \(\left|x\right|\)). Practical conclusion: the phrasing βgiven \(A\)β with \(\mathbb {P}\left(A\right) = 0\) must specify the quantity through which the condition is taken.
The classical version (Klenke, Exercise 8.3.5): a point is uniform on a sphere; at a fixed longitude, the latitude \(\Phi\) is distributed not uniformly along the meridian, but with density \(\propto \left|\cos \varphi \right|\) (the meridian is βthickerβ near the equator β squeeze a narrow spherical wedge and look at its width), even though at a fixed latitude the longitude is uniform. βUniformity on a great circleβ depends on which coordinate carves out that circle.
The law of total variance
Proposition 1 (The law of total variance) For \(\xi \in L^2\): \[
\operatorname {Var}\left[\xi \right] = \mathbb {E}\left[\operatorname {Var}\left[ \xi \mid \eta \right]\right] + \operatorname {Var}\left[\mathbb {E}\left[ \xi \mid \eta \right]\right],
\] where \(\operatorname {Var}\left[ \xi \mid \eta \right] := \mathbb {E}\left[ \left(\xi - \mathbb {E}\left[\xi \right][\eta ]\right)^2 \mid \eta \right]\) is the conditional variance.
Read as a decomposition of uncertainty: average within-group plus between-group variance (in statistics β analysis of variance; in ML β error decomposition). The proof is the tower property plus the Pythagorean decomposition from the best-forecast theorem (topic 11); work it out.
The Gaussian case: a reminder
For jointly Gaussian vectors, conditional distributions can be written out explicitly β this is the theorem on the conditional distribution of a Gaussian vector from topic 10: the conditional distribution is again Gaussian,
\[
\mathbb {E}\left[ X \mid Y \right] = \mu _X + \Sigma _{XY}\Sigma _{YY}^{-1}(Y - \mu _Y), \qquad \operatorname {Var}\left[ X \mid Y \right] = \Sigma _{XX} - \Sigma _{XY}\Sigma _{YY}^{-1}\Sigma _{YX},
\]
the forecast is linear, the conditional covariance is constant. In the terms of this topic: the Gaussian regular conditional distribution is the kernel \(K(y, \cdot ) = \mathscr {N}\left(\mu_X + \Sigma_{XY}\Sigma_{YY}^{-1}(y - \mu_Y), \; \Sigma_{XX} - \Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}\right)\), and all the dependence on the condition sits in the shift of the mean. The one-dimensional version with correlation \(\rho\):
\[
\mathbb {E}\left[ X \mid Y \right] = \mu _X + \rho \frac{\sigma _X}{\sigma _Y}(Y - \mu _Y), \qquad \operatorname {Var}\left[ X \mid Y \right] = \sigma _X^2\left(1 - \rho ^2\right)
\]
β the linear regression equation, with the famous effect of regression to the mean (Galton): for \(\left|\rho \right| < 1\), the forecast is always closer to the mean (in units of standard deviation) than the observation. The son of a very tall father is on average tall, but shorter than the father; a brilliant quarter for a company is on average followed by a more ordinary one β not βMurphyβs lawβ, but \(\rho < 1\). Failure to understand this effect is a source of countless false conclusions about the βeffectivenessβ of punishments and rewards.
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/120/02_regression_line_fig.tex.
Sufficient statistics (optional)
Regular conditional distributions are the language in which one of the central notions of statistics is formulated. Suppose the distribution of the data depends on an unknown parameter: \(\mathbb {P} \in \left\{ \mathbb {P}_\theta , \theta \in \Theta \right\}\).
Definition 3 A statistic \(T\) (a function of the data) is called sufficient for the family \(\left\{ \mathbb {P}_\theta \right\}\) if the conditional distribution of the data given \(T\) does not depend on \(\theta\): there is a single kernel \(K(t, \cdot )\) serving as the regular conditional distribution for every \(\theta\).
Meaning: once \(T\) is known, nothing more about the parameter can be squeezed out of the remaining details of the data β all the information about \(\theta\) is compressed into \(T\) without loss. This gives an enormous reduction: instead of \(n\) numbers, one or two.
\(X_1, \ldots , X_n\) are independent \(\operatorname {Bern}(\theta )\), \(T = X_1 + \ldots + X_n\). Show that \(T\) is sufficient.
For a tuple \(x = (x_1, \ldots , x_n)\) with sum \(t\): \[
\mathbb {P}\left(X = x \mid T = t\right) = \frac{\mathbb {P}\left(X = x\right)}{\mathbb {P}\left(T = t\right)} = \frac{\theta ^{t}(1-\theta )^{n - t}}{C_n^{t}\, \theta ^{t}(1-\theta )^{n-t}} = \frac{1}{C_n^{t}}
\] β the parameter cancels: given the number of successes, their arrangement is equally likely among all \(C_n^t\) possibilities, whatever \(\theta\) is. Hence \(T\) is sufficient: the order of zeros and ones carries no information about the success probability. The general criterion sparing us the direct check is the factorization theorem (Shiryaev, Theorem 7): for a dominated family with densities \(g_\theta\), the statistic \(T\) is sufficient if and only if \(g_\theta (x) = \widehat{g}_\theta (T(x))\, h(x)\) β the dependence on \(\theta\) passes only through \(T\).
Theorem 4 (Rao-Blackwell; Shiryaev, Theorem 8; statement) Let \(T\) be sufficient and let \(\widehat{\theta }\) be any estimator of the parameter with finite second moment. Then \(\widetilde{\theta } := \mathbb {E}\left[ \widehat{\theta } \mid T \right]\) is a valid estimator (it does not depend on \(\theta\), thanks to sufficiency!), and \[
\mathbb {E}\left[\left(\widetilde{\theta } - \theta \right)^2\right] \leq \mathbb {E}\left[\left(\widehat{\theta } - \theta \right)^2\right] \qquad \text{for every } \theta .
\]
This is a direct consequence of the best-forecast theorem: projecting onto \(\sigma (T)\) can only decrease the mean-square error. Practical reading: any estimator can be improved for free by averaging it over a sufficient statistic β hence good estimators are always functions of sufficient statistics. Course 3 makes this idea one of its cornerstones.
Problems
A point \((X, Y)\) is uniformly distributed on the triangle \(\left\{ x, y \geq 0, \; x + y \leq 1\right\}\). Find the conditional density \(p_{Y \mid X}(y \mid x)\), the regression function \(\mathbb {E}\left[ Y \mid X = x \right]\), and the variable \(\mathbb {E}\left[ Y \mid X \right]\).
(Klenke, Example 8.30.) \(Z_1 \sim \operatorname {Pois}(\lambda_1)\) and \(Z_2 \sim \operatorname {Pois}(\lambda_2)\) are independent. Find the conditional distribution of \(Z_1\) given \(Z_1 + Z_2 = n\).
(Klenke, Exercise 8.3.3.) A point \((X, Y)\) is uniformly distributed (a) on the disk \(x^2 + y^2 \leq 1\); (b) on the square \([-1,1]^2\). Let \(R = \sqrt{X^2+Y^2}\), \(\Theta\) the polar angle. In both cases find the conditional distribution of \(\Theta\) given \(R = r\) and compare them.
(Bayesian pricing.) A clientβs accident intensity is \(\Lambda \sim \Gamma (\alpha , \beta )\), and given \(\Lambda = \lambda\), the number of accidents in a year is \(N \sim \operatorname {Pois}(\lambda )\) (the model is specified by the kernel \(K(\lambda , \cdot ) = \operatorname {Pois}(\lambda )\)). Find the conditional distribution of \(\Lambda\) given \(N = n\) and the forecast \(\mathbb {E}\left[ \Lambda \mid N = n \right]\). Show that the forecast is a weighted average of the prior mean and the observation.
\(X_1, X_2, X_3 \sim \mathscr {N}\left(1, 1\right)\) are independent; \(U = 2X_1 - X_2 + X_3\), \(V = X_1 + 2X_2 + 3X_3\). Find the conditional distribution of \(V\) given \(U = 3\).
(The law of total variance in action.) Under the conditions of Waldβs formula (topic 11: \(S = \xi_1 + \ldots + \xi_N\), claims i.i.d. with \(\mathbb {E}\left[\xi_1\right] = m\), \(\operatorname {Var}\left[\xi_1\right] = \sigma^2\), number of claims \(N\) independent of the claims), find \(\operatorname {Var}\left[S\right]\).
Rejection sampling
(Klenke, Exercise 8.3.6.) Let \(P\) and \(Q\) be distributions on a countable set \(E\), and \(f(e) := \frac{Q(\left\{ e\right\} )}{P(\left\{ e\right\} )} \leq c\) for all \(e\) with \(P(\left\{ e\right\} ) > 0\). A generator can produce independent \(X_1, X_2, \ldots \sim P\) and independent \(U_1, U_2, \ldots \sim \operatorname {U}[0,1]\). Set \[
N = \min \left\{ n \colon U_n \leq f(X_n)/c\right\} , \qquad Y := X_N.
\] Prove that \(Y \sim Q\), and find \(\mathbb {E}\left[N\right]\).