This text is still being edited: it may contain typos, inaccuracies, and locally unpolished spots. If you find something, please let the author know (contact details on the course page).
You already know the one-dimensional normal distribution well: it arises as the limit in the CLT, and describes measurement errors and the spread of traits in nature and society. But in practice one rarely encounters exactly one random variable β usually there are several, and they are dependent (a personβs height and weight, asset returns, the coordinates of a random point). The question arises: what should count as a βnormal distributionβ for a random vector?
Our task is to transfer the notion of a Gaussian distribution from the line to \(\mathbb {R}^{n}\), preserving all the important properties: stability under affine transformations, a simple description via parameters (a mean vector and a covariance matrix), and β especially valuable and far from obvious β the equivalence of uncorrelatedness and independence of the components. The multivariate Gaussian distribution is one of the central objects of probability and statistics: it arises in the multivariate CLT, underlies linear regression and the least-squares method, describes Gaussian processes (including Brownian motion, the stochastic processes course), and is actively used in machine learning (Gaussian processes, principal component analysis).
Plan for this topic: geometric language (affine subspaces) \(\to\) Gaussian distributions in dimensions \(0\), \(1\), \(n\) \(\to\) four equivalent definitions \(\to\) examples and counterexamples \(\to\) uncorrelatedness and independence \(\to\) conditional expectation in the Gaussian world \(\to\) the multivariate CLT.
Introduction: affine subspaces
Let us recall a construction from linear algebra that will serve as the main geometric language for describing Gaussian vectors.
Definition 1 Let \(L \subseteq \mathbb {R}^{n}\) be a linear subspace and \(v_0 \in \mathbb {R}^{n}\). The set \[
H = v_0 + L := \left\{ v_0 + l \colon l \in L\right\}
\] is called the affine subspace generated by the subspace \(L\) and the shift \(v_0\); its dimension is \(\dim H := \dim L\).
In other words, an affine subspace is a linear subspace shifted by a vector: it may not pass through the origin, but it βpreserves the shapeβ of the original \(L\). (From now on, βsubspaceβ without qualification always means linear, i.e. passing through zero; shifted ones are always called affine.) Affine subspaces arise naturally as solution sets of linear systems:
Proposition 1 Let \(A \in \mathbb {R}^{m \times n}\), \(b \in \mathbb {R}^{m}\). The solution set of the system \(Ax = b\) is either empty or is an affine subspace of \(\mathbb {R}^{n}\) of dimension \(n - \operatorname {rk} A\).
Let \(x_0\) be a particular solution. If \(x\) is any solution, then \(A(x - x_0) = 0\), i.e. \(x - x_0 \in \ker A\); conversely, \(x_0 + l\) is a solution for any \(l \in \ker A\). Hence the solution set equals \(x_0 + \ker A\), and \(\dim \ker A = n - \operatorname {rk} A\) by the rank-nullity theorem.
In low dimensions, affine subspaces look as follows:
in \(\mathbb {R}\): points \(\left\{ c\right\}\) (dimension \(0\)) and the whole line (dimension \(1\)) β nothing in between;
in \(\mathbb {R}^{2}\): points, arbitrary lines (not necessarily through zero), and the whole plane;
in \(\mathbb {R}^{3}\): points, lines, planes, and the whole space.
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/100/02_affine_subspaces_fig.tex.
The Gaussian distribution in dimensions 0 and 1
Dimension 0
Let us begin with the degenerate case. By definition \(\mathbb {R}^{0} := \left\{ 0\right\}\) β a set consisting of a single zero vector. There is exactly one probability measure on it: \(\mathbb {P}\left(\left\{ 0\right\} \right) = 1\).
Definition 2 The standard \(0\)-dimensional Gaussian distribution is the unique distribution on \(\mathbb {R}^{0}\): the measure assigning probability \(1\) to the point \(0\).
The definition looks like a joke, but it is a working part of the construction: it is exactly what will handle the degenerate cases in higher dimensions.
Dimension 1
The standard normal distribution can be introduced in two equivalent ways β both familiar to you (topics 1 and 8):
Lemma 1 (The standard normal distribution) For a random variable \(Y\), the following are equivalent:
\(Y\) has density \(f(y)=\frac{1}{\sqrt{2\pi }}e^{-y^2/2}\);
\(Y\) has characteristic function \(\varphi_Y(t)=e^{-t^2/2}\). We write \(Y \sim \mathscr {N}\left(0, 1\right)\).
Now let us describe the entire family of one-dimensional Gaussian distributions β in a deliberately cumbersome form, which will then generalize verbatim to \(\mathbb {R}^{n}\).
Theorem 1 (Equivalent definitions of the one-dimensional Gaussian distribution) For a random variable \(X\), the following are equivalent:
\(X\) is an affine transformation of a standard normal variable: there exist \(\mu \in \mathbb {R}\), \(a \in \mathbb {R}\) such that
\[
X = \mu + a Y, \qquad Y \sim \mathscr {N}\left(0, 1\right);
\]
the characteristic function has the form \(\varphi_X(t) = e^{i\mu t - \frac{1}{2}\sigma^2 t^2}\) for some \(\mu \in \mathbb {R}\), \(\sigma^2 \geq 0\);
the support of the distribution of \(X\) is an affine subspace of \(\mathbb {R}\), and on it the distribution is βbell-shapedβ: either \(X = \mu\) a.s. (the case \(\sigma = 0\): the support is a point, an affine subspace of dimension \(0\)), or \(X\) has density
\[
f_X(x) = \frac{1}{\sqrt{2\pi }\, \sigma }\, e^{-\frac{(x-\mu )^2}{2\sigma ^2}}, \qquad \sigma > 0
\]
(the support is the whole line, an affine subspace of dimension \(1\)). Here the parameters are consistent: \(\mu = \mathbb {E}\left[X\right]\), \(\sigma^2 = \operatorname {Var}\left[X\right] = a^2\). We write \(X \sim \mathscr {N}\left(\mu , \sigma^2\right)\).
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/100/03_gaussian_1d_cases_fig.tex.
Why such cumbersomeness, if the one-dimensional normal distribution lives perfectly well without it? Because it is exactly in this form β (1) an affine image of the standard distribution, (2) a special CF, (3) a support-subspace with a bell on it β that each item generalizes verbatim to the multivariate case.
Corollary 1 (Affine stability) If \(X \sim \mathscr {N}\left(\mu , \sigma^2\right)\) and \(a, b \in \mathbb {R}\), then \(aX + b \sim \mathscr {N}\left(a\mu + b, \, a^2\sigma^2\right)\).
The Gaussian distribution in
The standard vector and the general definition
Let us proceed by the same scheme: first the standard \(n\)-dimensional vector, then an arbitrary Gaussian vector as its affine image.
Lemma 2 (The standard Gaussian vector) For a random vector \(Y = (Y_1, \ldots , Y_n)\) in \(\mathbb {R}^{n}\), the following are equivalent:
the components \(Y_1, \ldots , Y_n\) are independent and each \(\sim \mathscr {N}\left(0, 1\right)\);
\(Y\) has density \(f_n(y)=\frac{1}{(2\pi )^{n/2}}\, e^{-\left\| y\right\|^2/2}\), where \(\left\| y\right\|^2 = y_1^2 + \ldots + y_n^2\);
\(Y\) has characteristic function \(\varphi_{Y}(t)=e^{-\left\| t\right\|^2/2}\), \(t \in \mathbb {R}^{n}\). We write \(Y \sim \mathscr {N}\left(0, I_n\right)\).
\((1)\Rightarrow (2)\): the density of a vector with independent components is the product of the densities (topic 2): \(\prod_j \frac{1}{\sqrt{2\pi }}e^{-y_j^2/2} = f_n(y)\). \((2)\Rightarrow (3)\): from the multiplicative structure of \(f_n\), the characteristic function factors into a product of one-dimensional ones: \(\prod_j e^{-t_j^2/2} = e^{-\left\| t\right\|^2/2}\) (lemma Lemma 1). \((3)\Rightarrow (1)\): the joint CF factors into a product of the CFs of the coordinates β this is equivalent to independence of the coordinates (the factorization criterion, topics 2 and 8), and each coordinate is standard by lemma Lemma 1.
The density in item 2 depends only on \(\left\| y\right\|\): the standard vector is isotropic (invariant under rotations). The combination βindependent coordinates + spherical symmetryβ is a unique privilege of the Gaussian distribution (Maxwellβs theorem); it will come into play more than once below.
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/100/04_std_normal_2d_density_fig.tex.
Definition 3 A random vector \(X\) in \(\mathbb {R}^{n}\) is called Gaussian if there exist \(\mu \in \mathbb {R}^{n}\) and a matrix \(A\) such that \[
X = \mu + AY, \qquad Y \sim \mathscr {N}\left(0, I_m\right),
\] where \(A \in \mathbb {R}^{n \times m}\) (one may take \(m = n\); rectangular matrices are allowed β see the remark below).
No non-degeneracy of \(A\) is required β and this is essential: degenerate Gaussian vectors, whose entire mass lives on an affine subspace of smaller dimension, are ubiquitous in applications (regression residuals, frequencies with a fixed sum).
Parameters: expectation, covariance matrix, CF
In the one-dimensional case the distribution was completely determined by the pair \((\mu , \sigma^2)\). Let us show that in \(\mathbb {R}^{n}\) too, a Gaussian vector is determined by two parameters β a mean vector and a covariance matrix, which replaces the variance.
Theorem 2 (Parameters of a Gaussian vector) Let \(X = \mu + AY\), \(Y \sim \mathscr {N}\left(0, I_m\right)\). Then:
\(\mathbb {E}\left[X\right] = \mu\);
\(\operatorname {Var}\left[X\right] = AA^T =: \Sigma\) (the covariance matrix, \(\Sigma_{ij} = \operatorname {Cov}\left[ X_i, X_j \right]\));
the characteristic function
\[
\varphi _{X}(t) = \exp \left(i\, \mu ^T t - \tfrac 12\, t^T \Sigma \, t\right), \qquad t \in \mathbb {R}^{n}.
\]
- Linearity: \(\mathbb {E}\left[X\right] = \mu + A\, \mathbb {E}\left[Y\right] = \mu\). (2) \(\operatorname {Var}\left[X\right] = \mathbb {E}\left[AY(AY)^T\right] = A\, \mathbb {E}\left[YY^T\right]A^T = AA^T\), since \(\mathbb {E}\left[YY^T\right] = I_m\) (ones on the diagonal β variances, zeros off it β independence). (3) \(\varphi_X(t) = \mathbb {E}\left[e^{it^T(\mu + AY)}\right] = e^{it^T\mu }\, \varphi_Y(A^Tt) = e^{it^T\mu }\, e^{-\frac12\left\| A^Tt\right\|^2}\), and \(\left\| A^Tt\right\|^2 = t^TAA^Tt = t^T\Sigma t\).
Thus the matrix \(A\) enters the distribution only through the product \(\Sigma = AA^T\) β \(A\) itself is nothing more than βscaffoldingβ.
Theorem 3 (Uniqueness of the parameters) The distribution of a Gaussian vector is uniquely determined by the pair \((\mu , \Sigma )\):
if \(A_1A_1^T = A_2A_2^T = \Sigma\), then the vectors \(\mu + A_1Y_1\) and \(\mu + A_2Y_2\) have the same distribution;
conversely, for any \(\mu \in \mathbb {R}^{n}\) and symmetric positive semi-definite \(\Sigma\), there exists a Gaussian vector with \(\mathbb {E}\left[X\right] = \mu\), \(\operatorname {Var}\left[X\right] = \Sigma\). We write \(X \sim \mathscr {N}\left(\mu , \Sigma \right)\).
- By theorem Theorem 2(3), both CFs equal \(e^{i\mu^Tt - \frac12 t^T\Sigma t}\); the uniqueness theorem for CFs (the multivariate version) identifies the distributions. (2) By the spectral theorem \(\Sigma = QDQ^T\) with orthogonal \(Q\) and diagonal \(D = \operatorname {diag}(\lambda_1, \ldots , \lambda_n)\), \(\lambda_i \geq 0\). Set \(A := QD^{1/2}\); then \(AA^T = QDQ^T = \Sigma\), and the vector \(\mu + AY\) is the one we sought.
Definition 4 The number \(k := \operatorname {rk}\Sigma \in \left\{ 0, 1, \ldots , n\right\}\) is called the rank of the Gaussian vector \(X \sim \mathscr {N}\left(\mu , \Sigma \right)\). When \(k = n\) the vector is called non-degenerate (full rank), when \(k < n\) β degenerate.
Lemma 3 For any \(A \in \mathbb {R}^{n \times m}\), \(\operatorname {rk}(AA^T) = \operatorname {rk} A\) β the rank of the vector can also be computed from the matrix \(A\) in the representation.
It suffices to check \(\ker (AA^T) = \ker (A^T)\): the inclusion \(\supseteq\) is obvious, and if \(AA^Tx = 0\), then \(0 = x^TAA^Tx = \left\| A^Tx\right\|^2\), whence \(A^Tx = 0\). The nullities are equal β so are the ranks.
Dimension reduction
If \(k < n\), then to construct \(X\) a \(k\)-dimensional standard vector actually suffices β as in the one-dimensional case, where for \(\sigma = 0\) the \(0\)-dimensional distribution sufficed.
Proposition 2 (Dimension reduction) Let \(X = \mu + AY\) be a Gaussian vector of rank \(k = \operatorname {rk}A\). Then there exist a matrix \(A' \in \mathbb {R}^{n \times k}\) of full column rank \(k\) and a vector \(Y' \sim \mathscr {N}\left(0, I_k\right)\) such that \(X = \mu + A'Y'\).
The idea of the proof is the singular value decomposition: \(A = U_k \Sigma_k V_k^T\) (the zero singular values contribute nothing), \(A' := U_k\Sigma_k\), \(Y' := V_k^TY\); checking \(Y' \sim \mathscr {N}\left(0, I_k\right)\) is a direct computation of the CF, using \(V_k^TV_k = I_k\).
Let \(n = 2\), \(\mu = 0\), \(A=\begin{psmallmatrix} 1& 1\\ 1& 1\end{psmallmatrix}\) β a matrix of rank \(1\). Find a representation of the vector \(X = AY\) via a single standard normal variable.
We have \(X = (Y_1+Y_2, \, Y_1+Y_2)\). The variable \(Y' := \frac{Y_1+Y_2}{\sqrt2}\) is standard: the sum \(Y_1 + Y_2 \sim \mathscr {N}\left(0, 2\right)\) (convolution, topic 3), and dividing by \(\sqrt2\) normalizes the variance. Then \[
X = \begin{psmallmatrix} \sqrt2\\ \sqrt2\end{psmallmatrix} Y' =: A'Y', \qquad A' \in \mathbb {R}^{2 \times 1}, \quad Y'\sim \mathscr {N}\left(0, 1\right)
\] β the same distribution, but with one (rather than two) independent standard variable. The support of \(X\) is the diagonal \(\left\{ x_1 = x_2\right\}\), an affine subspace of dimension \(1 = \operatorname {rk}A\).
Lemma 4 (Affine image of a Gaussian vector) Let \(X\) be a Gaussian vector in \(\mathbb {R}^{n}\), \(C \in \mathbb {R}^{m \times n}\), \(b \in \mathbb {R}^{m}\). Then \(Z = b + CX\) is a Gaussian vector in \(\mathbb {R}^{m}\), with \(Z \sim \mathscr {N}\left(b + C\mu , \; C\Sigma C^T\right)\).
Substituting \(X = \mu + AY\): \(Z = (b + C\mu ) + (CA)Y\) β again an affine image of a standard vector (with rectangular matrix \(CA\), as in the remark above). The parameters follow from theorem Theorem 2.
The Gaussian world is closed under linear algebra: sums, differences, projections, and rotations of Gaussian vectors are Gaussian, and everything is computed by matrix multiplication.
Marginal distributions and the criterion via linear combinations
Proposition 3 (Linear combinations of coordinates) Let \(X \sim \mathscr {N}\left(\mu , \Sigma \right)\) in \(\mathbb {R}^{n}\), \(c \in \mathbb {R}^{n}\). Then \[
c^T X = c_1X_1 + \ldots + c_nX_n \; \sim \; \mathscr {N}\left(c^T\mu , \; c^T\Sigma c\right).
\] In particular (\(c = e_i\)): each coordinate \(X_i \sim \mathscr {N}\left(\mu_i, \Sigma_{ii}\right)\).
This is a special case of lemma Lemma 4 with \(m = 1\). The converse is false: normality of each coordinate separately does not imply Gaussianity of the vector β counterexamples below (the βcheckerboardβ example and \((\xi , \eta \xi )\)). Normality of the marginals is necessary but not sufficient. But if we require normality of all linear combinations at once β that is already sufficient:
Theorem 4 (Criterion for Gaussianity via linear combinations) A vector \(X\) is Gaussian if and only if for every \(c \in \mathbb {R}^{n}\) the variable \(c^T X\) has a one-dimensional Gaussian distribution (possibly degenerate).
(\(\Rightarrow\)) β proposition Proposition 3. (\(\Leftarrow\)) Each coordinate is Gaussian, so the second moments are finite and \(\mu := \mathbb {E}\left[X\right]\), \(\Sigma := \operatorname {Var}\left[X\right]\) are well defined. The variable \(c^TX\) is Gaussian with parameters \(c^T\mu\), \(c^T\Sigma c\), so its CF at the point \(s = 1\) equals \[
\varphi _X(c) = \mathbb {E}\left[e^{i \, c^TX}\right] = \varphi _{c^TX}(1) = e^{i c^T\mu - \frac12 c^T\Sigma c}.
\] The point \(c\) is arbitrary β so the CF of the vector \(X\) has exactly the Gaussian form from theorem Theorem 2(3), and by theorem Theorem 3(2) plus uniqueness of the CF, the vector \(X\) is distributed as \(\mathscr {N}\left(\mu , \Sigma \right)\).
The full-rank density
Proposition 4 (Density) If \(\det \Sigma \neq 0\), then \(X \sim \mathscr {N}\left(\mu , \Sigma \right)\) has density \[
f_X(x) = \frac{1}{(2\pi )^{n/2}\sqrt{\det \Sigma }} \exp \left(-\tfrac 12 (x - \mu )^T \Sigma ^{-1} (x-\mu )\right), \qquad x \in \mathbb {R}^{n}.
\] The level curves are ellipsoids with axes along the eigenvectors of \(\Sigma\) and semi-axes \(\propto \sqrt{\lambda_i}\). When \(\det \Sigma = 0\), there is no density on \(\mathbb {R}^{n}\): the entire mass lies on the affine subspace \(\mu + \operatorname {Im}A\) of dimension \(\operatorname {rk}\Sigma\).
(Derivation β the change of variables from topic 2 applied to the representation \(X = \mu + AY\) with non-degenerate \(A\): the Jacobian \(\left|\det A^{-1}\right| = 1/\sqrt{\det \Sigma }\).)
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/100/01_gaussian_contours_fig.tex.
Four equivalent definitions
Let us gather everything together β the multivariate analogue of theorem Theorem 1. To speak of a density on an affine subspace \(H\) of dimension \(k < n\) (a degenerate vector has no ordinary \(n\)-dimensional density), it suffices to choose orthonormal coordinates \(z \in \mathbb {R}^{k}\) on \(H\) and specify the density in them.
Theorem 5 (Equivalent definitions of a Gaussian vector) For a random vector \(X\) in \(\mathbb {R}^{n}\), the following are equivalent:
\(X\) is an affine image of a standard Gaussian vector: \(X = \mu + AY\), \(Y \sim \mathscr {N}\left(0, I_m\right)\);
the characteristic function has the form \(\varphi_X(t) = \exp \left(i\mu^Tt - \frac12 t^T\Sigma t\right)\) with symmetric positive semi-definite \(\Sigma\);
the support of the distribution of \(X\) is an affine subspace \(H \subseteq \mathbb {R}^{n}\) of dimension \(k\), and in suitable orthonormal coordinates on \(H\) the distribution has a non-degenerate Gaussian density \(\frac{1}{(2\pi )^{k/2}\sqrt{\det \Sigma '}}e^{-\frac12 z^T(\Sigma ')^{-1}z}\) (for \(k = 0\): \(X\) is a.s. constant);
any linear combination \(c^T X\), \(c \in \mathbb {R}^{n}\), has a one-dimensional Gaussian distribution (possibly degenerate). In all items, \(\mu = \mathbb {E}\left[X\right]\), \(\Sigma = \operatorname {Var}\left[X\right]\), \(k = \operatorname {rk}\Sigma\).
Scheme of the proof
\((1)\Leftrightarrow (2)\) β theorems Theorem 2(3) and Theorem 3; \((2)\Leftrightarrow (4)\) β theorem Theorem 4. The link with (3): in the direction \((1)\Rightarrow (3)\), dimension reduction (proposition Proposition 2) gives \(X = \mu + A'Y'\) with \(A'\) of full column rank \(k\); the support is the image of \(\mathbb {R}^{k}\) under a linear injection, i.e. the affine subspace \(H = \mu + \operatorname {Im}A'\), and in an orthonormal basis of \(\operatorname {Im}A'\) (the left singular vectors of \(A'\)), the coordinates of \(X\) are a non-degenerate linear image of \(Y'\), which has a Gaussian density (proposition Proposition 4 in dimension \(k\)). Conversely, \((3)\Rightarrow (1)\): the bell in coordinates on \(H\) is a linear image of a standard vector (spectral decomposition \(\Sigma ' = RDR^T\), the substitution \(W = D^{-1/2}R^Tz\)), and, unwinding the parametrization of \(H\), we obtain the representation \(X = \mu + (ERD^{1/2})W\), where \(E\) is the matrix of basis vectors.
In total, four equivalent viewpoints on one object: an affine image of independent standard variables; a distribution with a special CF; a bell on an affine subspace; an object all of whose one-dimensional projections are normal. We will now use all four to quickly check the (non-)Gaussianity of specific vectors.
When a vector is Gaussian, and when it is not
Let us work through four telling examples in \(\mathbb {R}^{2}\) β one certainly Gaussian and three borderline ones, where intuition misleads.
Is \((X_1, X_2) \sim \mathscr {N}\left(0, I_2\right)\) Gaussian?
Yes, by the very definition Definition 3 (\(\mu = 0\), \(A = I_2\)), of full rank \(2\): the density is a rotationally symmetric bell \(f_2(x_1,x_2)=\frac{1}{2\pi }e^{-(x_1^2+x_2^2)/2}\), the support is all of \(\mathbb {R}^{2}\) (the unique two-dimensional affine subspace of the plane).
Take the density \(f_2\) of the standard vector, zero it out in the quadrants where \(x_1x_2 > 0\), and double it where \(x_1 x_2 < 0\): \[
g(x_1,x_2) := \begin{cases} 2f_2(x_1,x_2), & x_1x_2<0,\\ 0, & x_1x_2>0. \end{cases}
\] Show that \(g\) is a valid density, that the vector with density \(g\) is not Gaussian, but that both of its coordinates are standard Gaussian variables.
Validity: by symmetry \(\mathbb {P}\left(X_1X_2 < 0\right) = \frac12\) for the standard vector, so \(\iint g = 2 \cdot \frac12 = 1\).
Not Gaussian β via the support (item 3). The support of \(g\) is the union of two closed quadrants \(H = \left\{ x_1x_2 \leq 0\right\}\). It has interior points (were it an affine subspace, it would have dimension \(2\)), but \(H \neq \mathbb {R}^{2}\) (the point \((1,1) \notin H\)) β while the only two-dimensional affine subspace of the plane is the whole plane. So \(H\) is not an affine subspace, and by theorem Theorem 5(3) the vector is not Gaussian.
Not Gaussian β via a linear combination (item 4). Take \(S = X_1 + X_2\). Its density at zero:
\[
f_S(0) = \int _{\mathbb {R}} g(x, -x)\, dx = 2\int _{\mathbb {R}} f(x)^2\, dx = \frac{2}{2\pi }\int _{\mathbb {R}} e^{-x^2}dx = \frac{1}{\sqrt{\pi }}.
\]
On the other hand, \(\mathbb {E}\left[S\right] = 0\), and \(\operatorname {Cov}\left[ X_1, X_2 \right] = 2\iint_{x_1x_2<0}x_1x_2 f_2 = -\frac{2}{\pi }\) (the product of two one-dimensional integrals \(\int_0^\infty x f(x)dx = \frac{1}{\sqrt{2\pi }}\) on each half of the region), so were \(S\) Gaussian we would have \(S \sim \mathscr {N}\left(0, 2 - \frac{4}{\pi }\right)\) and \(f_S(0) = \frac{1}{\sqrt{2\pi (2 - 4/\pi )}} = \frac{1}{\sqrt{4\pi - 8}}\). But \(\pi \neq 4\pi - 8\) β a contradiction: \(S\) is not Gaussian.
The coordinates are Gaussian. For \(x_1 > 0\) the condition \(x_1x_2 < 0\) means \(x_2 < 0\):
\[
f_{X_1}(x_1) = \int _{-\infty }^{0} 2f(x_1)f(x_2)\, dx_2 = 2f(x_1)\cdot \tfrac 12 = f(x_1),
\]
and likewise for \(x_1 < 0\): \(X_1 \sim \mathscr {N}\left(0, 1\right)\), and by symmetry so is \(X_2\). So both marginals are standard normal, while the vector is not.
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/100/05_checkerboard_density_fig.tex.
Let \(\xi \sim \mathscr {N}\left(0, 1\right)\), \(X = (\xi , \xi )\). Is this a Gaussian vector?
Yes: \(X = \begin{psmallmatrix} 1\\ 1\end{psmallmatrix}\xi\) is a linear image of a one-dimensional standard vector (representation (1) with a rectangular \(2 \times 1\) matrix). The support is the diagonal \(H = \left\{ x_1 = x_2\right\}\), a one-dimensional subspace; there is no two-dimensional density (\(\Sigma = \begin{psmallmatrix} 1& 1\\ 1& 1\end{psmallmatrix}\), \(\det \Sigma = 0\), rank \(1\)). But along the diagonal, in the orthonormal coordinate \(z = \frac{x_1+x_2}{\sqrt2}\), the vector behaves like an ordinary one-dimensional Gaussian variable \(Z = \sqrt2\, \xi \sim \mathscr {N}\left(0, 2\right)\) with a bell-shaped density β exactly case (3) of theorem Theorem 5 with \(k = 1\).
Let \(\xi \sim \mathscr {N}\left(0, 1\right)\), and independently \(\eta = \pm 1\) with probabilities \(\frac12\). Investigate the vector \(X = (\xi , \eta \xi )\).
When \(\eta = 1\) the vector lies on the diagonal \(H_+ = \left\{ x_1 = x_2\right\}\), when \(\eta = -1\) β on the anti-diagonal \(H_- = \left\{ x_1 = -x_2\right\}\); the support is the cross \(H_+ \cup H_-\) (each line with probability \(\frac12\)).
Not Gaussian β the support. The cross is not an affine subspace: it is larger than each of the lines, but does not coincide with \(\mathbb {R}^{2}\) (the point \((1,0)\) does not lie on the cross). Theorem Theorem 5(3) rules out Gaussianity.
Not Gaussian β a linear combination. \(S = X_1 + X_2 = \xi (1 + \eta )\): when \(\eta = -1\) this is identically zero, when \(\eta = 1\) it equals \(2\xi\). So \(S\) is a mixture of \(\mathscr {N}\left(0, 4\right)\) and an atom at zero with weights \(\frac12\): a non-degenerate Gaussian distribution has no atoms, while a degenerate one is entirely an atom. Hence \(S\) is not Gaussian, and by criterion (4) the vector is not Gaussian.
The marginals are Gaussian. \(X_1 = \xi\) β obviously. For \(X_2 = \eta \xi\), compute the CF (independence of \(\eta\) and \(\xi\), evenness of \(\varphi_\xi\)):
\[
\varphi _{X_2}(t) = \tfrac 12\mathbb {E}\left[e^{it\xi }\right] + \tfrac 12\mathbb {E}\left[e^{-it\xi }\right] = \varphi _\xi (t) = e^{-t^2/2},
\]
i.e. \(X_2 \sim \mathscr {N}\left(0, 1\right)\) β a βrandomly reflectedβ copy of \(\xi\) is again standard.
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/100/06_two_diagonals_fig.tex.
The main lesson of these examples: to establish Gaussianity of a vector, it is not enough to check its components β one must either exhibit an affine transformation of a standard vector (1), or check the CF (2), or (often easiest for counterexamples) look at the support (3), or find a βbadβ linear combination (4).
Conditional expectation of one component given another
You already know the general construction of conditional expectation (topic 11) and its geometry: \(\mathbb {E}\left[ X \mid \mathcal{G} \right]\) is the orthogonal projection onto \(L^2(\mathcal{G})\). In general, for \(\mathcal{G} = \sigma (Y)\) this projection is an arbitrarily complicated nonlinear function of \(Y\). For Gaussian vectors a miracle occurs: the projection onto all of \(L^2(\sigma (Y))\) coincides with the projection onto the tiny subspace \(\operatorname {span}\left\{ 1, Y_1, \ldots , Y_q\right\}\) β the conditional expectation turns out to be an affine function of \(Y\).
Theorem 7 (The conditional distribution of part of a Gaussian vector) Let \((X, Y)\) be a jointly Gaussian vector, \(X \in \mathbb {R}^{p}\), \(Y \in \mathbb {R}^{q}\), with mean blocks \(\mu_X, \mu_Y\) and covariance blocks \(\Sigma_{XX}, \Sigma_{XY}, \Sigma_{YX}, \Sigma_{YY}\), where \(\Sigma_{YY}\) is non-degenerate. Then the conditional distribution of \(X\) given \(Y\) is Gaussian, with parameters \[
\mathbb {E}\left[ X \mid Y \right] = \mu _X + \Sigma _{XY}\Sigma _{YY}^{-1}(Y - \mu _Y), \qquad \operatorname {Var}\left[ X \mid Y \right] = \Sigma _{XX} - \Sigma _{XY}\Sigma _{YY}^{-1}\Sigma _{YX},
\] and the conditional covariance is a constant matrix, not depending on the value of \(Y\).
Set \(B := \Sigma_{XY}\Sigma_{YY}^{-1}\), \[
\widehat{X} := \mu _X + B(Y - \mu _Y), \qquad Z := X - \widehat{X}.
\] The vector \(\widehat{X}\) is an affine function of \(Y\), i.e. \(\sigma (Y)\)-measurable; the pair \((Z, Y)\) is a linear image of the Gaussian vector \((X, Y)\), hence jointly Gaussian (lemma Lemma 4). Compute the parameters of \(Z\):
\(\mathbb {E}\left[Z\right] = \mu_X - \left(\mu_X + B(\mu_Y - \mu_Y)\right) = 0\);
the cross-covariance \(\operatorname {Cov}\left[ Z, Y \right] = \Sigma_{XY} - B\Sigma_{YY} = \Sigma_{XY} - \Sigma_{XY} = 0\);
\(\operatorname {Var}\left[Z\right] = \Sigma_{XX} - 2\, \Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX} + B\Sigma_{YY}B^T = \Sigma_{XX} - \Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}\).
The pair \((Z, Y)\) is Gaussian with zero cross-covariance β by theorem Theorem 6, \(Z\) and \(Y\) are independent. Hence, given \(Y = y\), the distribution of \(Z\) is its unconditional distribution \(\mathscr {N}\left(0, \Sigma_{XX} - \Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}\right)\), not depending on \(y\); and \(X = \widehat{X} + Z\), where \(\widehat{X}\), given this condition, is the constant \(\mu_X + B(y - \mu_Y)\). Altogether, the conditional distribution of \(X\) is Gaussian with a shifted mean and constant covariance.
Corollary 2 (The conditional second moment) For \(p = 1\) (scalar \(X\)): \[
\mathbb {E}\left[ X^2 \mid Y \right] = \operatorname {Var}\left[ X \mid Y \right] + \left(\mathbb {E}\left[ X \mid Y \right]\right)^2.
\]
Let us work through three examples β one symbolic and two numerical.
\((X, Y)\) is a Gaussian vector with means \(a_X, a_Y\), variances \(\sigma_X^2, \sigma_Y^2\), and covariance \(\sigma_{XY}\). Find \(\mathbb {E}\left[ X \mid Y \right]\) and \(\mathbb {E}\left[ X \mid X+Y \right]\).
Directly by theorem Theorem 7 (scalar blocks): \[
\mathbb {E}\left[ X \mid Y \right] = a_X + \frac{\sigma _{XY}}{\sigma _Y^2}\, (Y - a_Y).
\] For the second one, set \(S = X + Y\); the pair \((X, S)\) is Gaussian (a linear image), with \(\operatorname {Cov}\left[ X, S \right] = \sigma_X^2 + \sigma_{XY}\) and \(\operatorname {Var}\left[S\right] = \sigma_X^2 + 2\sigma_{XY} + \sigma_Y^2\): \[
\mathbb {E}\left[ X \mid X+Y \right] = a_X + \frac{\sigma _X^2+\sigma _{XY}}{\sigma _X^2+2\sigma _{XY}+\sigma _Y^2}\, \left(X+Y-a_X-a_Y\right).
\] Notice the scheme: to condition on a function of the components, include it in a Gaussian vector and apply the theorem to the new pair.
\((X, Y) \sim \mathscr {N}\left((1, -1), \Sigma \right)\), \(\Sigma = \begin{psmallmatrix} 5& 2\\ 2& 1\end{psmallmatrix}\). Find \(\mathbb {E}\left[ (X-Y)^2 \mid X+Y \right]\).
Set \(U = X - Y\), \(S = X + Y\); the pair \((U, S)\) is Gaussian. By bilinearity of covariance: \[
\begin{gathered}
\mathbb {E}\left[U\right] = 2, \qquad \mathbb {E}\left[S\right] = 0, \\
\operatorname {Var}\left[U\right] = 5 + 1 - 2\cdot2 = 2, \qquad \operatorname {Var}\left[S\right] = 5 + 1 + 2\cdot2 = 10, \qquad \operatorname {Cov}\left[ U, S \right] = \operatorname {Var}\left[X\right] - \operatorname {Var}\left[Y\right] = 4
\end{gathered}
\] (the cross-covariances in \(\operatorname {Cov}\left[ X-Y, X+Y \right]\) cancel each other out). By theorem Theorem 7: \[
\mathbb {E}\left[ U \mid S \right] = 2 + \frac{4}{10}\, S = 2 + \frac25 S, \qquad \operatorname {Var}\left[ U \mid S \right] = 2 - \frac{16}{10} = \frac25,
\] and by corollary Corollary 2: \[
\mathbb {E}\left[ (X-Y)^2 \mid X+Y \right] = \frac25 + \left(2 + \frac25 (X+Y)\right)^2 = \frac{22}{5}+\frac85(X+Y)+\frac{4}{25}(X+Y)^2.
\]
\((X, Y, Z) \sim \mathscr {N}\left((-1, 0, 2), \Sigma \right)\), \(\Sigma = \begin{psmallmatrix} 3& -2& 2\\ -2& 4& -1\\ 2& -1& 3\end{psmallmatrix}\). Find \(\mathbb {E}\left[ X - Y - Z \mid X + 2Y + Z \right]\).
Set \(U = c_1^T(X,Y,Z)\), \(S = c_2^T(X,Y,Z)\) with \(c_1 = (1,-1,-1)^T\), \(c_2 = (1,2,1)^T\); the pair \((U, S)\) is Gaussian. Compute in matrix form: \[
\Sigma c_2 = \begin{psmallmatrix} 1\\ 5\\ 3\end{psmallmatrix}, \qquad \operatorname {Var}\left[S\right] = c_2^T\Sigma c_2 = 1 + 10 + 3 = 14, \qquad \operatorname {Cov}\left[ U, S \right] = c_1^T\Sigma c_2 = 1 - 5 - 3 = -7,
\] and \(\mathbb {E}\left[U\right] = -3\), \(\mathbb {E}\left[S\right] = 1\). By theorem Theorem 7: \[
\mathbb {E}\left[ U \mid S \right] = -3 - \frac{7}{14}\, (S - 1) = -\frac52 - \frac12\, \left(X + 2Y + Z\right).
\] The entire computation is two matrix-vector products: βprobability = linear algebraβ.
In brief
A Gaussian vector is an affine image \(\mu + AY\) of a standard vector; the distribution is determined by the pair \((\mu , \Sigma = AA^T)\); CF: \(\exp \left(i\mu^Tt - \frac12 t^T\Sigma t\right)\).
The rank \(\operatorname {rk}\Sigma\) separates non-degenerate vectors (with a density having ellipsoidal level sets) from degenerate ones (mass on an affine subspace of dimension \(\operatorname {rk}\Sigma\) β with a bell on it).
Four equivalent definitions: affine image / CF / bell on an affine subspace / all linear combinations normal. Normality of the marginals alone is not sufficient (the βcheckerboardβ example, the cross \((\xi , \eta \xi )\)).
Linear images of Gaussian vectors are Gaussian; parameters transform via \(\mu \mapsto b + C\mu\), \(\Sigma \mapsto C\Sigma C^T\).
For jointly Gaussian vectors, uncorrelatedness is equivalent to independence.
The conditional expectation of one part given another is an affine function of the condition: \(\mathbb {E}\left[ X \mid Y \right] = \mu_X + \Sigma_{XY}\Sigma_{YY}^{-1}(Y - \mu_Y)\); the conditional covariance is constant. The linear forecast is optimal among all forecasts.
The multivariate CLT
Theorem 8 (The multivariate CLT) Let \(X^{(1)}, X^{(2)}, \ldots\) be independent identically distributed random vectors in \(\mathbb {R}^{d}\) with \(\mathbb {E}\left[X^{(1)}\right] = \mu\) and covariance matrix \(\Sigma\). Then \[
\sqrt{n}\left(\frac{X^{(1)} + \ldots + X^{(n)}}{n} - \mu \right) \xrightarrow {d} \mathscr {N}\left(0, \Sigma \right).
\]
The proof is a free application of everything we have built: by the criterion of linear combinations (theorem Theorem 4) and LΓ©vyβs theorem, it suffices to check convergence of the CF at every point \(t\), and this is the one-dimensional CLT for the projections \(t^T X^{(i)}\) (the CramΓ©r-Wold device). The limiting dependence of the coordinates is exactly the covariance one.
Problems
Let \(Y_1, Y_2 \sim \mathscr {N}\left(0, 1\right)\) be independent, \[
X_1 = Y_1 - Y_2 + 1, \qquad X_2 = 2Y_1 + Y_2, \qquad X_3 = X_1 + X_2.
\] Find \(\mu = \mathbb {E}\left[X\right]\) and \(\Sigma = \operatorname {Var}\left[X\right]\) for \(X = (X_1, X_2, X_3)\). What is the rank of \(X\)? Describe the support of the distribution as an affine subspace of \(\mathbb {R}^{3}\).
Prove the βfifth definitionβ: a vector \(X\) is Gaussian if and only if there exists an orthogonal matrix \(C\) such that the components of \(CX\) are independent and normal.
\((X,Y)\) is a Gaussian vector with \(\operatorname {Var}\left[X\right]=\operatorname {Var}\left[Y\right]\). Prove that \(X+Y\) and \(X-Y\) are independent.
The characteristic function of the vector \((X, Y)\) equals \[
\varphi (t_1, t_2) = \exp \left(-\tfrac 12\left(2t_1^2 + a\, t_1 t_2 + 3 t_2^2\right)\right).
\] For which \(a\) is this a valid CF? For which \(a\) are the variables \(X + 2Y\) and \(2X - Y\) independent?
\(X, Y \sim \mathscr {N}\left(0, \sigma^2\right)\) are independent. Prove: (a) \(X/Y \sim \operatorname {Cauchy}(0,1)\); (b) \(\dfrac {X-Y}{X+Y} \sim \operatorname {Cauchy}(0,1)\).
Gaussian marginals without joint Gaussianity, a threshold version
Let \(X \sim \mathscr {N}\left(0, 1\right)\), \(c > 0\), and \[
Y = \begin{cases} X, & \left|X\right| \leq c, \\ -X, & \left|X\right| > c. \end{cases}
\] Prove: (a) \(Y \sim \mathscr {N}\left(0, 1\right)\); (b) the vector \((X, Y)\) is not Gaussian; (c) there exists \(c_0\) for which \(X\) and \(Y\) are uncorrelated but dependent.
A die is thrown \(n\) times; \(\nu_j\) is the number of times face \(j\) comes up. Find the limiting distribution of the vector \[
\zeta _n = \sqrt{n}\left(\frac{\nu _1}{n} - \frac16, \; \frac{\nu _2}{n} - \frac16\right).
\] What is the rank of the limiting distribution of the full vector of all six frequencies?