This text is still being edited: it may contain typos, inaccuracies, and locally unpolished spots. If you find something, please let the author know (contact details on the course page).
Real data is almost never one-dimensional: height and weight, stock price and oil price, response time and server load. A pair of random variables is not βtwo random variablesβ but a single object β a random vector: its distribution contains information not only about each coordinate but also about the dependence between them. Losing this information is costly: the 2008 financial crisis was, to a significant extent, a story about how risk models knew the marginal distributions of defaults individually but modeled their joint distribution incorrectly. In this topic we will learn to work with joint distributions: finding marginals, checking independence, and β the main computational technique β transforming densities under smooth changes of variables (the Jacobian).
Joint distribution
Definition 1 A random vector is a measurable map \(\xi = (\xi_1, \ldots , \xi_n)\colon \Omega \to \mathbb {R}^{n}\) (equivalently: each coordinate \(\xi_i\) is a random variable). Its distribution is the measure \(\mathbb {P}_\xi (B) = \mathbb {P}\left(\xi \in B\right)\), \(B \in \mathscr {B}\left(\mathbb {R}^{n}\right)\); the joint distribution function is \[
F_\xi (x_1, \ldots , x_n) = \mathbb {P}\left(\xi _1 \leq x_1, \ldots , \xi _n \leq x_n\right).
\]
As in the one-dimensional case, the joint CDF is a complete passport of the distribution (βcornerβ rays form a \(\pi\)-system generating \(\mathscr {B}\left(\mathbb {R}^{n}\right)\)). But the properties βnon-decreasing + correct limitsβ no longer characterize a CDF: more is needed.
Proposition 1 (Probability of a rectangle) For \(n = 2\) and any \(a_1 < b_1\), \(a_2 < b_2\) \[
\mathbb {P}\left(\xi _1 \in (a_1, b_1], \; \xi _2 \in (a_2, b_2]\right) = F(b_1, b_2) - F(a_1, b_2) - F(b_1, a_2) + F(a_1, a_2) \geq 0.
\] Nonnegativity of this combination (together with right-continuity and the βcorrectβ limits; for \(n > 2\) β an analogous alternating sum over the vertices of a box) is exactly the characterization of joint CDFs.
Definition 2 A vector \(\xi\) has a (joint) density \(p_\xi\) if \[
\mathbb {P}\left(\xi \in B\right) = \int _B p_\xi (x_1, \ldots , x_n) \, dx_1 \cdots dx_n \qquad \text{for all } B \in \mathscr {B}\left(\mathbb {R}^{n}\right).
\] The marginal density of a coordinate is obtained by integrating out the others: \[
p_{\xi _1}(x_1) = \int _{\mathbb {R}^{n-1}} p_\xi (x_1, x_2, \ldots , x_n) \, dx_2 \cdots dx_n.
\]
The direction βjoint \(\to\) marginalsβ is mechanical integration. The reverse direction does not work: marginals do not determine the joint distribution.
Let \(X \sim \operatorname {U}[0,1]\). Consider two vectors: \((X, X)\) and \((X, Y)\), where \(Y \sim \operatorname {U}[0,1]\) is independent of \(X\). Show that they have the same marginal distributions but different joint distributions.
The marginals of both vectors are \(\operatorname {U}[0,1]\) in each coordinate. But the joint distributions are radically different: the vector \((X,X)\) lives on the diagonal of the square \([0,1]^2\) β a set of zero area, it does not even have a joint density! β while \((X,Y)\) is uniformly spread over the whole square. For instance, \[
\mathbb {P}\left(X = X\right) = 1, \qquad \mathbb {P}\left(X = Y\right) = 0.
\] The entire difference lies in the dependence structure. This is exactly why in risk management one cannot βglueβ a portfolio model from marginal models of the assets: a joint model is needed (in actuarial practice this gap is closed by copulas β functions that encode dependence separately from the marginals).
Independence and criteria
From topic 1: variables are independent if the \(\sigma\)-algebras they generate are independent, and by the criterion via \(\pi\)-systems it suffices to factorize the joint CDF. For vectors with a density it is even more convenient:
Proposition 2 (Independence criteria) Let \(\xi = (\xi_1, \ldots , \xi_n)\). The following conditions are equivalent:
\(\xi_1, \ldots , \xi_n\) are independent;
\(F_\xi (x_1, \ldots , x_n) = F_{\xi_1}(x_1) \cdots F_{\xi_n}(x_n)\) for all \(x_i\);
(if a joint density exists) \(p_\xi (x_1, \ldots , x_n) = p_{\xi_1}(x_1) \cdots p_{\xi_n}(x_n)\) almost everywhere. A practically stronger form of item 3: if \(p_\xi (x_1, \ldots , x_n) = g_1(x_1) \cdots g_n(x_n)\) for some nonnegative functions \(g_i\), then the coordinates are independent (the normalizing constants take care of themselves).
Change of variables
The main computational tool of this topic is transporting a density under a smooth change of coordinates. The one-dimensional case (topic 1) generalizes as follows:
Theorem 1 (Change of variables theorem) Let a random vector \(\xi\) have density \(p_\xi\) concentrated on an open set \(D \subseteq \mathbb {R}^{n}\), and let \(\Psi \colon D \to G \subseteq \mathbb {R}^{n}\) be a diffeomorphism (a bijection, smooth together with its inverse). Then the vector \(\eta = \Psi (\xi )\) has density \[
p_\eta (y) = p_\xi \left(\Psi ^{-1}(y)\right) \cdot \left|\det J_{\Psi ^{-1}}(y)\right|, \qquad y \in G,
\] where \(J_{\Psi^{-1}}\) is the Jacobian matrix of the inverse map.
The meaning of the Jacobian is a local volume distortion coefficient: a small box of volume \(dv\) near the point \(y\) is the image of a box of volume \(\left|\det J_{\Psi^{-1}}(y)\right| \, dv\) near the point \(\Psi^{-1}(y)\), and the probability of landing in it must be preserved. The proof is exactly this observation, applied via the change-of-variables formula for a multiple Lebesgue integral (ETMT).
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/020/01_jacobian_grid_fig.tex.
Working protocol for applying theorem Theorem 1 (learn it as an algorithm):
write out \(\Psi\) and find the image \(G\) of the support (do not forget the indicators!);
find the inverse map \(\Psi^{-1}\);
compute \(\det J_{\Psi^{-1}}\) (it is often easier to compute \(\det J_{\Psi }\) and invert: \(\det J_{\Psi^{-1}}(y) = 1\big/\det J_\Psi \left(\Psi^{-1}(y)\right)\));
assemble the formula and check normalization at least by a rough estimate.
If you need the density of a single function of the vector (say, \(\xi_1 + \xi_2\)), complete it to a full-dimensional change of variables (\(\eta_1 = \xi_1 + \xi_2\), \(\eta_2 = \xi_2\)), then integrate out the extra coordinate β we will turn this trick into an automatic reflex in topic 3 on convolutions.
A random vector \((X, Y)\) has a uniform distribution on the disk \(D=\left\{ (x, y)\colon x^{2}+y^{2} \leq 1\right\}\).
Find the densities of the random variables \(X\) and \(Y\). Are they independent?
Let \((R, \Theta )\) be the polar coordinates of the point \((X, Y)\). Find the density of the vector \((R, \Theta )\) and the densities of \(R\) and \(\Theta\). Are \(R\) and \(\Theta\) independent?
Is \(F(x, y)=\left(1-\frac{1}{x+y}\right) \; \mathbb {1}_{x+y \geq 1}\) the distribution function of some random vector?
A random vector \((X, Y)\) has density \[
p_{X, Y}(x, y)= \begin{cases} c x y, & 0 \leq x \leq 1, \; 0 \leq y \leq \sqrt{x}, \\ 0, & \text{otherwise}. \end{cases}
\] Find \(c\) and the marginal densities. Are \(X\) and \(Y\) independent? What is the probability \(\mathbb {P}\left(2X + Y \leq 1\right)\)?
Given independent \(\xi_{1}, \xi_{2} \sim \operatorname {U}[0,1]\) and \(\xi_{3} \sim \operatorname {Exp}(\lambda )\). Find \(\mathbb {P}\left(\xi_{1}+\xi_{3} \geq 3\right)\).
Let \(\xi_{1}, \xi_{2} \sim \operatorname {Exp}(1)\) be independent. Prove that the random variables \(\dfrac {\xi_{1}}{\xi_{1}+\xi_{2}}\) and \(\xi_{1}+\xi_{2}\) are independent, and find their distributions.
\(X, Y, Z \sim \operatorname {Exp}(1)\) are independent. What is the probability that a triangle with sides \(X, Y, Z\) exists?
Given independent \(\xi_{1} \sim \operatorname {Exp}(\lambda_1)\) and \(\xi_{2} \sim \operatorname {Pois}(\lambda_2)\). Find \(\mathbb {P}\left(\xi_{1} \geq 2 \xi_{2}\right)\).
Order statistics
Let \(\xi_{1}, \ldots , \xi_{n}\) be independent identically distributed variables with CDF \(F\) and density \(p\). Denote by \(\xi_{(1)} \leq \ldots \leq \xi_{(n)}\) their order statistics (the same values, arranged in increasing order). Find:
the density of the vector \(\left(\xi_{(1)}, \ldots , \xi_{(n)}\right)\);
the density of \(\xi_{(k)}\);
the density of the vector \(\left(\xi_{(1)}, \xi_{(n)}\right)\).
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/020/02_support_region_fig.tex.