This text is still being edited: it may contain typos, inaccuracies, and locally unpolished spots. If you find something, please let the author know (contact details on the course page).
Three storylines of this topic are united by one question: what can be said about the sum of random variables? Sums are the main characters of applied probability: the total loss of an insurance portfolio, the accumulated measurement error, the total demand for a product, the return of a portfolio of assets. Convolution answers the question βwhat is the distribution of the sum of independent termsβ, covariance answers βwhat happens to the variance of a sum of dependent termsβ, and inequalities (Cauchy-Bunyakovsky-Schwarz, Jensen, Lyapunov) are the working tools we will use to estimate moments for the rest of the course.
Convolution
Theorem 1 (The convolution formula) Let \(\xi \; \perp \! \! \! \perp \; \eta\). Then the distribution of the sum \(\xi + \eta\) is the convolution of the distributions:
(discrete case) \(\mathbb {P}\left(\xi + \eta = z\right) = \sum_{x} \mathbb {P}\left(\xi = x\right) \mathbb {P}\left(\eta = z - x\right)\);
(continuous case) if \(\xi\) has density \(p_\xi\), then \(\xi + \eta\) has density
\[
p_{\xi + \eta }(z) = \int _{\mathbb {R}} p_\xi (x) \, dF_\eta (z - x) \; \overset {\text{if } p_\eta \text{ exists}}{=} \; \int _{\mathbb {R}} p_\xi (x) \, p_\eta (z-x) \, dx.
\]
The derivation of the continuous case is a textbook application of the technique from topic 2: complete the sum to a full-dimensional change of variables \(\Psi (x, y) = (x + y, y)\) with Jacobian \(1\), obtain \(p_{(\xi +\eta ,\, \eta )}(z, y) = p_\xi (z - y) p_\eta (y)\), and integrate out the extra coordinate. Note: it suffices for one term to have a density β convolution βsmooths outβ (the sum of a continuous variable with anything is continuous).
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/030/01_uniform_convolution_fig.tex.
It is convenient to collect convolutions into a table of βstableβ families β check every row (some are in the problems below, some become trivial in topic 8 via characteristic functions):
| \(\operatorname {Pois}(\lambda _1) * \operatorname {Pois}(\lambda _2)\) |
\(\operatorname {Pois}(\lambda _1 + \lambda _2)\) |
| \(\Gamma (\alpha _1, \lambda ) * \Gamma (\alpha _2, \lambda )\) |
\(\Gamma (\alpha _1 + \alpha _2, \lambda )\) |
| \(\mathscr {N}\left(a_1, \sigma _1^2\right) * \mathscr {N}\left(a_2, \sigma _2^2\right)\) |
\(\mathscr {N}\left(a_1 + a_2, \sigma _1^2 + \sigma _2^2\right)\) |
| \(\operatorname {Cauchy} * \operatorname {Cauchy}\) |
\(\operatorname {Cauchy}\) (with summed parameters) |
Let \(\xi \sim \operatorname {Pois}(\lambda_1)\) and \(\eta \sim \operatorname {Pois}(\lambda_2)\) be independent. Prove that \(\xi + \eta \sim \operatorname {Pois}(\lambda_1 + \lambda_2)\).
The variables \(\xi_{1} \sim \operatorname {Bern}(0{.}5)\) and \(\xi_{2} \sim \operatorname {U}[0,1]\) are independent. Find the distribution of \(\xi_{1}+\xi_{2}\).
Let \(\xi_1, \ldots , \xi_n \sim \operatorname {Exp}(\lambda )\) be independent, \(S_n = \xi_1 + \ldots + \xi_n\).
Prove that \(S_n \sim \Gamma (n, \lambda )\).
(Commuter trains.) Trains arrive at independent \(\operatorname {Exp}(\lambda )\)-intervals. Let \(X_T\) be the number of trains that have passed by time \(T\). Prove that \(X_T \sim \operatorname {Pois}(\lambda T)\).
Let \(\xi , \eta \sim \mathscr {N}\left(0, 1\right)\) be independent. Find the distribution of \(\xi /\eta\).
\(X, Y, Z \sim \operatorname {U}[0,1]\) are independent. Find the distribution of \(W=(X Y)^{Z}\).
Covariance and correlation
Definition 1 For \(\xi , \eta \in L^2\), the covariance and correlation are: \[
\operatorname {Cov}\left[ \xi , \eta \right] = \mathbb {E}\left[(\xi - \mathbb {E}\left[\xi \right])(\eta - \mathbb {E}\left[\eta \right])\right] = \mathbb {E}\left[\xi \eta \right] - \mathbb {E}\left[\xi \right]\mathbb {E}\left[\eta \right], \qquad \operatorname {Corr}\left[ \xi , \eta \right] = \frac{\operatorname {Cov}\left[ \xi , \eta \right]}{\sqrt{\operatorname {Var}\left[\xi \right]\operatorname {Var}\left[\eta \right]}}.
\]
Proposition 1 (Properties)
Covariance is bilinear and symmetric; \(\operatorname {Cov}\left[ \xi , \xi \right] = \operatorname {Var}\left[\xi \right]\).
\(\operatorname {Var}\left[\sum_i \xi_i\right] = \sum_i \operatorname {Var}\left[\xi_i\right] + 2\sum_{i < j} \operatorname {Cov}\left[ \xi_i, \xi_j \right]\); for pairwise uncorrelated terms, variances add.
If \(\xi \; \perp \! \! \! \perp \; \eta\), then \(\operatorname {Cov}\left[ \xi , \eta \right] = 0\). The converse is false (problem below).
\(\left|\operatorname {Corr}\left[ \xi , \eta \right]\right| \leq 1\), with equality attained if and only if \(\eta = a\xi + b\) a.s. for some constants \(a \neq 0\), \(b\).
The covariance matrix of a vector \(\xi\), \(\Sigma_{ij} = \operatorname {Cov}\left[ \xi_i, \xi_j \right]\), is symmetric and positive semi-definite: for any \(v \in \mathbb {R}^{n}\)
\[
v^T \Sigma v = \operatorname {Var}\left[v^T \xi \right] \geq 0.
\]
Item 5 is a microscopic formula with large consequences: the variance of any portfolio \(v^T\xi\) (weights \(v\), assets \(\xi\)) is completely determined by the covariance matrix. Markowitz portfolio theory rests on exactly this: diversification works precisely to the extent that covariances are negative (or small); correlation is the language in which finance talks about risk. The flip side (the 2008 lesson from topic 2): correlation measures only linear dependence, and uncorrelatedness is nowhere near independence.
Let \(X \sim \mathscr {N}\left(0, 1\right)\) and \(Y = X^2\). Prove that \(X\) and \(Y\) are uncorrelated but dependent.
Prove item 4 of proposition Proposition 1: \(\left|\operatorname {Corr}\left[ \xi , \eta \right]\right| \leq 1\), with equality if and only if \(\eta\) is a linear function of \(\xi\) (a.s.).
(Limiting diversification.) Let \(\xi_1, \ldots , \xi_n\) be variables with unit variances and pairwise correlations all equal to the same number \(\rho\).
Prove that \(\rho \geq -\frac{1}{n-1}\).
Let \(\rho \geq 0\). Show that the variance of the average \(\frac{1}{n}\sum \xi_i\) tends to \(\rho\), not to zero, as \(n \to \infty\).
Inequalities
Theorem 2 (Jensen) Let \(g\) be convex and let \(\xi\), \(g(\xi )\) be integrable. Then \[
g\left(\mathbb {E}\left[\xi \right]\right) \leq \mathbb {E}\left[g(\xi )\right].
\] For concave functions the inequality is reversed. If \(g\) is strictly convex, equality is attained only for degenerate \(\xi = \operatorname {const}\) a.s.
The proof is one line via a supporting line: a convex \(g\) at the point \(m = \mathbb {E}\left[\xi \right]\) has a line \(\ell (x) = g(m) + k(x - m) \leq g(x)\); take the expectation of both sides.
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/030/02_jensen_fig.tex.
Corollary 1 (Lyapunovβs inequality) For \(0 < r < s\) \[
\left(\mathbb {E}\left[\left|\xi \right|^r\right]\right)^{1/r} \leq \left(\mathbb {E}\left[\left|\xi \right|^s\right]\right)^{1/s}.
\] In particular, finiteness of a higher moment implies finiteness of all lower moments.
Indeed, the function \(x \mapsto x^{s/r}\) is convex on \([0, \infty )\), and by Jensen \(\left(\mathbb {E}\left[\left|\xi \right|^r\right]\right)^{s/r} \leq \mathbb {E}\left[\left|\xi \right|^s\right]\). Together with the already-used Cauchy-Bunyakovsky-Schwarz inequality and the HΓΆlder and Minkowski inequalities from ETMT (\(L^p\)-spaces), this is our complete βestimation arsenalβ for the semester.
Let \(\xi > 0\), with \(\xi\) and \(1/\xi\) integrable. Prove that \(\mathbb {E}\left[\frac{1}{\xi }\right] \geq \frac{1}{\mathbb {E}\left[\xi \right]}\), and determine when equality is attained. Give an interpretation: average speed and average time.
Let \(\xi\) be a non-degenerate random variable with \(\mathbb {E}\left[\xi^2\right] < \infty\), \(\mathbb {E}\left[\xi \right] = a > 0\). Compare \(\mathbb {E}\left[\ln \xi \right]\) with \(\ln a\) (assuming \(\xi >0\) and \(\ln \xi\) integrable), and prove that \(\mathbb {E}\left[\xi^4\right] \geq \left(\mathbb {E}\left[\xi^2\right]\right)^2 \geq \left(\mathbb {E}\left[\xi \right]\right)^4\).