This text is still being edited: it may contain typos, inaccuracies, and locally unpolished spots. If you find something, please let the author know (contact details on the course page).
The second part of the course is about limit theorems for distributions: the central limit theorem asserts that the distribution of a normalized sum is close to normal. For this to become a theorem rather than a slogan, we need a precise notion of closeness of distributions β weak convergence. This notion also underlies all of asymptotic statistics (confidence intervals are statements about weak limits), numerical schemes (binomial trees in finance converge weakly to continuous models), and the theory of stochastic processes (Donskerβs invariance principle: a random walk converges weakly to the Wiener process). Our program: the correct definition (and why the βnaiveβ one does not work), Alexandrovβs theorem on equivalent descriptions, and the pair βtightness + Prokhorovβs theoremβ β the compactness tool used to prove existence-of-limit theorems.
Definition and first examples
Definition 1 A sequence of probability measures \(\mathbb {P}_n\) on \((\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right))\) converges weakly to a measure \(\mathbb {P}\) (written \(\mathbb {P}_n \Rightarrow \mathbb {P}\)) if \[
\int _{\mathbb {R}} f \, d\mathbb {P}_n \xrightarrow [n \to \infty ]{} \int _{\mathbb {R}} f \, d\mathbb {P} \qquad \text{for every bounded continuous } f.
\] Random variables \(\xi_n \xrightarrow {d} \xi\) if \(\mathbb {P}_{\xi_n} \Rightarrow \mathbb {P}_\xi\); equivalently (topic 4): \(F_{\xi_n}(x) \to F_\xi (x)\) at every point of continuity of \(F_\xi\) (convergence in the main sense).
Why the definition via the CDF excludes discontinuity points can be seen in the simplest example.
Let \(\xi_n = \frac{1}{n}\) (degenerate variables). Show that \(\xi_n \xrightarrow {d} 0\), even though \(F_{\xi_n}(0) \not\to F_0(0)\).
Common sense demands that β\(\frac1n\) converges to \(0\)β in any reasonable sense. Let us check Definition 1: for bounded continuous \(f\) we have \(\int f \, d\mathbb {P}_{\xi_n} = f\left(\frac1n\right) \to f(0)\) β weak convergence holds. But the CDF: \[
F_{\xi _n}(0) = \mathbb {P}\left(\xi _n \leq 0\right) = 0 \quad \not\to \quad F_0(0) = 1.
\] At the point \(x = 0\) β a discontinuity point of the limiting CDF β there is no pointwise convergence, and there should not be: the mass βcreeps upβ to zero from the right, and any threshold event \(\left\{ \xi \leq 0\right\}\) sees it as a jump. The definition via continuous \(f\) is insensitive to this boundary effect β that is precisely its point. (And if we had required convergence of the CDF everywhere, then \(\frac1n \to 0\) would fail to hold β such a βdefinitionβ is useless.)
Theorem 1 (Alexandrov, portmanteau) The following conditions are equivalent:
\(\mathbb {P}_n \Rightarrow \mathbb {P}\);
\(\limsup_n \mathbb {P}_n(C) \leq \mathbb {P}(C)\) for all closed \(C\);
\(\liminf_n \mathbb {P}_n(G) \geq \mathbb {P}(G)\) for all open \(G\);
\(\mathbb {P}_n(A) \to \mathbb {P}(A)\) for all Borel \(A\) with \(\mathbb {P}(\partial A) = 0\);
\(F_n(x) \to F(x)\) at every point of continuity of \(F\).
The asymmetry between items 2 and 3 is the same βmass creepingβ phenomenon: a closed set can suddenly gain mass on its boundary in the limit, while an open one can lose it. Item 4 is the practical one: probabilities of sets converge if the limiting measure does not load the boundary. Proof β Shiryaev [@shiryaev2016probability], Ch. 3, Β§1.
Let \(\xi_n = \frac1n\) (as in the example). For which of the sets \(A_1 = (0, \infty )\), \(A_2 = [0, \infty )\), \(A_3 = \left\{ 0\right\}\), \(A_4 = [\frac12, 1]\) does \(\mathbb {P}\left(\xi_n \in A_i\right) \to \mathbb {P}\left(0 \in A_i\right)\) hold? Compare with item 4 of Alexandrovβs theorem.
Let \(\xi_n\) be uniformly distributed on the grid \(\left\{ \frac1n, \frac2n, \ldots , \frac{n}{n}\right\}\). Prove that \(\xi_n \xrightarrow {d} \operatorname {U}[0,1]\).
Let \(\xi_n\) have density \(p_n(x) = \left(1 + \cos (2\pi n x)\right) \; \mathbb {1}_{[0,1]}(x)\). Prove that \(\xi_n \xrightarrow {d} \operatorname {U}[0,1]\), even though the densities \(p_n\) do not converge to the density \(\; \mathbb {1}_{[0,1]}\) at any point (except points where \(\cos = 0\)).
Tightness and Prokhorovβs theorem
A weak limit may fail to exist for a trivial reason: the mass escapes to infinity. Example: \(\xi_n \sim \operatorname {U}[n, n+1]\) β each CDF is honest, but \(F_{\xi_n}(x) \to 0\) for every \(x\): the βlimiting functionβ is identically zero, not a CDF. The condition that forbids the escape of mass:
Definition 2 A family of measures \(\left\{ \mathbb {P}_\alpha \right\}\) is tight if for every \(\varepsilon > 0\) there is a compact \(K = [-C, C]\) with \(\sup_\alpha \mathbb {P}_\alpha (\mathbb {R} \setminus K) \leq \varepsilon\).
Theorem 2 (Prokhorov) A family of probability measures on \(\mathbb {R}\) (and more generally on a Polish space) is relatively compact in the sense of weak convergence (every sequence has a subsequence converging weakly to a probability measure) if and only if it is tight.
The scheme of β\(\Leftarrow\)β on the line: by Hellyβs selection theorem, every sequence of CDFs has a pointwise convergent subsequence (diagonal method over rational points); tightness guarantees that the limiting function collects the full unit mass, i.e. is itself a CDF. Prokhorovβs theorem is the probabilistic analogue of compactness: it turns the proof of convergence into two steps β (i) tightness (no escape of mass) + (ii) uniqueness of the limit point (usually via characteristic functions, topic 8). By this scheme we will prove the CLT, and the stochastic processes course will prove the invariance principle. See Shiryaev, Ch. 3, Β§2 [@shiryaev2016probability].
The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/070/01_weak_convergence_panels_fig.tex.
Prove: if \(\sup_n \mathbb {E}\left[\left|\xi_n\right|\right] = C < \infty\), then the family of distributions \(\left\{ \mathbb {P}_{\xi_n}\right\}\) is tight. Give an example of a tight family for which all expectations are infinite.
Let \(\xi_n \xrightarrow {d} \xi\) and \(\eta_n \xrightarrow {d} \eta\). Must \(\xi_n + \eta_n \xrightarrow {d} \xi + \eta\) hold? What if additionally \(\xi_n \; \perp \! \! \! \perp \; \eta_n\) for each \(n\) and \(\xi \; \perp \! \! \! \perp \; \eta\)?