Convergence of Random Variables

Published

8 September 2026

Draft

This text is still being edited: it may contain typos, inaccuracies, and locally unpolished spots. If you find something, please let the author know (contact details on the course page).

The entire second half of the course consists of limit theorems, and before proving them we need to discuss the possible modes of convergence.

1 Four modes of convergence

Definition 1 Let $\xi , \xi_1, \xi_2, \ldots$ be random variables (for the first three items — on the same probability space).

$\xi_n \to \xi$ almost surely (a.s.) if $\mathbb {P}\left(\omega \colon \xi_n(\omega ) \to \xi (\omega )\right) = 1$.
$\xi_n \xrightarrow {\mathbb {P}} \xi$ in probability if $\mathbb {P}\left(\left|\xi_n - \xi \right| \geq \varepsilon \right) \to 0$ for every $\varepsilon > 0$.
$\xi_n \xrightarrow {L^p} \xi$ in $L^p$ ($p \geq 1$) if $\mathbb {E}\left[\left|\xi_n - \xi \right|^p\right] \to 0$.
$\xi_n \xrightarrow {d} \xi$ in distribution if $F_{\xi_n}(x) \to F_\xi (x)$ at every point of continuity of $F_\xi$.

The first two are the familiar convergence almost everywhere and convergence in measure from ETMT, the third is convergence in the Lebesgue space $L^p$. The fourth is new and of a fundamentally different sort: it speaks only about distributions, the variables can live on different spaces (we will deal with it in detail in topic 7).

Theorem 1 (Diagram of implications)

$\xi_n \to \xi$ a.s. $\Longrightarrow$ $\xi_n \xrightarrow {\mathbb {P}} \xi$;
$\xi_n \xrightarrow {L^p} \xi$ $\Longrightarrow$ $\xi_n \xrightarrow {\mathbb {P}} \xi$ (Markov’s inequality); $L^s \Rightarrow L^r$ for $r < s$ (Lyapunov);
$\xi_n \xrightarrow {\mathbb {P}} \xi$ $\Longrightarrow$ $\xi_n \xrightarrow {d} \xi$;
from $\xi_n \xrightarrow {\mathbb {P}} \xi$ it follows that there exists a subsequence $\xi_{n_k} \to \xi$ a.s.;
if $\xi_n \xrightarrow {d} c$, where $c$ is a constant, then $\xi_n \xrightarrow {\mathbb {P}} c$. No other arrows hold in general.

Interactive illustration placeholder

The interactive illustration has not been created yet. See the source: /home/nedosh/Documents/online_math_notes/math-notes_FPMI_2/courses/nedosh_courses/courses/04-probability/02_PT/contents/specials/040/01_convergence_diagram_fig.tex.

You have essentially proved all the positive statements except item 3 in ETMT (convergence in measure and $L^p$); item 3 is treated in topic 7. The negative statements — counterexamples — must be known by name; the two main ones are as follows.

Example 1 - The “typewriter”: in probability but not a.s.

On $\Omega = [0,1]$ with Lebesgue measure, construct a sequence $\xi_n \xrightarrow {\mathbb {P}} 0$ for which $\xi_n(\omega )$ does not converge for any $\omega$.

Solution

We sweep across $[0,1]$ with indicators of ever shorter intervals, like the carriage of a typewriter: the first “row” is $\; \mathbb {1}_{[0,1]}$; the second is $\; \mathbb {1}_{[0,\frac12]}, \; \mathbb {1}_{[\frac12,1]}$; the third is $\; \mathbb {1}_{[0,\frac14]}, \ldots , \; \mathbb {1}_{[\frac34,1]}$; and so on. For the $n$-th term the length of the interval $\to 0$, so $\mathbb {P}\left(\left|\xi_n\right| \geq \varepsilon \right) \to 0$ — convergence in probability (and even in any $L^p$: $\mathbb {E}\left[\xi_n^p\right] =$ length of the interval $\to 0$). But every point $\omega$ falls into intervals of every “row”, i.e. $\xi_n(\omega ) = 1$ infinitely often and $\xi_n(\omega ) = 0$ infinitely often: there is no limit at any point. Convergence in probability controls each moment in time separately but says nothing about the fate of an individual trajectory.

Interactive illustration placeholder

Example 2 - The “growing spike”: a.s. but not in L^1

On the same space, construct $\xi_n \to 0$ a.s. for which $\mathbb {E}\left[\xi_n\right] \not\to 0$.

Solution

Take $\xi_n = n \cdot \; \mathbb {1}_{[0, \frac1n]}$. For every $\omega > 0$, from some index on $\xi_n(\omega ) = 0$ — convergence a.s. (and in probability). But \[ \mathbb {E}\left[\xi _n\right] = n \cdot \frac{1}{n} = 1 \not\to 0. \] A small probability of a large value can sustain the expectation: “rare but sharp”. This effect is no toy example: sample averages of heavy-tailed returns, insurance losses with rare catastrophes — wherever moments are governed by tails, convergence of values does not imply convergence of averages. The remedy is uniform integrability (below).

Also useful is a criterion for a.s. convergence that reduces it to “probabilistic” language:

Proposition 1 (Criterion for a.s. convergence) $\xi_n \to \xi$ a.s. if and only if for every $\varepsilon > 0$ \[ \mathbb {P}\left(\sup _{k \geq n} \left|\xi _k - \xi \right| \geq \varepsilon \right) \xrightarrow [n \to \infty ]{} 0. \]

Compare with the definition of convergence in probability: there, the probability is of $\left|\xi_n - \xi \right|$; here, of the supremum over the entire “tail” of the sequence. The difference between “bad events at time $n$ are rare” and “after time $n$ there will be no more bad events” is exactly the difference between $\mathbb {P}$- and a.s.-convergence.

Problem 1

Let $\sum_{n=1}^{\infty }\mathbb {E}\left[\left|X_n - X\right|^p\right] < \infty$ for some $p > 0$ (for $L^p$ convergence assume $p \geq 1$). Prove that \[ X_n \xrightarrow [n \to \infty ]{L^p, \text{compl.}} X \]

Problem 2

Prove that \[ X_n \xrightarrow [n \to \infty ]{L^{\infty }} X \iff \text{for every $\varepsilon > 0$ } \mathbb {P}\left(\left|X_n - X\right| > \varepsilon \right) = 0 \text{ for all sufficiently large $n$} \]

2 Linear properties

Lemma 1 Let

3 Inheritance of convergence

Theorem 2 (On inheritance) Let $g\colon \mathbb {R} \to \mathbb {R}$ be continuous. If $\xi_n \to \xi$ a.s. / in probability / in distribution, then $g(\xi_n) \to g(\xi )$ in the same sense.

Theorem 3 (On inheritance) Let $g\colon \mathbb {R} \to \mathbb {R}$ be uniformly continuous. Then complete convergence is inherited.

Theorem 4 (On inheritance) Let $g\colon \mathbb {R} \to \mathbb {R}$ be Lipschitz. Then convergence in mean is inherited.

Theorem 5 (Slutsky) Let $\xi_n \xrightarrow {d} \xi$ and $\eta_n \xrightarrow {\mathbb {P}} c$, where $c$ is a constant. Then \[ \xi _n + \eta _n \xrightarrow {d} \xi + c, \qquad \xi _n \eta _n \xrightarrow {d} c\xi , \qquad \xi _n / \eta _n \xrightarrow {d} \xi / c \; (c \neq 0). \]

A warning: two convergences in distribution cannot be added ($\xi_n \xrightarrow {d} \xi$, $\eta_n \xrightarrow {d} \eta$ $\not\Rightarrow$ $\xi_n + \eta_n \xrightarrow {d} \xi + \eta$: take $\xi_n = \eta$, $\eta_n = -\eta$ for a symmetric $\eta$). Slutsky’s theorem is the exception where one of the limits is degenerate; in statistics this is the main tool: replacing an unknown variance with a consistent estimator inside the CLT (studentization) is exactly Slutsky’s theorem. Course 3 uses it at every step (the delta method).

Problem 3

Let $\xi_{1}, \xi_2, \ldots \sim \operatorname {U}[0,1]$ be independent, $M_{n}=\max \left(\xi_{1}, \ldots , \xi_{n}\right)$. Investigate the convergence of $M_n$: (a) in distribution; (b) in probability; (c) almost surely; (d) in $L_{2}$. Then find the limit in distribution of the variable $n(1 - M_n)$.

Problem 4

Let $\xi_n$ and $\xi$ take values in $\mathbb {Z}$. Prove that $\xi_{n} \xrightarrow {d} \xi$ if and only if $\mathbb {P}\left(\xi_n = m\right) \to \mathbb {P}\left(\xi = m\right)$ for every $m \in \mathbb {Z}$.

Problem 5

Let $\xi_n \xrightarrow {\mathbb {P}} \xi$ and $\left|\xi_n\right| \leq \eta$ a.s. for all $n$, where $\mathbb {E}\left[\eta^p\right] < \infty$. Prove that $\xi_n \xrightarrow {L^p} \xi$.

Problem 6*

3.1 A metric for convergence in probability

For random variables $\xi , \eta$ set \[ \rho (\xi , \eta )= \mathbb {E}\left[\frac{\left|\xi - \eta \right|}{1 + \left|\xi - \eta \right|}\right]. \] Prove that $\xi_{n} \xrightarrow {\mathbb {P}} \xi$ if and only if $\rho \left(\xi_{n}, \xi \right) \to 0$.

Problem 7

Let $p \in \mathbb {N}$. Is it true that \[ X_n \xrightarrow [n \to \infty ]{L^p} X \iff X_n^p \xrightarrow [n \to \infty ]{L} X^p \] Consider 4 cases: $p$ even or odd; $X_n, X$ nonnegative or of arbitrary sign.

4 Uniform integrability

The “growing spike” example showed: convergence of values does not imply convergence of averages — mass can escape into “rare but large” values. That is, we have a sequence of random variables from $L^1$, converging a.s., but not converging in $L^1$. Moreover, this sequence was bounded in $L^1$ (i.e. the $L^1$ norms were bounded by a common constant)! Under a.s. convergence, boundedness in $L^1$ is not sufficient to guarantee convergence in $L^1$. It turns out that any condition “stronger” than boundedness in $L^1$ is sufficient.

Theorem 6 (Uniform integrability) Let $\left\{ \xi_\alpha \right\}_{\alpha \in A}$ be some family of random variables. The following conditions are equivalent:

there exists a non-decreasing convex function $G: \mathbb {R}_+ \to \mathbb {R}_+$, growing faster than the identity (i.e. $G(t)/t \to +\infty$ as $t \to +\infty$), such that the $G$-moments of the family are bounded: there exists $M>0$ such that

\[ \mathbb {E}\left[G\left(\left|\xi _{\alpha }\right|\right)\right] \leq M \]

for every $\alpha \in A$

the distributions do not escape to infinity too strongly:

\[ \sup _{\alpha \in A} \mathbb {E}\left[\left|\xi _\alpha \right| \cdot \; \mathbb {1}_{\left|\xi _\alpha \right| > C}\right] \xrightarrow [C \to \infty ]{} 0. \]

The family $\left\{ \xi_{\alpha }\right\}$ is bounded in $L^1$ (i.e. there exists $M>0$ such that $\mathbb {E}\left[\left|\xi_{\alpha }\right|\right] \leq M$ for all $\alpha \in A$), and for every $\varepsilon > 0$ there exists $\delta > 0$ such that for every measurable $A$ with $\mathbb {P}\left(A\right) \leq \delta$

\[ \mathbb {E}\left[\left|\xi \right| \cdot \; \mathbb {1}_{A}\right] \leq \varepsilon \] A family satisfying any of these equivalent conditions is called uniformly integrable (UI).

Theorem 7 (Vitali) Let $1 \leq p < +\infty$ be an arbitrary (finite!) number. The following conditions are equivalent:

$\xi_n \xrightarrow {L^p} \xi$;
$\xi_n \xrightarrow {\mathbb {P}} \xi$ and the family $\left\{ \left|\xi_n\right|^p\right\}$ is uniformly integrable;
$\xi_n \xrightarrow {\mathbb {P}} \xi$ and $\mathbb {E}\left[\left|\xi_n\right|^p\right] \to \mathbb {E}\left[\left|\xi \right|^p\right]$.

This is the exact answer to the question “when can we pass to the limit under the expectation”, extending dominated convergence: a dominating function is a special case of UI. In the stochastic processes course, Vitali’s theorem is an indispensable tool (optimal stopping, martingales: uniform integrability of a martingale is the condition for “closability”). In Shiryaev, UI and passages to the limit — Ch. 2, §6 [@shiryaev2016probability].

Problem 8

Check that the “growing spike” $\xi_n = n\; \mathbb {1}_{[0, 1/n]}$ is not a UI family. (b) Let $\xi_n \xrightarrow {\mathbb {P}} \xi$ and $\sup_n \mathbb {E}\left[\xi_n^2\right] < \infty$. Prove that $\mathbb {E}\left[\xi_n\right] \to \mathbb {E}\left[\xi \right]$. Is it true that $\mathbb {E}\left[\xi_n^2\right] \to \mathbb {E}\left[\xi^2\right]$?

Problem 9

Prove that if $\xi_{n} \xrightarrow {d} \xi$ and $F_{\xi }$ is continuous, then $F_{\xi_{n}} \to F_{\xi }$ uniformly on $\mathbb {R}$.