Probability distributions. Random variables. Expectation

Published

18 October 2026

Draft

This lecture note is still being edited: the text may contain typos, inaccuracies, and locally unpolished spots. If you find something, please let the author know (contact information on the course page).

1 Random variables

Many experiments have outcomes that are conveniently expressed as numbers. For example, the result of rolling a die is the number of pips on the face that comes up, from the set $\{ 1, \ldots , 6\}$. When building a probabilistic model, this numerical outcome is put in correspondence with an elementary outcome.

A function that assigns a number to each elementary outcome, and for which one can define the probability of it landing in various (Borel) subsets of $\mathbb {R}$, is called a random variable. Introducing this notion is an important tool in probability theory, since it allows us to:

Substantially simplify the computation of probabilities of various events,
Formulate and prove the key limit theorems.

Before giving a rigorous definition of a random variable, it is useful to recall the general notion of a measurable function.

Definition 1 Let $\Omega$ be some set, and let $\mathcal{F}$ be a $\sigma$-algebra on it. Let, moreover, $\mathcal{E}$ be a $\sigma$-algebra on some set $E$.

A set together with some sigma-algebra on it is called a measurable space. $\left(\Omega , \mathcal{F}\right)$ and $\left(E, \mathcal{E}\right)$ are two measurable spaces.
A map $X: \Omega \rightarrow E$ from one measurable space to another is called measurable (more precisely, $\mathcal{F} \mid \mathcal{E}$-measurable), if for every set $B \in \mathcal{E}$ its preimage lies in $\mathcal{F}$:

\[ \underbrace{X^{-1}(B) = \{ \omega \in \Omega : X(\omega ) \in B\} = \left\{ X \in B\right\} }_{\text{various notations for the preimage}} \in \mathcal{F} \]

The meaning: if a probability (probability measure) $\mathbb {P}$ is given on $\left(\Omega , \mathcal{F}\right)$, then the measurability of the function $X:\Omega \to E$ allows us to measure the probability that $X$ lands in an arbitrary set from the sigma-algebra on the codomain $E$.
In probability theory, a measurable map is called a random element. That is, if $(\Omega , \mathcal{F}, \mathbb {P})$ is a probability space, and $X: \Omega \to E$ is a measurable map, then $X$ is called a random element (with values in $(E, \mathcal{E})$).
If $(E, \mathcal{E}) = \left(\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right)\right)$, then the random element $X$ is called a random variable. That is, a random variable is a measurable function (a map into $\left(\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right)\right)$).

Let us repeat once more that measurability allows us to compute the probability that, say, a random variable does not exceed $7$: $\mathbb {P}\left(X \leq 7\right)$. Indeed, $\left\{ X \leq 7\right\} = \left\{ \omega : \; X(\omega ) \leq 7\right\} = X^{-1}\left((-\infty , 7]\right)$, the set $(-\infty , 7]$ is Borel, hence $\left\{ X \leq 7\right\} \in \mathcal{F}$, and this set may be inserted into the probability.
If a map $\xi : \Omega \rightarrow \mathbb {R}^{n}$ is $\mathcal{F} \mid \mathscr {B}\left(\mathbb {R}^{n}\right)$-measurable, then it is called a random vector.
A map $f: \mathbb {R}^{n} \rightarrow \mathbb {R}^{k}$ is called Borel (or Borel measurable), if it is $\mathcal{B}\left(\mathbb {R}^{n}\right) \mid \mathcal{B}\left(\mathbb {R}^{k}\right)$-measurable.

measurability and the Borel sigma-algebrameasurability The definition of a random variable relies on the Borel sigma-algebra $\mathscr {B}\left(\mathbb {R}\right)$, which we will rigorously construct in the chapter on systems of sets and the Borel sigma-algebra, while the systematic study of measurable functions awaits us in the chapter on measurable functions. For now, the following understanding suffices: $\mathscr {B}\left(\mathbb {R}\right)$ contains all sets encountered in practice (all intervals and everything obtainable from them via countable unions, intersections, and complements), while measurability is a technical consistency condition linking $X$ with the sigma-algebras, one that holds automatically for essentially every function encountered in practice. In the discrete case ($\mathcal{F} = 2^{\Omega }$), literally every function is measurable.

It should be noted that the measurability condition is largely of theoretical character. In real practice it is very hard to come across a non-measurable function; the overwhelming majority of functions possess measurability, i.e., are random elements (variables or vectors). All the standard functions from $\mathbb {R}$ to $\mathbb {R}$ that you know are Borel measurable. And in the discrete case, for instance, absolutely every function $\Omega \mapsto \mathbb {R}$ is measurable, since $\mathcal{F} = 2^{\Omega }$, i.e., any preimage lands in $\mathcal{F}$.

To simplify the notation of probabilities of events related to random variables, the following notation is introduced:

\[ \begin{align} \mathbb {P}\left(\left\{ \omega : \xi (\omega ) \in B\right\} \right) & =: \mathbb {P}\left(\xi \in B\right) \\ \mathbb {P}\left(\left\{ \omega : \xi (\omega ) < x\right\} \right) & =: \mathbb {P}\left(\xi < x\right) \\ \mathbb {P}\left(\left\{ \omega : \xi (\omega ) = x\right\} \right) & =: \mathbb {P}\left(\xi =x \right) \\ \mathcal{F}_{\xi } & := \left\{ \xi ^{-1}(B) \mid B \in \mathscr {B}\left(\mathbb {R}^{n}\right)\right\} \\ \end{align} \]

Let us give the simplest examples of random variables.

The model of a coin toss with probability $p$ of heads: $\Omega = \left\{ \omega_{\text{H}}, \omega_{\text{T}}\right\}$, $\mathcal{F}=2^{\Omega }$, $\mathbb {P}\left(\left\{ \omega_{\text{H}}\right\} \right) = p$. Let, moreover,

\[ \xi : \Omega \rightarrow \left\{ 0,1\right\} , \qquad \xi \left(\omega _{\text{H}}\right)=1, \quad \xi \left(\omega _{\text{T}}\right)=0 \]

Then we have: $\mathbb {P}\left(\xi = 1\right) = p$, $\mathbb {P}\left(\xi = 0\right) = 1-p$.
Let $\Omega$ be an arbitrary set, let $\mathcal{F}$ be a $\sigma$-algebra on it, and let $A \in \mathcal{F}$. The random variable $I_{A}: \Omega \rightarrow \{ 0,1\}$, where $I_{A}(\omega )=1$ if and only if $\omega \in A$, is called the indicator of the event $A$.

Note that the measurability property in the definition of random variables (more broadly, random vectors and elements) is needed in order to compute probabilities of properties tied to the values of the random variable. In other words, we are interested in the probabilities $\mathbb {P}\left(\xi \in B\right)$, $B \in \mathcal{E}$. It is easy to prove that the function $\mathbf{P}_{\xi }: \mathcal{B}\left(\mathbb {R}^{n}\right) \rightarrow [0,1]$, whose values are given by

\[ \mathbf{P}_{\xi }\left(B\right) := \mathbb {P}\left(\xi \in B\right) = \mathbb {P}\left(\left\{ \omega \; : \; \xi (\omega ) \in B\right\} \right), \]

is a probability measure. In this connection the following definitions are introduced.

Figure 1: The random variable $X: \Omega \to \mathbb {R}$ “pushes forward” the probability measure $\mathbb {P}$ from the domain $(\Omega , \mathcal{F})$ to the codomain $(\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right))$. On the codomain $(\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right))$ a probability measure $\mathbf{P}_X$ appears (in English it is called precisely that, ). It is given by the rule $\mathbf{P}_{X}\left(B\right) := \mathbb {P}\left(X^{-1}(B)\right) = \mathbb {P}\left(X \in B\right)$, $B \in \mathscr {B}\left(\mathbb {R}\right)$ (i.e., $\mathbf{P}_{X} = \mathbb {P} \circ X^{-1}$). This measure $\mathbf{P}_X$ is called the $X$.

Definition 2 The probability measure $\mathbf{P}_{\xi }$ on $(\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right))$ is called the distribution of the random variable $\xi$.

2 Probability distribution

The term distribution (or probability distribution), without reference to a random variable, is essentially a synonym for a probability measure (probability). The only difference is that a probability may be defined on an arbitrary measurable space $(\Omega , \mathcal{F})$, whereas a distribution is, as a rule, defined on $(\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right))$ (more broadly, on $(\mathbb {R}^{n}, \mathscr {B}\left(\mathbb {R}^{n}\right))$).

The most common types of distributions are the discrete and the absolutely continuous probability distributions. We will consider the discrete ones.

but where are the continuous distributions?densities In this chapter we deliberately restrict ourselves to discrete distributions. The familiar “continuous” distributions — uniform on an interval, normal $\mathscr {N}\left(a, \sigma^2\right)$, exponential — are specified not by masses of individual points but by a density. To honestly define what a density is, and how to compute probabilities and expectations from it, one needs the Lebesgue integral (the chapters on constructing the Lebesgue integral–on its properties) and the Radon–Nikodym theorem (the chapter on signed measures). This is yet another — and perhaps the main applied — reason to build measure theory.

The support of a distribution is the smallest closed set of measure $1$. Notation: $\operatorname {supp}\mathbf{P}$. We have

\[ \operatorname {supp}\mathbf{P} = \bigcap _{\substack { B \subset \mathbb {R} \\ B \text{ -- closed} \\ \mathbf{P}\left(B\right) = 1}} B \]

For example, the support of Lebesgue measure on the segment $[0,1]$ is the segment $[0,1]$. The support of the Dirac measure at the point $c$ is $\left\{ c\right\}$ (see below).

A distribution $\mathbf{P}$ is called discrete if its support is an at most countable set. Such a distribution is conveniently specified simply by the masses of the individual points of the support: the value $\mathbf{P}\left(\left\{ x\right\} \right)$, $x \in \operatorname {supp}\mathbf{P}$, is called the mass function (or probability function; the commonly used abbreviation PMF comes from the English probability mass function).

Let us give examples of discrete distributions.

The Dirac measure at the point $c \in \mathbb {R}$: $\delta_c$. We have $\operatorname {supp}\delta_{c} = \left\{ c\right\}$, $\mathbf{P}\left(\left\{ c\right\} \right) = 1$.
The discrete uniform distribution $\operatorname {Unif}\left\{ 1,\ldots ,N\right\}$ on the set $\{ 1, \ldots , N\}$ is defined as follows:

\[ \operatorname {supp}\operatorname {Unif}\left\{ 1,\ldots ,N\right\} = \left\{ 1, \ldots , N\right\} , \qquad \mathbf{P}\left(\left\{ k\right\} \right) = \frac{1}{N}, \quad k \in \left\{ 1, \ldots , N\right\} \]

In other words,

\[ \operatorname {Unif}\left\{ 1,\ldots ,N\right\} = \frac{1}{N}\delta _1 + \frac{1}{N}\delta _2 + \ldots + \frac{1}{N}\delta _N \]

It is easy to check that this is indeed a distribution: all masses are positive and sum to $1$.
The Bernoulli distribution with parameter $p \in (0,1)$: $\operatorname {Ber}(p)$.

\[ \operatorname {supp}\operatorname {Ber}(p) = \left\{ 0,1\right\} , \qquad \mathbf{P}\left(\left\{ 0\right\} \right) = 1-p, \quad \mathbf{P}\left(\left\{ 1\right\} \right) = p \]

In other words,

\[ \operatorname {Ber}(p) = (1-p)\cdot \delta _0 + p\cdot \delta _{1} \]

Physical model: the toss of a biased coin, where heads is assigned $1$ and tails $0$.
The binomial distribution $\operatorname {Bin}(n,p)$ with parameters $(n, p)$, $n \in \mathbb {N}, p \in (0,1)$:

\[ \mathbf{P}\left(\left\{ k\right\} \right) = C_{n}^{k}p^{k}(1-p)^{n-k}, \qquad k \in \left\{ 0, \ldots , n\right\} = \operatorname {supp}\operatorname {Bin}(n,p) \]

Physical model: the number of successes in $n$ independent Bernoulli trials.
The Poisson distribution $\operatorname {Pois}(\lambda )$ with parameter $\lambda >0$:

\[ \mathbf{P}\left(\left\{ k\right\} \right) = \frac{\lambda ^{k}}{k!} e^{-\lambda }, \qquad k \in \mathbb {Z}_{+} = \left\{ 0,1,2, \ldots \right\} = \operatorname {supp}\operatorname {Pois}(\lambda ) \]

Physical model: the number of successes in a sequence of independent Bernoulli trials with parameter $\lambda / n$, where $n \gg \lambda$.
The geometric distribution $\operatorname {Geom}(p)$ with parameter $p \in (0,1)$:

\[ \mathbf{P}\left(\left\{ k\right\} \right) = (1-p)^{k-1}p, \qquad k \in \mathbb {N} = \operatorname {supp}\operatorname {Geom}(p) \]

Physical model: the number of coin tosses up to and including the first heads.

Random variables $\xi , \eta$ are called independent if the events $\left\{ \xi \in B_1\right\}$, $\left\{ \eta \in B_2\right\}$ are independent for arbitrary Borel sets $B_1, B_2 \in \mathscr {B}\left(\mathbb {R}\right)$. That is,

\[ \mathbb {P}\left(\xi \in B_1, \eta \in B_2\right) = \mathbb {P}\left(\xi \in B_1\right) \cdot \mathbb {P}\left(\eta \in B_2\right), \qquad \forall \; B_1, B_2 \in \mathscr {B}\left(\mathbb {R}\right) \]

Example 1

For two independent random variables $\xi \sim \operatorname {Pois}(\lambda )$ and $\eta \sim \operatorname {Pois}(\theta )$, find the distribution of $\xi +\eta$.

The notation $\xi \sim \operatorname {Pois}(\lambda )$ means that $\xi$ has the Poisson distribution: $\mathbf{P}_{\xi } = \operatorname {Pois}(\lambda )$.

Solution

The support of $\xi$, $\eta$ (i.e., the support of their distributions) is the set $\mathbb {Z}_{+}$. Hence their sum, too, takes values in $\mathbb {Z}_{+}$ with probability $1$: $\operatorname {supp}(\xi + \eta ) \subseteq \mathbb {Z}_+$¹. To solve the problem, it suffices to find the mass function of $\xi + \eta$. For an arbitrary $k \in \mathbb {Z}_+$ we have \[ \mathbf{P}_{\xi + \eta }\left(\left\{ k\right\} \right) = \mathbb {P}\left(\xi + \eta = k\right) = \sum _{i=0}^{k} \mathbb {P}\left(\xi = i, \; \eta = k-i\right) \] Using the independence of $\xi$, $\eta$, we get \[ \begin{align} \mathbb {P}\left(\xi = i, \; \eta = k-i\right) & = \mathbb {P}\left(\xi \in \left\{ i\right\} , \; \eta \in \left\{ k-i\right\} \right) = \mathbb {P}\left(\xi = i\right) \cdot \mathbb {P}\left(\eta = k-i\right) = e^{-(\lambda + \theta )} \cdot \frac{\lambda ^{i}}{i!} \cdot \frac{\theta ^{k-i}}{(k-i)!} = \\ & = e^{-(\lambda + \theta )} \cdot \frac{1}{k!} \cdot C_{k}^{i} \cdot \lambda ^{i} \cdot \theta ^{k-i} \end{align} \] Next, using the binomial theorem, we obtain \[ \mathbf{P}_{\xi + \eta }\left(\left\{ k\right\} \right) = e^{-(\lambda + \theta )} \cdot \frac{1}{k!} \cdot \sum _{i=0}^{k} C_{k}^{i} \cdot \lambda ^{i} \cdot \theta ^{k-i} = e^{-(\lambda + \theta )} \cdot \frac{1}{k!} \cdot (\lambda + \theta )^{k} \]

Answer

$\xi + \eta \sim \operatorname {Pois}(\lambda + \theta )$

In the example above, as one can observe, although the discussion was framed in terms of a random variable, the actual object of study was not the map $\Omega \to \mathbb {R}$ itself, but its probability distribution. This happens often in probability theory: a random variable and its distribution are used interchangeably. It is important, nonetheless, to distinguish between them. The distribution of a random variable is only one of its characteristics. Two completely different random variables (as functions) may have the same distribution.

Problem 1

For two independent random variables $\xi \sim \operatorname {Bin}(n, p)$ and $\eta \sim \operatorname {Bin}(m, p)$, find the distribution of $\xi +\eta$.

3 From a distribution to a random variable

As we have established, every random variable has a distribution, i.e., it generates a probability measure on $(\mathbb {R}, \mathcal{B}(\mathbb {R}))$. In real problems, however, one often does not mention the underlying probability space $(\Omega , \mathcal{F}, \mathbb {P})$ at all, and simply speaks of some random variable with a given distribution $\mathbf{P}$.

A question arises: does such a random variable exist at all, in principle? Can we construct $(\Omega , \mathcal{F}, \mathbb {P})$ and a measurable map $X \colon \Omega \to \mathbb {R}$ such that $\mathbf{P} = \mathbf{P}_{X}$? The answer to this question is yes. Here is the standard construction:

\[ (\Omega , \mathcal{F}, \mathbb {P}) := (\mathbb {R}, \mathcal{B}(\mathbb {R}), \mathbf{P}), \qquad X(\omega ) := \omega \qquad \text{(i.e., $X = \operatorname {Id}$)}. \]

It can be simplified somewhat by taking, instead of $\mathbb {R}$, only the support of the distribution $\mathbf{P}$:

\[ \Omega := \operatorname {supp}\mathbf{P}, \qquad \mathcal{F} := \left\{ B \cap \operatorname {supp}\mathbf{P}, \; B \in \mathscr {B}\left(\mathbb {R}\right)\right\} , \qquad \mathbb {P} := \mathbf{P}|_{\operatorname {supp}\mathbf{P}} \]

Example 2

Prove the existence of a random variable with the Poisson distribution with parameter $\lambda > 0$.
Prove the existence of two independent Poisson random variables with parameters $\lambda_1, \lambda_2 \in (0,+\infty )$.

Solution

(1)

Let $\Omega = \mathbb {Z}_+$, $\mathcal{F} = 2^{\mathbb {Z}_+}$, $\mathbb {P} = \operatorname {Pois}(\lambda )$, $X = \operatorname {Id}$.

(2)

Let $\Omega = \mathbb {Z}_+ \times \mathbb {Z}_+$, $\mathcal{F} = 2^{\Omega }$. Next, for an arbitrary $\omega = (k_1, k_2) \in \mathbb {Z}_+ \times \mathbb {Z}_+$ let

\[ \mathbb {P}\left(\left\{ \omega \right\} \right) := \frac{\lambda _1^{k_1}}{k_1!}e^{-\lambda _1} \cdot \frac{\lambda _2^{k_2}}{k_2!}e^{-\lambda _2} \]

Let $X := \operatorname {Id}$. We have $X: \Omega \to \mathbb {Z}_+ \times \mathbb {Z}_+$, i.e., $X$ has two coordinate functions: $X = (X_1, X_2)$. It is precisely $X_1, X_2$ that turn out to be independent Poisson random variables with parameters $\lambda_1$, $\lambda_2$ (check this yourself).

4 Cumulative distribution function

Definition 3 Let $\mathbf{P}$ be a distribution on $(\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right))$. The function

\[ F_{\mathbf{P}}(x) := \mathbf{P}\left((-\infty , x]\right), \quad x \in \mathbb {R} \]

is called the cumulative distribution function (cdf) of the probability measure $\mathbf{P}$.

If $\xi$ is a random variable, then the cumulative distribution function of $\xi$ is the function

\[ F_{\xi }(x) := F_{\mathbf{P}_{\xi }}(x) = \mathbb {P}\left(\xi \in (-\infty , x]\right) = \mathbb {P}\left(\xi \leq x\right), \quad x \in \mathbb {R} \]

Let us note the properties of the cumulative distribution function.

$F(x)$ is a non-decreasing function;
$\lim_{x \rightarrow -\infty } F(x)=0$, $\lim_{x \rightarrow +\infty } F(x)=1$;
$F(x)$ is right-continuous and has left-hand limits at every point $x \in \mathbb {R}$.

Before going further, let us look at the graphs of the cumulative distribution functions for the discrete distributions introduced in the previous section — all three properties listed above are clearly visible on them (in particular, the jumps at the points of the support, and the fact that the filled dot is always the right-hand end of a jump).

Cumulative distribution functions of the six discrete distributions from the previous section. An open circle ○ marks a value that is not attained (the left-hand limit); a filled circle ● marks a value that is attained at that point (by right-continuity). For the Poisson and geometric distributions only the first few jumps are shown — further on the cdf keeps approaching 1 in small steps, but never actually equals 1 at any finite point (the support is infinite).

It turns out that to specify a probability distribution it suffices to specify the probability measure only on the sets $(-\infty , x]$ for all real $x$.

Theorem 1 Let $F=F(x)$ be a function on the real line $\mathbb {R}$ satisfying properties 1–3. Then on $(\mathbb {R}, \mathscr {B}\left(\mathbb {R}\right))$ there exists a probability distribution $\mathbf{P}$, unique, such that $F$ is the cumulative distribution function corresponding to this distribution.

This theorem justifies introducing the following notion: any function $F$ satisfying conditions 1)–3) is called a cumulative distribution function on the real line $\mathbb {R}$.

a cdf determines the distribution uniquelycdf-defines-distrib The theorem above is pure measure theory, and its debt will be repaid in full. The existence of a measure corresponding to a function $F$ is Carathéodory’s extension theorem (the chapter on outer measure and Carathéodory’s theorem), applied to the set function $(a,b] \mapsto F(b)-F(a)$; uniqueness follows from the uniqueness-of-extension theorem for $\pi$-systems (same chapter). The resulting construction is called the Lebesgue–Stieltjes measure (the chapter on Lebesgue measure).

Example 3

Let $F(x)$ be the cumulative distribution function on the real line corresponding to the probability distribution $\mathbf{P}$. Prove the equality $\mathbf{P}\left((a, b]\right) = F(b) - F(a)$ for arbitrary $-\infty <a<b<\infty$.

Solution

By the definition of a probability measure, \[ \mathbf{P}\left((-\infty , a]\right) + \mathbf{P}\left((a, b]\right) = \mathbf{P}\left((-\infty , b]\right) \] By the definition of the cumulative distribution function, $F(a)+ \mathbf{P}\left((a, b]\right) = F(b)$. Finally we get $\mathbf{P}\left((a, b]\right) = F(b) - F(a)$.

Let us give the simplest examples of cumulative distribution functions.

Let

\[ F(x)= \begin{cases} 0, & x<0 \\ x, & 0 \leq x \leq 1 \\ 1, & x>1\end{cases} \]

In this case the corresponding probability distribution $\lambda$ is called Lebesgue measure on the segment $[0,1]$. It is clear that for any $a<b \in [0,1]$ we have

\[ \lambda ((a, b))=\lambda ([a, b))=\lambda ((a, b])=\lambda ([a, b])=b-a \]
Let

\[ F(x)= \begin{cases} 0, & x<c \\ 1, & x \geq c\end{cases} \]

The corresponding probability distribution:

\[ \mathbf{P}\left(A\right) = \delta _c(A) = \begin{cases} 0, & c \notin A \\ 1, & c \in A\end{cases} \]

for $A \in \mathscr {B}\left(\mathbb {R}\right)$. That is, this distribution corresponds to a unit mass concentrated at the point $c$. Such a probability measure is called the Dirac measure (notation: $\delta_c$).

Problem 2

For $\xi \sim \operatorname {Geom}(p)$, find the cumulative distribution function $F_{\xi }(n)$. Find the so-called survival function \[ S_{\xi }(n) = 1 - F_{\xi }(n) = \mathbb {P}\left(\xi > n\right), \] $n \geq 0$.

Problem 3

For independent random variables $X_{1}, \ldots , X_{n} \sim \operatorname {Geom}(p)$, find the distribution of $\min \left\{ X_{1}, \ldots , X_{n}\right\}$.

Problem 4

For independent random variables $X_{1}, \ldots , X_{n} \sim \operatorname {Geom}(p)$, find the distribution of $X_{1}+\ldots +X_{n}$.

5 Miscellaneous problems

Example 4

Suppose $n$ butterflies are flying about. Vladimir, a butterfly enthusiast, has caught $X \sim \operatorname {Bin}(n, p)$ of them. Each caught butterfly turns out to be beautiful with probability $\theta$, independently of the total number of butterflies and of the other butterflies. Let $Y$ be the number of beautiful butterflies among those caught. Find the distribution of $Y$.

Solution

First approach, conceptual. Each butterfly may be caught with probability $p$, and independently of that may be beautiful with probability $\theta$. That is, it is simultaneously caught and beautiful with probability $p \theta$ (i.e., has distribution $\operatorname {Ber}(p \theta )$), independently of the other butterflies. $Y$ is the sum of the caught-and-beautiful indicators. A sum of independent Bernoulli random variables with a common parameter $p \theta$ is a random variable distributed as $\operatorname {Bin}(n, p \theta )$.
Second approach: writing it out explicitly. The support of $X$ is the set $\{ 0,\ldots ,n\}$, so $Y$ has the same support (or a smaller one). Note that if there are $k$ beautiful ones among those caught ($Y = k$), then the total number caught is at least $k$ (and at most $n$ by assumption, $k \leq X \leq n$). That is,

\[ \left\{ Y = k\right\} = \bigsqcup _{i=k}^{n} \left\{ Y=k, X = i\right\} \]

By the law of total probability we have

\[ \begin{align} \mathbb {P}\left(Y = k\right) & = \sum _{i = k}^{n}\mathbb {P}\left(Y = k, X= i\right) = \sum _{i = k}^{n}\mathbb {P}\left(Y = k \mid X= i\right) \cdot \mathbb {P}\left(X = i\right) = \\ & = \sum _{i=k}^{n} \binom {i}{k} \theta ^{k}(1-\theta )^{i-k} \cdot \binom {n}{i} p^{i} (1-p)^{n-i} = \\ & = \binom {n}{k} (p \theta )^k \sum _{i=k}^{n} \binom {n-k}{i-k} p^{i-k} (1-\theta )^{i-k} (1-p)^{n-i} \\ & = \binom {n}{k} (p \theta )^k \sum _{j=0}^{n-k} \binom {n-k}{j} [p(1-\theta )]^j (1-p)^{n-k-j} \\ & = \binom {n}{k} (p \theta )^k [p(1-\theta ) + (1-p)]^{n-k} \\ & = \binom {n}{k} (p \theta )^k (1 - p\theta )^{n-k} \end{align} \]

where we substituted $j = i - k$ and used the binomial theorem.

Thus, $Y \sim \operatorname {Bin}(n, p\theta )$.

Answer

$\operatorname {Bin}(n, p \theta )$

Problem 5

Some insect lays $X \sim \operatorname {Pois}(\lambda )$ eggs. Of these, $Y$ offspring hatch, where the probability that the insect develops from an egg equals $p$, and the development of the insect from an egg does not depend on either the number of eggs or the development of the other eggs. Find the distribution of $Y$.

Problem 6

Osya and Kisa solve problems independently of each other: Osya works on Probability Theory and Measure Theory, and Kisa works on real analysis. It is known that the number of problems solved by Osya has distribution $\operatorname {Pois}(\lambda )$, and the number of problems solved by Kisa has distribution $\operatorname {Pois}(\theta )$. Find the probability that Osya solved $k$ problems, given that together they solved $n$ problems.

6 Expectation for a discrete distribution

Expectation is, informally, the average value taken by a random variable. Thus, the average value that comes up on a die is $3.5$, and the expectation of the random variable equal to the value shown by the die is likewise $3.5$.

If a random variable has a discrete distribution, then the expectation is naturally defined as a weighted sum of the values of the random variable.

Definition 4 The expectation of a random variable $\xi$ having a discrete distribution is the quantity $\mathbb {E}\left[\xi \right]$, equal to \[ \mathbb {E}\left[\xi \right] := \sum _{k \in \operatorname {supp}\xi } k \cdot \mathbb {P}\left(\xi =k\right), \]

If the set $X$ is infinite, then the sum in the definition of the expectation may be infinite, or may fail to exist at all. When the sum is infinite, we say that the expectation is infinite. If the sum does not exist, then the expectation does not exist either.

If $\varphi : \mathbb {R} \rightarrow \mathbb {R}$ is an arbitrary Borel function, then by definition $\mathbb {E}\left[\varphi (\xi )\right] = \sum_{k \in \varphi (X)} k \cdot \mathbb {P}\left(\varphi (\xi ) = k\right)$, where $\varphi (X)$ is the range of the transformed random variable $\varphi (\xi )$. In practice, finding the range of $\varphi (\xi )$ and the distribution of $\varphi (\xi )$ is often rather involved; instead, the following formula is used:

\[ \mathbb {E}\left[\varphi (\xi )\right]=\sum _{x \in X} \varphi (x) \mathbb {P}\left(\xi =x\right) \]

where $X$ is the set of values of the random variable $\xi$.

Example 5

Compute $\mathbb {E}\left[\xi \right]$, $\mathbb {E}\left[\xi^2\right]$, $\mathbb {E}\left[e^{\xi }\right]$, if

$\xi \sim \operatorname {Bern}(p)$, $p \in (0,1)$. That is, $\mathbb {P}\left(\xi = 1\right) = 1 - \mathbb {P}\left(\xi = 0\right) = p$.
$\xi \sim \operatorname {Pois}(\lambda )$, $\lambda > 0$. That is, $\mathbb {P}\left(\xi = k\right) = \frac{\lambda^k}{k!}e^{-\lambda }$.

Solution

(1)

\[ \begin{align} \mathbb {E}\left[\xi \right] & = 0 \cdot \mathbb {P}\left(\xi = 0\right) + 1 \cdot \mathbb {P}\left(\xi = 1\right) = p \\ \mathbb {E}\left[\xi ^2\right] & = 0^2 \cdot \mathbb {P}\left(\xi = 0\right) + 1^2 \cdot \mathbb {P}\left(\xi = 1\right) = p \\ \mathbb {E}\left[e^\xi \right] & = e^0 \cdot \mathbb {P}\left(\xi = 0\right) + e^1 \cdot \mathbb {P}\left(\xi = 1\right) = (1 - p) + pe \end{align} \]

(2)

\[ \begin{align} \mathbb {E}\left[\xi \right] & = \sum _{k=0}^{\infty } k \mathbb {P}\left(\xi =k\right)=\lambda e^{-\lambda } \sum _{k=1}^{\infty } \frac{\lambda ^{k-1}}{(k-1)!}=\lambda \\ \mathbb {E}\left[\xi ^2\right] & = \sum _{k=0}^{\infty } k^2 \mathbb {P}\left(\xi =k\right)= e^{-\lambda } \sum _{k=1}^{\infty } \quad \underbrace{k}_{(k-1) + 1} \frac{\lambda ^k}{(k-1)!} = e^{-\lambda } \cdot \left(\sum _{k=2}^{\infty } \frac{\lambda ^k}{(k-2)!} + \sum _{k=1}^{\infty } \frac{\lambda ^k}{(k-1)!}\right) = \\ & = e^{-\lambda } \cdot \left(\lambda ^2 e^{\lambda } + \lambda e^{\lambda }\right) = \lambda ^2 + \lambda \\ \mathbb {E}\left[e^\xi \right] & = \sum _{k=0}^{\infty } e^{k} \mathbb {P}\left(\xi =k\right) = e^{-\lambda } \sum _{k=0}^{\infty } \frac{(\lambda e)^k}{k!} = e^{-\lambda } \cdot e^{\lambda e} = e^{\lambda (e - 1)} \end{align} \]

Properties:

Expectation is linear:
- $\mathbb {E}\left[\alpha \cdot \xi \right] = \alpha \cdot \mathbb {E}\left[\xi \right]$, $\alpha \in \mathbb {R}$, $\xi$ a random variable.
- $\mathbb {E}\left[\xi + \eta \right] = \mathbb {E}\left[\xi \right] + \mathbb {E}\left[\eta \right]$ for arbitrary random variables $\xi$, $\eta$ (provided the expectations exist).
If the random variables are independent, then expectation is multiplicative:

\[ \mathbb {E}\left[\xi \cdot \eta \right] = \mathbb {E}\left[\xi \right] \cdot \mathbb {E}\left[\eta \right] \]

However, in general, expectation is typically not multiplicative: $\mathbb {E}\left[\xi \cdot \eta \right] \neq \mathbb {E}\left[\xi \right] \cdot \mathbb {E}\left[\eta \right]$. In particular, if $\xi$ has some non-degenerate distribution (not a Dirac distribution), then, for example, $\mathbb {E}\left[\xi^2\right] \neq \left(\mathbb {E}\left[\xi \right]\right)^2$.

expectation in the general caseexpectation We have defined expectation only for discrete distributions — as the sum of a series. For an arbitrary random variable, $\mathbb {E}\left[\xi \right]$ is defined as the Lebesgue integral $\int_{\Omega } \xi \, d\mathbb {P}$, which we will construct in the chapter on constructing the Lebesgue integral. There (and in the chapter on its properties) the properties we are already using will be honestly proved in full generality: linearity, multiplicativity for independent variables, passages to the limit. In the discrete case our definition agrees with the general one, and linearity is proved elementarily (by rearranging sums) — check this yourself.

7 Variance

Of course, from a single known average value of a random variable it is hard to judge its distribution. Additional information is provided by the mean squared deviation from the expectation, which is called the variance.

Definition 5 The variance of a random variable $\xi$ is the quantity \[ \operatorname {Var}\left[\xi \right] = \mathbb {E}\left[(\xi - \mathbb {E}\left[\xi \right])^2\right] \]

By linearity of expectation there is a convenient formula for the variance:

\[ \operatorname {Var}\left[\xi \right] = \mathbb {E}\left[\xi ^2 - 2\xi \cdot \mathbb {E}\left[\xi \right] + \left(\mathbb {E}\left[\xi \right]\right)^2\right] = \mathbb {E}\left[\xi ^2\right] - 2 \cdot \mathbb {E}\left[\xi \right] \cdot \mathbb {E}\left[\xi \right] + \left(\mathbb {E}\left[\xi \right]\right)^2 = \mathbb {E}\left[\xi ^2\right] - \left(\mathbb {E}\left[\xi \right]\right)^2 \]

Example 6

A random variable $\xi$ has the Poisson distribution with parameter $\lambda$. Find $\operatorname {Var}\left[\xi \right]$.

Solution

We already obtained above $\mathbb {E}\left[\xi \right] = \lambda$, $\mathbb {E}\left[\xi^2\right] = \lambda^2 + \lambda$. Hence \[ \operatorname {Var}\left[\xi \right] = \mathbb {E}\left[\xi ^2\right] - \left(\mathbb {E}\left[\xi \right]\right)^2 = \lambda ^2 + \lambda - \lambda ^2 = \lambda \]

Example 7

Let $R_n = \left\{ 1, \ldots , n\right\}$. The random variable $\xi$ equals the number of elements of $R_{n}$ that remain in their own place under a random permutation. Find $\mathbb {E}\left[\xi \right]$ and $\operatorname {Var}\left[\xi \right]$.

Solution

For each $i \in \{ 1, \ldots , n\}$ let the random variable $\xi_{i}$ be the indicator of the event $A_{i}$, which consists in the element $i$ remaining in its own place after the random permutation. Note that $\xi_i \sim \operatorname {Ber}(p_i)$, where $p_i = \mathbb {P}\left(A_i\right)$. We have

\[ \xi =\xi _{1}+\ldots +\xi _{n} \]

Expectation:

\[ \mathbb {E}\left[\xi _i\right] = \mathbb {P}\left(A_{i}\right)=\frac{(n-1)!}{n!}=\frac{1}{n} \]

By the linearity property of expectation we get

\[ \mathbb {E}\left[\xi \right]=\mathbb {E}\left[\xi \right]_{1}+\ldots +\mathbb {E}\left[\xi \right]_{n}=1 \]

Variance:

\[ \begin{align} \operatorname {Var}\left[\xi \right] & = \mathbb {E}\left[\xi \right]^{2}-(\mathbb {E}\left[\xi \right])^{2} = \mathbb {E}\left[\left(\xi _{1}+\ldots +\xi _{n}\right)^2\right] - 1 = \\ & = \sum _{i, j=1}^{n} \mathbb {E}\left[\xi _{i} \xi _j\right] - 1 =\sum _{i=1}^{n} \mathbb {E}\left[\xi _{i}^2\right]+2 \sum _{1 \leq i<j \leq n} \mathbb {E}\left[\xi _{i} \xi _{j}\right]-1= \\ & =\sum _{i=1}^{n} \mathbb {E}\left[\xi _{i}\right]+n(n-1) \mathbb {E}\left[\xi _{1} \xi _{2}\right]-1=1+n(n-1) \frac{(n-2)!}{n!}-1= \\ & = 1 \end{align} \]

Problem 7

A six-sided die is tossed independently $n$ times. The random variable $\xi$ equals the number of ones rolled, and $\eta$ equals the sum of the pips rolled. Find the expectations of these random variables.

Problem 8

A random variable $\xi$ has a discrete distribution on $\mathbb {N}$, \[ \mathbf{P}\left(\left\{ k\right\} \right) = \frac{C}{k(k+1)} \qquad \text{ for } \quad k \in \mathbb {N} \] Find the constant $C \in \mathbb {R}$. Find the expectation for the resulting distribution.

Footnotes

The notation $\operatorname {supp}(\xi + \eta )$ denotes the support of the distribution of $\xi + \eta$, i.e., $\operatorname {supp}(\xi + \eta ) = \operatorname {supp}\mathbf{P}_{\xi + \eta }$.↩︎