This lecture note is still being edited: the text may contain typos, inaccuracies, and locally unpolished passages. If you find something, please let the author know (contact information on the course page).
So far we have worked with the Lebesgue integral of a fixed measurable function. Now we will look at the set of measurable functions as a single object — a vector space — and equip it with the norm
\[
\left\| f\right\| _{p} = \left(\int \left|f\right|^p \, d\mu \right)^{1/p}.
\]
This gives us the spaces \(L^p(\mu )\) — the main character of this chapter and among the most important examples of infinite-dimensional function spaces in all of analysis, probability theory, mathematical statistics, and the theory of random processes. For example:
random variables with finite second moment form the space \(L^2(\mathbb {P})\), and \(L^2\)-geometry (orthogonality, projections, best mean-square approximation) is exactly the language in which conditional expectation and linear regression are formulated;
convergence in \(L^p\) is one of the basic types of convergence of random variables, alongside a.s. convergence and convergence in probability;
in mathematical statistics, the asymptotics of estimators are often studied via \(L^p\)-convergence, and many statistical tests rely on empirical processes that naturally live in Banach spaces of continuous functions;
in the theory of random processes, the sample paths of a process (say, Brownian motion) are, as a rule, elements of some functional Banach space (typically \(C[0,T]\)), and questions about the convergence of distributions of processes are formulated as the convergence of probability measures precisely on such a space.
To really get to grips with \(L^p(\mu )\), we will need two additional structures, to which §2 and §3 of this chapter are devoted: the structure of a Banach space (completeness — without it, the limit passages needed for convergence in \(L^p\) and for functional limit theorems in the theory of random processes would not work) and, for the case \(p=2\) separately, the structure of a Hilbert space (the geometry of orthogonality and projections — on which, for example, the definition of conditional expectation rests). The exposition in this chapter follows (Axler 2020, chaps. 6–8), although we will substantially trim the material on the Baire category theorem — that is a topic for a separate course in functional analysis; here we will only note its existence.
Definition and basic properties of \(L^p(\mu )\)
Definitions: the space of integrable functions \(\mathcal{L}^p(\mu )\) and the quotient space \(L^p(\mu )\)
Fix a measure space \((X, \mathcal F, \mu )\) and a number \(0 < p \leq \infty\).
Definition 1 For a measurable function \(f: X \to \mathbb F\) define the \(p\)-norm
\[
\left\| f\right\| _{p} := \left(\int \left|f\right|^p \, d\mu \right)^{1/p}.
\]
For \(p = \infty\) define the essential supremum
\[
\left\| f\right\| _{\infty } := \operatorname {esssup}_{x \in X}\left|f(x)\right| = \inf \left\{ t>0 \; : \; \mu \left(\left|f\right| > t\right) = 0\right\}
\]
(i.e., the smallest constant bounding \(\left|f\right|\) outside a set of \(\mu\)-measure \(0\)).
Denote by \(\mathcal{L}^p(X,\mathcal{F}, \mu )\) (or briefly \(\mathcal{L}^p(\mu )\)) the set of measurable functions \(f: X \to \mathbb {F}\) with finite \(p\)-norm:
\[
\mathcal{L}^p(X,\mathcal{F}, \mu ) := \left\{ f : X \to \mathbb {F} \; : \; f \text{ is } \mathcal{F}\mid \mathscr {B}\left(\mathbb {F}\right)\text{-meas., } \left\| f\right\| _{p} < \infty \right\}
\]
Lemma 1 (\(\mathcal{L}^p(\mu )\) is a vector space) Let \((X, \mathcal{F}, \mu )\) be a measure space and \(0 < p < \infty\). Then
\(\left\| f+g\right\|_{p}^p \leq 2^p \left(\left\| f\right\|_{p}^p + \left\| g\right\|_{p}^p\right)\)
\(\left\| af\right\|_{p} = \left|a\right| \left\| f\right\|_{p}\) for all \(f, g \in \mathcal{L}^p(\mu )\) and all \(a \in \mathbb {F}\). Moreover, with the usual operations of addition and scalar multiplication of functions, \(\mathcal{L}^p(\mu )\) is a vector space.
The word “norm” in Definition Definition 1 is provisional for now: as we will see, \(\left\| \cdot \right\|_{p}\) is often not a norm on \(\mathcal L^p(\mu )\) in the strict sense — positive definiteness fails. Namely, if there exists a nonempty set \(E \in \mathcal{F}\) such that \(\mu (E)=0\), then the \(p\)-norm of the indicator function \(\; \mathbb {1}_{E}\) is zero, while the function itself is nonzero. Consequently, in this case the form \(\left\| \cdot \right\|_{p}\) is not positive definite (\(\left\| f\right\|_{p} = 0 \iff f = 0_{V}\) fails to hold), i.e., it is not a norm on \(V = \mathcal{L}^p(\mu )\). The standard way to fix this problem is to identify functions that differ only on a set of \(\mu\)-measure \(0\).
Definition 2 Suppose \((X, \mathcal{F}, \mu )\) is a measure space and \(0<p \leq \infty\).
\(\mathcal{Z}(\mu )\) denotes the set of \(\mathcal{F}\)-measurable functions from \(X\) to \(\mathbb {F}\) that equal 0 almost everywhere.
For \(f \in \mathcal{L}^p(\mu )\) let \([f]\) be the subset of \(\mathcal{L}^p(\mu )\) defined by the formula
\[
[f] = \left\{ f + z \; : \; z \in \mathcal{Z}(\mu )\right\}
\]
The set \(\mathcal{Z}(\mu )\) is closed under multiplication by scalars. Moreover, \(\mathcal{Z}(\mu )\) is closed under addition, since the union of two sets of \(\mu\)-measure \(0\) is a set of \(\mu\)-measure \(0\). Thus, \(\mathcal{Z}(\mu )\) is a subspace of \(\mathcal{L}^p(\mu )\).
Note that if \(f, g \in \mathcal{L}^p(\mu )\), then \([f] = [g]\) if and only if \(f(x)=g(x)\) for almost all \(x \in X\).
Suppose that \(\mu\) is a measure, and \(0<p \leq \infty\).
Let \(L^p(\mu )\) denote the collection of subsets of \(\mathcal{L}^p(\mu )\) defined by the relation
\[
L^p(\mu )=\left\{ [f] \; : \; f \in \mathcal{L}^p(\mu )\right\} .
\]
For \([f], [g] \in L^p(\mu )\) and \(\alpha \in \mathbb {F}\) define \([f]+[g]\) and \(\alpha [f]\) as follows:
\[
[f]+[g]=[f + g] \quad \text{ and } \quad \alpha [f]= [\alpha f].
\]
The elements of \(L^p(\mu )\) can be viewed as equivalence classes of functions in \(\mathcal{L}^p(\mu )\), where two functions are equivalent if they agree almost everywhere.
In functional analysis, elements of \(L^p(\mu )\) are often called functions, where two functions are considered equal if they differ only on a set of \(\mu\)-measure \(0\). This abuse of notation is harmless, provided that the operations performed on such “functions” give the same results when the functions are changed on a set of measure \(0\).
Note the typographical distinction between \(\mathcal{L}^p(\mu )\) and \(L^p(\mu )\). An element of the calligraphic \(\mathcal{L}^p(\mu )\) is a function; an element of \(L^p(\mu )\) is a set of functions, any two of which agree almost everywhere.
Suppose that \(\mu\) is a measure, and \(0<p \leq \infty\). Define \(\| \cdot \|_p\) on \(L^p(\mu )\) as follows:
\[
\| [f]\| _p := \| f\| _p
\]
where \(f \in \mathcal{L}^p(\mu )\) is an arbitrary representative of \([f]\). Note that if \(f, g \in \mathcal{L}^p(\mu )\) and \([f] = [g]\), then \(\left\| f\right\|_{p} = \left\| g\right\|_{p}\). Thus, the definition above is well posed: it does not depend on the particular representative.
For readers familiar with quotient spaces: you may notice that \(L^p(\mu )\) is the quotient space
\[
\mathcal{L}^p(\mu ) / \mathcal{Z}(\mu ).
\]
Show that the previously introduced sequence spaces \(\ell^{p}\) and \(\ell^{\infty }\) are special cases of Lebesgue spaces.
Show that the previously introduced spaces \(\mathbb {F}^{n}\) with the norm \(\left\| \cdot \right\|_{p}\) are also special cases of Lebesgue spaces.
\(\ell^{p}\) and \(\ell^{\infty }\) are sequence spaces, i.e., spaces of functions from \(\mathbb {N}\) to the field \(\mathbb {F}\). So here \(X = \mathbb {N}\), \(\mathcal{F} = 2^{\mathbb {N}}\).
Next, we need to define the measure. In which case does the Lebesgue integral simply turn into a sum over all values? In the case where \(\mu\) is the counting measure \(\#\):
\[
\# (A) = \begin{cases} \left|A\right|, & A \text{ finite} \\ +\infty , & \text{otherwise} \end{cases}, \qquad A \subset \mathbb {N}
\]
In total,
\[
\ell ^{p} = \mathcal{L}^{p}(\mathbb {N}, 2^{\mathbb {N}}, \# )
\]
\(0 < p \leq +\infty\).
The vector space \(\mathbb {F}^{n}\) of length-\(n\) tuples of scalars from the field \(\mathbb {F}\) can be identified with \(\mathbb {F}^{\left\{ 1,\ldots ,n\right\} }\), i.e., the space of functions from \(\left\{ 1,\ldots ,n\right\}\) to \(\mathbb {F}\). For instance, for \(n=4\) the tuple \((-2, \pi , 5.4, 0)\) corresponds to the function
\[
f(k) = \begin{cases} -2, & k = 1 \\ \pi , & k = 2\\ 5.4, & k = 3 \\ 0, & k = 4 \end{cases}
\]
The vector spaces \(\mathbb {F}^{n}\) and \(\mathbb {F}^{\left\{ 1,\ldots ,n\right\} }\) are isomorphic, so we will often use them interchangeably. The norm
\[
\left\| x\right\| _{p} = \begin{cases} \left(\sum _{k=1}^{n}\left|x_k\right|^p\right)^{1/p}, & p < +\infty \\ \max _{k \in \left\{ 1,\ldots , n\right\} }\left|x_k\right|, & p = +\infty \end{cases}
\]
transfers under this isomorphism from \(\mathbb {F}^{n}\) to \(\mathbb {F}^{\left\{ 1,\ldots ,n\right\} }\), and, up to an isometric (i.e., norm-preserving) isomorphism of normed spaces, we can write
\[
\left(\mathbb {F}^{n}, \left\| \cdot \right\| _{p}\right) = \left(\mathbb {F}^{\left\{ 1,\ldots ,n\right\} }, \left\| \cdot \right\| _{p}\right) = \mathcal{L}^p\left(\left\{ 1,\ldots ,n\right\} , 2^{\left\{ 1,\ldots ,n\right\} }, \# \right)
\]
Again the counting measure.
In general, \(\ell^p(X)\) denotes the space of functions on \(X\) with the counting measure:
\[
\ell ^p(X) := \mathcal{L}^p\left(X, 2^{X}, \# \right)
\]
In particular,
\[
\ell ^{p} = \ell ^{p}(\mathbb {N}), \qquad \left(\mathbb {F}^n, \left\| \cdot \right\| _{p}\right) = \ell ^p(\left\{ 1,\ldots ,n\right\} )
\]
(up to isomorphism of normed spaces).
Hölder’s and Minkowski’s inequalities
Definition 3 For \(1\leq p \leq \infty\) the conjugate (dual) exponent to \(p\) is the number \(p' \in [1,\infty ]\) such that \(\frac1p+\frac1{p'}=1\) (for \(p=1\) we set \(p'=\infty\), for \(p=\infty\) we set \(p'=1\)).
Find the conjugate exponents for \(1, \infty , 2, 4, \frac{5}{3}\).
\[
1' = \infty , \quad \infty ' = 1, \quad 2' = 2, \quad 4' = \frac{4}{3}, \quad \left(\frac{5}{3}\right)' = \frac{5}{2} = 2.5
\]
Lemma 2 (Young’s inequality) Let \(1<p<\infty\). Then for all \(a,b\geq 0\): \[
ab \leq \frac{a^p}{p} + \frac{b^{p'}}{p'}.
\]
For what relation between \(a,b\) (and \(p\)) is equality attained in Young’s inequality?
Theorem 1 (Hölder’s inequality) Let \((X,\mathcal F,\mu )\) be a measure space, \(1\leq p \leq \infty\), \(f,h: X \to \mathbb F\) measurable. Then \[
\left\| fh\right\| _{1} \leq \left\| f\right\| _{p}\cdot \left\| h\right\| _{p'}.
\]
Derive the Cauchy–Bunyakovsky–Schwarz inequality \(\left|\int fh \, d\mu \right| \leq \left\| f\right\|_{2}\left\| h\right\|_{2}\) as a special case of Hölder’s inequality.
Take \(p=p'=2\) and apply the triangle inequality for the modulus of a complex number to \(\int fh \, d\mu\).
Now Minkowski’s inequality, which makes up for that “shortfall” of the constant \(2^p\) from Lemma Lemma 1.
Theorem 2 (Minkowski’s inequality) Let \((X,\mathcal F,\mu )\) be a measure space, \(1 \leq p \leq \infty\), \(f,g \in \mathcal L^p(\mu )\). Then \[
\left\| f+g\right\| _{p} \leq \left\| f\right\| _{p} + \left\| g\right\| _{p}.
\]
Corollary 1 (\(L^p(\mu )\) is a normed space) For \(1\leq p \leq \infty\), \(L^p(\mu )\) is a vector space (Lemma Lemma 1), and \(\left\| \cdot \right\|_{p}\) is a norm on it: positive definiteness holds by construction (passing to equivalence classes \(L^p(\mu ) = \mathcal L^p(\mu )/\mathcal Z(\mu )\)), homogeneity is clear from the definition, and the triangle inequality is exactly Minkowski’s inequality.
Inclusions between \(L^p\)-spaces. Lyapunov’s inequality
On spaces of finite measure (in particular, on probability spaces), \(L^p\)-spaces are nested inside one another — and in a direction that may at first seem surprising: the larger \(p\) is, the smaller the space.
Theorem 3 (Lyapunov’s inequality: nesting of \(L^p\)-spaces for a finite measure) Let \((X, \mathcal{F}, \mu )\) be a measure space with \(\mu (X) < \infty\). Suppose that \(0 < p < q \leq \infty\). Then \(L^q(\mu ) \subset L^p(\mu )\). Moreover, if \(0 < p < q < \infty\), then
\[
\left\| f\right\| _{p} \leq \mu (X)^{\frac{1}{p} - \frac{1}{q}} \| f\| _q
\]
for all \(f \in L^q(\mu )\).
In particular, if \(\mu\) is a probability measure (i.e., \(\mu (X) = 1\)), then all random variables from \(L^p\) also lie in \(L^q\) for \(0 < p < q \leq +\infty\), and moreover
\[
\left(\mathbb {E}\left[\left|\xi \right|^p\right]\right)^{1/p} = \left\| \xi \right\| _{p} \leq \left\| \xi \right\| _{q} = \left(\mathbb {E}\left[\left|\xi \right|^q\right]\right)^{1/q}
\]
for an arbitrary random variable \(\xi\) from \(L^q\).
This (in particular) explains why, in probability theory, the existence of higher moments is a stronger requirement: for instance, if \(\mathbb {E}\left[X^4\right]<\infty\), then automatically \(\mathbb {E}\left[X^2\right]<\infty\) and \(\mathbb {E}\left[\left|X\right|\right]<\infty\), but not conversely.
Let \(\mathbb {P}\) be a probability measure, \(1\leq s<t<\infty\), and \(X_n \to X\) in \(L^t(\mathbb {P})\) (i.e., \(\mathbb {E}\left[\left|X_n-X\right|^t\right]\to 0\)). Prove that then \(X_n\to X\) in \(L^s(\mathbb {P})\) as well.
Apply Lyapunov’s inequality (Theorem Theorem 3) to \(f = X_n-X\).
Of course, the inclusion \(L^t\subset L^s\) (\(s<t\)) is generally strict: in \(L^s([0,1])\) (with respect to Lebesgue measure) there are functions not lying in \(L^t([0,1])\).
For which \(p>0\) does the function \(f(x) = x^{-\alpha }\) (\(\alpha >0\) a parameter) lie in \(\mathcal L^p\left((0,1)\right)\) (Lebesgue measure)?
We have \(\int_0^1 x^{-\alpha p}\, dx\). This is an improper Riemann integral (and simultaneously a Lebesgue integral — the function is nonnegative, monotone, hence Borel measurable; see the theorem on the agreement of the Riemann and Lebesgue integrals for a singularity of one sign — we use \(\int_0^1 = \lim_{\varepsilon \downarrow 0}\int_\varepsilon^1\) and monotone convergence), which converges if and only if \(\alpha p < 1\), i.e., \(p < 1/\alpha\). In total, \(f \in \mathcal L^p((0,1)) \iff p<1/\alpha\).
In particular, for \(\alpha = \tfrac 12\): \(f(x)=x^{-1/2} \in L^1((0,1))\) (even Riemann integrable in the usual sense, see the example in the chapter on the properties of the Lebesgue integral), but \(f \not\in L^2((0,1))\) — this is a concrete example of a function showing that the inclusion \(L^2\subset L^1\) on \((0,1)\) is strict.
On spaces of infinite measure (for example, with a counting measure), the situation with inclusions is exactly the opposite.
Let \(0 < p < q \leq \infty\).
Prove that \(\ell^{p} \subsetneq \ell^{q}\) (in particular, the inclusion here goes the other way compared to the case of finite measure!).
Prove that \(\ell^{p} \subsetneq \bigcap_{r > p} \ell^{r}\).
Prove that \(\left\| a\right\|_{p} \geq \left\| a\right\|_{q}\) for every sequence \(a = (a_{1}, a_{2}, \ldots )\) of elements of \(\mathbb {F}\).
For the inclusion \(\ell^p\subset \ell^q\), note that \(a_k \to 0\), so \(\left|a_k\right|\leq 1\) for large \(k\), and then \(\left|a_k\right|^q \leq \left|a_k\right|^p\). For strictness of the inclusion, consider \(a_k = k^{-1/p}\).
Recall (Corollary Corollary 1) that \(L^p(\mu )\) is already known to us as a normed space. In particular, for \(\mu = \#\) (the counting measure on \(\mathbb {N}\)), \(L^p(\# ) = \mathcal L^p(\# ) = \ell^p\) — no passage to equivalence classes is required, since the counting measure has no sets of measure zero other than the empty set (cf. Example Example 1).
Prove that if \(\mu\) is a measure, \(1 < p < \infty\), and \(f, g \in L^{p}(\mu )\) are such that
\[
\left\| f\right\| _{p} = \left\| g\right\| _{p} = \left\| \frac{f+g}{2}\right\| _{p}
\]
then \(f = g\).
Give an example showing that the result of part (a) may fail for \(p = 1\).
Give an example showing that the result of part (a) may fail for \(p = \infty\).
The Banach space \(L^p(\mu )\)
The word “norm” in the previous section was justified (Minkowski’s inequality). But to speak of convergence in \(L^p(\mu )\) — and even more so, of the limit actually existing, rather than merely of a formula for the norm being written down — we need to understand the abstract notion of completeness. This is a classical topic in functional analysis — Banach spaces; let us apply it here to our main object, the space \(L^p(\mu )\).
Metric spaces: the necessary minimum
Definition 4 A metric on a nonempty set \(V\) is a function \(d: V \times V \to [0,+\infty )\) such that
\(d(f,g) = 0\) if and only if \(f = g\) (positive definiteness);
\(d(f,g) = d(g,f)\) for all \(f,g \in V\) (symmetry);
\(d(f,h) \leq d(f,g) + d(g,h)\) for all \(f,g,h \in V\) (triangle inequality). The pair \((V,d)\) is called a metric space.
A standard example is \(\mathbb {R}^{n}\) with \(d(x,y) = \left|x-y\right|\). But we will primarily be interested in metrics on spaces of functions: say, on the set of continuous functions \(C[a,b]\) one can set \(d(f,g) = \sup_{x \in [a,b]}\left|f(x)-g(x)\right|\).
Definition 5 Let \((V,d)\) be a metric space, \(f_1, f_2, \ldots \in V\), \(f \in V\). We say that \(f_k \to f\) (limit in a metric space) if \(d(f_k, f) \to 0\) as \(k \to \infty\).
A sequence \(f_1, f_2, \ldots \in V\) is called a Cauchy sequence if for every \(\varepsilon >0\) there exists \(N \in \mathbb {N}\) such that \(d(f_j, f_k) < \varepsilon\) for all \(j,k \geq N\).
As in \(\mathbb {R}\), every convergent sequence is a Cauchy sequence (triangle inequality). The converse, in general, is false — and it is precisely the spaces where it does hold that will interest us.
Definition 6 A metric space \((V,d)\) is called complete if every Cauchy sequence in \(V\) converges to some element of \(V\).
For example, \(\mathbb {R}\) (with the usual metric) is complete — this is, in essence, the completeness axiom of the real numbers known from the analysis course. But \(\mathbb {Q}\) is not complete: a sequence of rational numbers approximating \(\sqrt2\) (say, decimal approximations) is a Cauchy sequence in \(\mathbb {Q}\), but does not converge to any rational number.
We will not need more about metric spaces (open/closed sets, compactness, etc.) — all we need are the notions of limit, Cauchy sequence, and completeness, applied to spaces of functions.
Normed and Banach spaces
Recall that a vector space over a field \(\mathbb F\) (for us \(\mathbb F = \mathbb {R}\) or \(\mathbb F = \mathbb {C}\)) is a set with operations of addition and scalar multiplication satisfying the standard axioms (commutativity and associativity of addition, existence of zero and of an opposite element, axioms for scalar multiplication). In the examples of interest to us, \(V\) will always be some space of functions \(X \to \mathbb F\) with pointwise operations, where all the vector space axioms can easily be verified.
Definition 7 A norm on a vector space \(V\) (over \(\mathbb F\)) is a function \(\left\| \cdot \right\| : V \to [0,+\infty )\) such that
\(\left\| f\right\| = 0 \iff f = 0\) (positive definiteness);
\(\left\| \alpha f\right\| = \left|\alpha \right|\left\| f\right\|\) for all \(\alpha \in \mathbb F, f \in V\) (homogeneity);
\(\left\| f+g\right\| \leq \left\| f\right\| + \left\| g\right\|\) for all \(f,g \in V\) (triangle inequality).
The pair \((V, \left\| \cdot \right\| )\) is called a normed space.
Give several examples of norms on finite- and infinite-dimensional spaces.
On \(\mathbb {R}^{n}\) (or \(\mathbb {C}^{n}\)): \(\left\| x\right\|_{1} = \left|x_1\right|+\ldots +\left|x_n\right|\), \(\left\| x\right\|_{\infty } = \max \left\{ \left|x_1\right|,\ldots ,\left|x_n\right|\right\}\), the usual Euclidean norm \(\left\| x\right\|_{2} = \sqrt{x_1^2+\ldots +x_n^2}\).
On the space \(C[a,b]\) of continuous functions on an interval: \(\left\| f\right\|_{\infty } = \sup_{x\in [a,b]}\left|f(x)\right|\) (the uniform norm), or \(\left\| f\right\|_{1} = \int_a^b \left|f\right|\).
On the space of sequences \(\ell^1 := \left\{ (a_1,a_2,\ldots ) \in \mathbb F^{\mathbb {N}}: \sum_{k=1}^\infty \left|a_k\right| < \infty \right\}\): \(\left\| a\right\|_{1} = \sum_{k=1}^\infty \left|a_k\right|\).
Verifying the norm axioms for all examples, except perhaps the triangle inequality, is trivial and is left to the reader.
Every normed space automatically becomes a metric space:
Lemma 3 Let \((V, \left\| \cdot \right\| )\) be a normed space. Then \(d(f,g) := \left\| f-g\right\|\) is a metric on \(V\).
Thus, all the notions of the previous section (limit, Cauchy sequence, completeness) automatically carry over to normed spaces: for example, \(f_k \to f\) means \(\left\| f_k - f\right\| \to 0\), and a Cauchy sequence is a sequence for which \(\left\| f_j - f_k\right\| \to 0\) as \(j,k \to \infty\).
Definition 8 A complete normed space is called a Banach space.
Let us figure out which of our examples are Banach spaces.
Theorem 4 \((\mathbb {R}^{n}, \left\| \cdot \right\|_{\infty })\) is a Banach space. Moreover, \(\mathbb {R}^{n}\) (and \(\mathbb {C}^{n}\)) is Banach with any norm.
The second statement (Banach-ness with any norm) follows from a standard fact of linear algebra: all norms on a finite-dimensional space are equivalent, i.e., for any two norms \(\left\| \cdot \right\|_a, \left\| \cdot \right\|_b\) on \(\mathbb {R}^{n}\) there exist constants \(0<c\leq C\) such that \(c\left\| x\right\|_a \leq \left\| x\right\|_b \leq C \left\| x\right\|_a\) for all \(x\); equivalent norms generate the same Cauchy sequences, so completeness with respect to one norm is equivalent to completeness with respect to any other.
What about infinite-dimensional function spaces? Here the choice of norm is already fundamentally important.
Theorem 5 \((C[a,b], \left\| \cdot \right\|_{\infty })\) is a Banach space.
Idea of the proof: the coordinatewise (more precisely, pointwise) limit of a Cauchy sequence in \(\left\| \cdot \right\|_{\infty }\) exists at every point \(x\in [a,b]\) (a numerical Cauchy sequence in \(\mathbb F\)), and Cauchy convergence in the uniform norm turns into uniform convergence \(f_k \to f\); the uniform limit of continuous functions is continuous — a standard fact of mathematical analysis.
But with the \(L^1\)-norm on the same space of continuous functions, everything breaks down.
Consider \((C[0,1], \left\| \cdot \right\|_{1})\), where \(\left\| f\right\|_{1} = \int_0^1 \left|f\right|\) is the Lebesgue integral (or the Riemann integral — in this case it does not matter). Prove that this space is not a Banach space.
Idea: we approximate (in \(L^1\)) a discontinuous function by continuous ones — the limit cannot exist in \(C[0,1]\). Let
\[
f_n(x) := \min \left(1, \, \max \left(0, \, n\left(x - \tfrac 12\right)\right)\right), \qquad n \in \mathbb {N}.
\]
This is a continuous piecewise linear function: it equals \(0\) on \([0,\tfrac 12]\), grows linearly from \(0\) to \(1\) on \([\tfrac 12, \tfrac 12+\tfrac 1n]\), and equals \(1\) on \([\tfrac 12+\tfrac 1n, 1]\) (for large \(n\); the formula is automatically correct also for \(n\) for which \(\tfrac 12+\tfrac 1n>1\)).
\((f_n)\) is a Cauchy sequence in \(\left\| \cdot \right\|_{1}\). For \(m>n\) the functions \(f_n, f_m\) coincide outside the interval \((\tfrac 12, \tfrac 12+\tfrac 1n)\) of length \(\tfrac 1n\), and on it both lie in \([0,1]\), so \(\left|f_n-f_m\right|\leq 1\) there. Hence \(\left\| f_n-f_m\right\|_{1} \leq \tfrac 1n \xrightarrow [n\to \infty ]{} 0\).
There is no limit in \(C[0,1]\). Suppose that \(f_n \to g\) in \(\left\| \cdot \right\|_{1}\) for some \(g \in C[0,1]\). Note that \(\left\| f_n - \; \mathbb {1}_{(1/2,1]}\right\|_{1} = \int_{1/2}^{1/2+1/n}\left|f_n(x) - 1\right|\, dx \leq \tfrac 1n \to 0\) (on the rest of the interval, \(f_n\) and the indicator coincide). Hence \(f_n \to \; \mathbb {1}_{(1/2,1]}\) and \(f_n \to g\) simultaneously in \(\left\| \cdot \right\|_{1}\), from which (triangle inequality) \(\left\| g - \; \mathbb {1}_{(1/2,1]}\right\|_{1} = 0\), i.e., \(g = \; \mathbb {1}_{(1/2,1]}\) a.e. (with respect to Lebesgue measure). But if a continuous \(g\) were to coincide with the indicator almost everywhere, then, by continuity of \(g\) and the density of points where \(g=0\) to the left of \(\tfrac 12\), and of points where \(g=1\) to the right of \(\tfrac 12\), we would get \(g(1/2^-) = 0\) and \(g(1/2^+)=1\) by continuity, contradicting \(g(1/2)=g(1/2)\) (incompatible left and right limits). Hence no such continuous \(g\) exists, and the Cauchy sequence \((f_n)\) does not converge in \(C[0,1]\).
The following completeness criterion is formally analogous to the criterion for absolute convergence of numerical series and will be our main technical tool below for proving the completeness of \(L^p(\mu )\).
Theorem 6 (completeness criterion via series) A normed space \(V\) is Banach if and only if, for every sequence \(g_1, g_2, \ldots \in V\) such that \(\sum_{k=1}^\infty \left\| g_k\right\| < \infty\), the series \(\sum_{k=1}^\infty g_k\) converges in \(V\) (i.e., the partial sums \(\sum_{k=1}^n g_k\) converge in \(V\) as \(n\to \infty\)).
(Idea: the direction “Banach \(\Rightarrow\) series converge” — the partial sums form a Cauchy sequence by the triangle inequality; the converse direction — any Cauchy sequence can be thinned to a subsequence with geometrically decreasing distances between consecutive terms, represented as the partial sums of a series, and the condition of the theorem applied, which gives convergence.)
Prove that \(\ell^1\) with the norm \(\left\| a\right\|_{\infty } := \sup_{k\in \mathbb {N}}\left|a_k\right|\) is not a Banach space (this is the norm \(\left\| \cdot \right\|_{\infty }\) inherited from \(\ell^\infty \supset \ell^1\), not the proper norm of \(\ell^1\)).
Consider \(g_n = \left(1, \tfrac 12, \tfrac 13, \ldots , \tfrac 1n, 0,0,\ldots \right) \in \ell^1\).
Linear functionals. The Hahn–Banach theorem
Definition 9 Let \(V, W\) be normed spaces. A linear map \(T: V \to W\) is called bounded if \[
\left\| T\right\| := \sup \left\{ \left\| Tf\right\| : f \in V, \; \left\| f\right\| \leq 1\right\} < \infty .
\] A linear functional on \(V\) is a linear map \(\varphi : V \to \mathbb F\) (into the field of scalars).
Theorem 7 A linear map \(T: V \to W\) between normed spaces is continuous if and only if it is bounded.
The set of bounded linear functionals on \(V\) is called the dual space and is denoted \(V'\); this is itself a normed (and even Banach) space with the norm \(\left\| \varphi \right\| = \sup \left\{ \left|\varphi (f)\right|: \left\| f\right\| \leq 1\right\}\).
Find the norm of the linear functional \(\varphi (f) = \int_0^1 f(x)\, dx\) on \((C[0,1], \left\| \cdot \right\|_{\infty })\).
On the one hand, for any \(f\) with \(\left\| f\right\|_{\infty }\leq 1\) we have \(\left|\varphi (f)\right| \leq \int_0^1 \left|f(x)\right|\, dx \leq \int_0^1 \left\| f\right\|_{\infty }\, dx \leq 1\), so \(\left\| \varphi \right\| \leq 1\). On the other hand, for \(f \equiv 1\) (constant) we have \(\left\| f\right\|_{\infty }=1\) and \(\varphi (f) = 1\). Hence the supremum is attained and \(\left\| \varphi \right\| = 1\).
The existence of discontinuous linear functionals on infinite-dimensional spaces is, oddly enough, a rather delicate fact, requiring a Hamel basis and Zorn’s lemma (we will not dwell on this). Far more useful for analysis is the following result — on the existence of continuous (= bounded) functionals “in sufficient quantity.”
Theorem 8 (Hahn–Banach, without proof) Let \(V\) be a normed space, \(U \subset V\) a subspace, \(\psi : U \to \mathbb F\) a bounded linear functional. Then \(\psi\) can be extended to a bounded linear functional \(\varphi : V \to \mathbb F\) with the same norm: \(\left\| \varphi \right\| = \left\| \psi \right\|\).
One of the most useful consequences of the Hahn–Banach theorem is that on any nontrivial normed space there are enough functionals to “sense” the norm of any vector.
Corollary 2 Let \(V\) be a normed space, \(f \in V\), \(f \neq 0\). Then there exists \(\varphi \in V'\) such that \(\left\| \varphi \right\| = 1\) and \(\varphi (f) = \left\| f\right\|\).
Indeed, it suffices to define \(\psi\) on the one-dimensional subspace \(U = \left\{ \alpha f: \alpha \in \mathbb F\right\}\) by the formula \(\psi (\alpha f) = \alpha \left\| f\right\|\) (then \(\left\| \psi \right\| =1\), \(\psi (f)=\left\| f\right\|\)) and extend it using the Hahn–Banach theorem.
Let \(V\) be a normed space such that \(V' = \left\{ 0\right\}\) (the only bounded linear functional on \(V\) is zero). Prove, using Corollary Corollary 2, that \(V = \left\{ 0\right\}\).
The topic of linear functionals and Banach spaces is much broader than what is described here: there is also the Baire category theorem and its consequences — the open mapping theorem, the closed graph theorem, the uniform boundedness principle (the Banach–Steinhaus principle). These results form the core of classical functional analysis, but lie beyond the scope of our course; those interested can consult, for example, (Axler 2020, sec. 6E) or any textbook on functional analysis.
Completeness: the Riesz–Fischer theorem
Now we can use the completeness criterion via series (Theorem Theorem 6) to obtain one of the most important properties of \(L^p\)-spaces.
Theorem 9 (Riesz–Fischer) Let \(\mu\) be a measure, \(1\leq p \leq \infty\). Then \(L^p(\mu )\) is a Banach space.
(Idea, for \(1\leq p<\infty\): by criterion Theorem 6 it suffices to take an arbitrary series \(\sum_k g_k\) with \(\sum_k\left\| g_k\right\|_{p}<\infty\) and exhibit its sum. The majorant \(G:=\sum_k\left|g_k\right|\) lies in \(\mathcal L^p(\mu )\) by Minkowski’s inequality and the monotone convergence theorem, hence is finite a.e.; where it is finite, the series \(\sum_k g_k(x)\) converges absolutely to some \(f(x)\), and \(\left|f-F_N\right|^p\leq G^p\) gives, by the dominated convergence theorem, \(\left\| f-F_N\right\|_{p}\to 0\).)
Density of simple functions
Theorem 10 Let \(1\leq p<\infty\), \(f \in \mathcal L^p(\mu )\). Then for every \(\varepsilon >0\) there exists a simple function \(s \in \mathcal L^p(\mu )\) such that \(\left\| f-s\right\|_{p}<\varepsilon\).
Convergence in \(L^p\) and other types of convergence
Convergence \(f_n \to f\) in \(L^p(\mu )\) (i.e., \(\left\| f_n-f\right\|_{p}\to 0\)) is the third type of convergence of measurable functions that we encounter in this course (after a.e. convergence and convergence in measure from the chapter on convergence of measurable functions). How are these convergences related?
Let \(1\leq p<\infty\), \(f_n \to f\) in \(L^p(\mu )\). Prove that \(f_n \to f\) in measure \(\mu\).
In particular, by F. Riesz’s theorem on convergence in measure (chapter on convergence of measurable functions), \(L^p\)-convergence implies the existence of a subsequence converging to \(f\) a.e. — entirely analogous to what happened with convergence in measure. The converse, of course, is false: a.e. convergence (and even uniform convergence) need not imply \(L^p\)-convergence without additional conditions (domination, etc.) — recall the counterexamples to Fatou’s lemma and the dominated convergence theorem from the chapter on the properties of the Lebesgue integral.
The Hilbert space \(L^2(\mu )\)
In §2 we introduced Banach spaces and, in particular, the spaces \(L^p(\mu )\) with norm \(\left\| \cdot \right\|_{p}\). Among all the \(L^p(\mu )\), the case \(p=2\) occupies a special place: it is precisely then that the norm is generated by some bilinear (in the complex case, sesquilinear) form — an inner product — and the space, besides lengths, acquires a geometry of angles and orthogonality. Such (complete) spaces are called Hilbert spaces.
The main reason we will need them in the course on probability theory, statistics, and random processes: the conditional expectation \(\mathbb {E}\left[X\mid \mathcal G\right]\) for \(X\in L^2\) is exactly the orthogonal projection of \(X\) onto the closed subspace \(L^2(\Omega ,\mathcal G,\mathbb {P})\). We will return to this observation below, right after proving the projection theorem — the main result of this section.
Inner products
Definition 10 Let \(V\) be a vector space over \(\mathbb F\), where \(\mathbb F = \mathbb {R}\) or \(\mathbb F = \mathbb {C}\). An inner product on the vector space \(V\) over the field \(\mathbb {F}\) (\(\mathbb {R}\) or \(\mathbb {C}\)) is a function \(\langle \cdot , \cdot \rangle : V \times V \to \mathbb {F}\) satisfying the properties:
Positive definiteness: \(\langle f, f \rangle \geq 0\) for all \(f \in V\), with \(\langle f, f \rangle = 0\) if and only if \(f = 0\).
Linearity in the first argument: \(\langle f + g, h \rangle = \langle f, h \rangle + \langle g, h \rangle\) and \(\langle \alpha f, g \rangle = \alpha \langle f, g \rangle\) for all \(f, g, h \in V\) and \(\alpha \in \mathbb {F}\).
Hermitian symmetry (conjugate symmetry): \(\langle f, g \rangle = \overline{\langle g, f \rangle }\) for all \(f, g \in V\). The pair \((V, \left\langle \cdot , \cdot \right\rangle )\) is called an inner product space (or a pre-Hilbert space).
From axioms 2-3 it is easy to derive linearity in the second argument with conjugation: \(\left\langle f, g+h \right\rangle = \left\langle f, g \right\rangle + \left\langle f, h \right\rangle\), \(\left\langle f, \alpha g \right\rangle = \bar\alpha \left\langle f, g \right\rangle\).
All the examples in this course coming from probability theory are real, so everywhere further in this section we take \(\mathbb F=\mathbb {R}\), unless explicitly stated otherwise. In this case axiom 3 turns into ordinary symmetry \(\left\langle f, g \right\rangle = \left\langle g, f \right\rangle\), and all the conjugations in the formulas below can simply be dropped. All definitions and theorems in this section carry over verbatim to the complex case (with a complex-conjugate bar appearing where appropriate); we will occasionally note this in remarks.
Examples.
Let \((X,\mathcal F,\mu )\) be a measure space. On \(L^2(\mu )\) define
\[
\left\langle f, g \right\rangle := \int fg \; d\mu .
\]
This is well defined: by Hölder’s inequality with \(p=q=2\) (proved in §1), for \(f,g\in L^2(\mu )\) we have \(\int \left|fg\right|\; d\mu \leq \left\| [\right\| 2]f\cdot \left\| [\right\| 2]g <\infty\), so the integral \(\int fg\; d\mu\) converges absolutely and is finite. The inner product \(\left\langle f, g \right\rangle = \int fg\, d\mu\) is exactly the form that generates the norm \(\left\| \cdot \right\|_{2}\) from §1 (see below for the definition of the norm generated by an inner product). This is the main example for us: almost everything we will need from probability theory later on is a special case of precisely this space.
Taking \(\mu = \#\) — the counting measure on \(\left\{ 1,\ldots ,n\right\}\) — we get \(\mathbb {R}^{n}\) with the standard inner product \(\left\langle (a_1,\ldots ,a_n), (b_1,\ldots ,b_n) \right\rangle = \sum_{i=1}^n a_ib_i\).
Taking \(\mu =\#\) — the counting measure on \(\mathbb {N}\) — we get \(\ell^2 := \left\{ (a_k)_{k=1}^\infty \; : \; \sum_k a_k^2 <\infty \right\}\) with the inner product \(\left\langle (a_k), (b_k) \right\rangle = \sum_{k=1}^\infty a_kb_k\).
Definition 11 Let \((V, \left\langle \cdot , \cdot \right\rangle )\) be an inner product space. The norm generated by the inner product is \[
\left\| f\right\| := \sqrt{ \left\langle f, f \right\rangle }, \qquad f \in V.
\]
From axiom 1 it immediately follows that \(\left\| f\right\| \geq 0\) and \(\left\| f\right\| = 0 \iff f=0\); from linearity and Hermitian symmetry — homogeneity \(\left\| \alpha f\right\| = \left|\alpha \right|\left\| f\right\|\) (check: \(\left\| \alpha f\right\|^2 = \left\langle \alpha f, \alpha f \right\rangle = \alpha \bar\alpha \left\langle f, f \right\rangle = \left|\alpha \right|^2\left\| f\right\|^2\)). What remains is the triangle inequality — it will follow from the following fundamental inequality.
Theorem 11 (Cauchy–Bunyakovsky–Schwarz inequality) Let \((V, \left\langle \cdot , \cdot \right\rangle )\) be a (real) inner product space, \(f,g\in V\). Then \[
\left| \left\langle f, g \right\rangle \right| \leq \left\| f\right\| \cdot \left\| g\right\| ,
\] with equality holding if and only if \(f,g\) are linearly dependent.
Corollary 3 (\(\left\| \cdot \right\|\) is indeed a norm) \(\left\| \cdot \right\|\) is a norm on \(V\): nonnegativity, nondegeneracy, and homogeneity were treated above, and the triangle inequality follows from Theorem Theorem 11: \[
\left\| f+g\right\| ^2 = \left\| f\right\| ^2+2 \left\langle f, g \right\rangle +\left\| g\right\| ^2 \leq \left\| f\right\| ^2 + 2\left\| f\right\| \left\| g\right\| + \left\| g\right\| ^2 = \left(\left\| f\right\| +\left\| g\right\| \right)^2.
\]
Theorem 12 (parallelogram identity) For all \(f,g\in V\) \[
\left\| f+g\right\| ^2+\left\| f-g\right\| ^2 = 2\left\| f\right\| ^2+2\left\| g\right\| ^2.
\]
Geometrically: the sum of the squares of the diagonals of a parallelogram equals the sum of the squares of all its sides. The parallelogram identity is exactly the property of a norm that distinguishes norms generated by an inner product from arbitrary norms: one can show (we will not prove this — see Axler, exercise 8A.16) that a norm on a vector space is generated by some inner product if and only if it satisfies the parallelogram identity.
Definition 12 (Hilbert space) A Hilbert space is an inner product space that is complete with respect to the norm generated by this inner product (i.e., a Banach space whose norm is generated by some inner product).
Examples. \(\mathbb {R}^{n}\) and \(\ell^2\) with the standard inner products are Hilbert spaces (completeness is a standard fact of linear algebra / analysis). The main example for us: \(L^2(\mu )\) is a Hilbert space for any measure space \((X,\mathcal F,\mu )\), since it is already known to us to be complete (the Riesz–Fischer theorem from §2), and its norm \(\left\| \cdot \right\|_{2}\) is generated by the inner product \(\left\langle f, g \right\rangle =\int fg\; d\mu\) introduced above.
Show that the norm \(\left\| \cdot \right\|_{1}\) on \(\mathbb {R}^{2}\) (i.e., \(\left\| (a,b)\right\|_{1} = \left|a\right| + \left|b\right|\)) does not satisfy the parallelogram identity. Deduce from this that this norm is not generated by any inner product on \(\mathbb {R}^{2}\) (in particular, \(\ell^1\) and \(L^1(\mu )\) are, in general, not Hilbert spaces).
Let \(\mu\) be a measure, \(f,g\in L^2(\mu )\). Show directly (without using Hölder’s inequality) that \(\int \left|fg\right|\; d\mu <\infty\), using the elementary inequality \(\left|ab\right|\leq \frac12(a^2+b^2)\) for \(a,b\in \mathbb {R}\).
Apply the inequality to \(a=\left|f(x)\right|\), \(b=\left|g(x)\right|\) and integrate.
Orthogonality. The projection theorem
Definition 13 Elements \(f,g\) of an inner product space are called orthogonal (\(f\perp g\)) if \(\left\langle f, g \right\rangle =0\).
Theorem 13 (Pythagorean theorem) If \(f\perp g\), then \(\left\| f+g\right\|^2 = \left\| f\right\|^2+\left\| g\right\|^2\).
Definition 14 Let \(U\subseteq V\) be an arbitrary subset of an inner product space \(V\). The orthogonal complement of \(U\) is \[
U^\perp := \left\{ h\in V \; : \; \left\langle g, h \right\rangle =0 \; \; \forall g\in U\right\} .
\]
Lemma 4 (properties of the orthogonal complement) Let \(U,W\) be subsets of an inner product space \(V\). Then
\(U^\perp\) is a closed linear subspace of \(V\);
\(U\cap U^\perp \subseteq \left\{ 0\right\}\);
if \(W\subseteq U\), then \(U^\perp \subseteq W^\perp\);
\(U \subseteq U^{\perp \perp } := \left(U^\perp \right)^\perp\).
(Items (b)–(d) are direct consequences of the definition; in (a), linearity is also clear, and closedness follows from continuity of the inner product: if \(h_n\in U^\perp\), \(h_n\to h\), then \(\left| \left\langle g, h \right\rangle \right|=\left| \left\langle g, h-h_n \right\rangle \right|\leq \left\| g\right\| \left\| h-h_n\right\| \to 0\) by the Cauchy–Bunyakovsky–Schwarz inequality, for any \(g\in U\).)
Definition 15 A subset \(U\) of a vector space is called convex if \((1-t)f+tg \in U\) for all \(f,g\in U\), \(t\in [0,1]\).
Every linear subspace is convex (check!), but not conversely (for example, a ball).
The following theorem is the technical centerpiece of this section. It is here that completeness is first genuinely used: without it (i.e., in an arbitrary inner product space that is not Hilbert), the statement is, in general, false — a closest point may fail to exist.
Theorem 14 (on projection onto a closed convex set) Let \(H\) be a Hilbert space, \(U\subseteq H\) a nonempty closed convex subset, \(f\in H\). Then there exists a unique element \(g\in U\) realizing the distance from \(f\) to \(U\): \[
\left\| f-g\right\| = \operatorname {dist}(f,U) := \inf _{u\in U}\left\| f-u\right\| .
\]
Definition 16 (orthogonal projection) Let \(U\) be a nonempty closed convex subset of a Hilbert space \(H\). The orthogonal projection of \(H\) onto \(U\) is the map \(P_U : H\to U\) that assigns to \(f\in H\) the unique (by Theorem Theorem 14) element of \(U\) closest to it.
Let \(H\) be a Hilbert space, \(U=\left\{ g\in H \; : \; \left\| g\right\| \leq 1\right\}\) the closed unit ball (it is convex and closed, so Theorem Theorem 14 applies). Find \(P_Uf\) for \(f\in H\).
If \(\left\| f\right\| \leq 1\), then \(f\in U\) and clearly the closest point is \(f\) itself: \(P_Uf=f\). If \(\left\| f\right\| >1\), then the claim is \(P_Uf = f/\left\| f\right\|\). Indeed, \(\left\| f/\left\| f\right\| \right\| =1\), so \(f/\left\| f\right\| \in U\); for any \(g\in U\) (i.e., \(\left\| g\right\| \leq 1\)) one can check (by expanding the inner products) that \(\left\| f-g\right\| \geq \left\| f\right\| - 1 = \left\| f-f/\left\| f\right\| \right\|\), i.e., \(f/\left\| f\right\|\) is indeed the closest point. This shows that \(P_U\) is defined not only on closed subspaces, but on arbitrary closed convex sets — however, from now on we will only be interested in the case of a subspace.
Let us specialize the projection theorem to the case where \(U\) is a closed linear subspace (rather than just a convex set). It is exactly this case that we will need for the Riesz theorem and (later in the probability theory course) for conditional expectation.
Theorem 15 (properties of orthogonal projection onto a subspace) Let \(U\) be a closed linear subspace of a Hilbert space \(H\), \(f\in H\). Then
\(f-P_Uf \perp g\) for every \(g\in U\);
if \(h\in U\) and \(f-h\perp g\) for every \(g\in U\), then \(h=P_Uf\);
\(P_U : H\to H\) is a linear map;
\(\left\| P_Uf\right\| \leq \left\| f\right\|\).
(Idea: (a) follows from the fact that \(P_Uf\) minimizes \(\left\| f-\cdot \right\|\) along the entire line \(P_Uf+\alpha g\subset U\), \(\alpha \in \mathbb {R}\) — a quadratic function of \(\alpha\) with a minimum at \(0\) must have zero derivative there, which gives orthogonality. (b) is a reformulation of (a): the characteristic property “\(f-h\perp U\)” uniquely determines \(h=P_Uf\). (c) linearity of \(P_U\) follows from (b), applied to \(h=P_Uf_1+\alpha P_Uf_2\). (d) the Pythagorean theorem for \(f=(f-P_Uf)+P_Uf\), with orthogonal summands by (a).)
Let \(U\) be a closed linear subspace of a Hilbert space \(H\). Prove that \(P_U\circ P_U = P_U\) (i.e., \(P_U\) is idempotent), and that \(P_Uf = f \iff f\in U\).
Now — a key geometric fact: a closed subspace and its orthogonal complement “split” the entire space.
Theorem 16 (orthogonal decomposition) Let \(U\) be a closed linear subspace of a Hilbert space \(H\). Then every element \(f\in H\) can be represented uniquely in the form \[
f = g+h, \qquad g\in U, \; \; h\in U^\perp ,
\] namely \(g=P_Uf\), \(h=f-P_Uf\). This is written briefly as \(H = U\oplus U^\perp\).
(Existence: \(f=P_Uf+(f-P_Uf)\), where the first summand is in \(U\) and the second in \(U^\perp\) by Theorem Theorem 15(a). Uniqueness: if \(f=g_1+h_1=g_2+h_2\) are two such decompositions, then \(g_1-g_2=h_2-h_1\in U\cap U^\perp =\left\{ 0\right\}\) by Lemma Lemma 4(b).)
Corollary 4 If \(U\) is a closed linear subspace of a Hilbert space \(H\), then \(U^{\perp \perp }=U\).
Let \(U\) be a nonempty closed convex subset of a Hilbert space \(H\), and let \(g\in U\) be the element of \(U\) closest to \(0\) (i.e., \(g = P_U0\)). Prove that \(\left\langle g, h \right\rangle \geq \left\| g\right\|^2\) for all \(h\in U\).
Apply the computation from the proof of Theorem Theorem 14(a) (see also the proof of Theorem Theorem 15(a)) to \(f=0\).
The Riesz representation theorem
The following result is a beautiful and practically important theorem, fully describing what bounded linear functionals on a Hilbert space look like.
Theorem 17 (F. Riesz, representation theorem) Let \(H\) be a Hilbert space, \(\varphi : H\to \mathbb {R}\) a bounded linear functional (i.e., \(\varphi\) is linear and \(\left\| \varphi \right\| := \sup \left\{ \left|\varphi (f)\right| \; : \; \left\| f\right\| \leq 1\right\} <\infty\)). Then there exists a unique \(u\in H\) such that \[
\varphi (f) = \left\langle f, u \right\rangle \qquad \forall f\in H.
\] Moreover, \(\left\| \varphi \right\| = \left\| u\right\|\).
Orthonormal systems and bases
Throughout this section we deal only with countable (finite or infinite) families — this suffices for all probabilistic applications (see the remark after the definition of an orthonormal basis).
Definition 17 A finite or countable system \((e_k)_k\) of elements of an inner product space \(V\) is called orthonormal if \[
\left\langle e_j, e_k \right\rangle = \begin{cases} 1, & j=k,\\ 0, & j\neq k. \end{cases}
\]
Examples. The standard basis vectors \(e_k=(0,\ldots ,0,1,0,\ldots )\) (a one in the \(k\)-th position) form an orthonormal system in \(\mathbb {R}^{n}\) and in \(\ell^2\).
The Gram–Schmidt process. Bessel’s inequality
Theorem 18 (Gram–Schmidt) Let \(f_1,f_2,\ldots\) be a finite or countable linearly independent system in an inner product space \(V\). Then there exists an orthonormal system \(e_1,e_2,\ldots\) (of the same length) such that for every \(n\) \[
\left\langle f_1,\ldots ,f_n\right\rangle = \left\langle e_1,\ldots ,e_n\right\rangle .
\]
Orthonormalize the system \(f_0(x)=1\), \(f_1(x)=x\) in \(L^2([-1,1])\) (with respect to Lebesgue measure) using the Gram–Schmidt process.
\(\left\| f_0\right\|^2 = \int_{-1}^1 1\; dx = 2\), so \(e_0 = f_0/\left\| f_0\right\| = 1/\sqrt2\).
Next, \(\left\langle f_1, e_0 \right\rangle = \int_{-1}^1 \frac{x}{\sqrt2}\; dx = 0\) (an odd function on a symmetric interval), so \(f_1\) is already orthogonal to \(e_0\), and it suffices to normalize it: \(\left\| f_1\right\|^2 = \int_{-1}^1x^2\; dx = \frac23\), so
\[
e_1(x) = \frac{x}{\sqrt{2/3}} = x\sqrt{\frac32}.
\]
(Continuing this process for \(f_2(x)=x^2\), \(f_3(x)=x^3,\ldots\), one obtains — up to normalization — the Legendre polynomials.)
Theorem 19 (Bessel’s inequality) Let \((e_k)_k\) be a finite or countable orthonormal system in an inner product space \(V\), \(f\in V\). Then \[
\sum _k \left| \left\langle f, e_k \right\rangle \right|^2 \leq \left\| f\right\| ^2.
\]
(Idea, for a finite index set \(1,\ldots ,n\) — the countable case follows by a limit passage: let \(g:=\sum_{k=1}^n \left\langle f, e_k \right\rangle e_k\); a direct check shows \(f-g\perp e_j\) for every \(j\), hence \(f-g\perp g\), and the Pythagorean theorem for \(f=(f-g)+g\) gives \(\left\| f\right\|^2 = \left\| f-g\right\|^2+\left\| g\right\|^2 \geq \left\| g\right\|^2 = \sum_{k=1}^n\left| \left\langle f, e_k \right\rangle \right|^2\).)
Orthonormal basis. Parseval’s identity
Definition 18 (orthonormal basis) A countable (finite or infinite) orthonormal system \((e_k)_{k=1}^\infty\) in a Hilbert space \(H\) is called an orthonormal basis of \(H\) if \[
\overline{ \left\langle e_k \; : \; k\right\rangle } = H,
\] i.e., finite linear combinations of the elements \(e_k\) are dense in \(H\).
Theorem 20 (Parseval’s identity) Let \((e_k)_{k=1}^\infty\) be an orthonormal basis of a Hilbert space \(H\), \(f,g\in H\). Then
\(f = \sum_{k=1}^\infty \left\langle f, e_k \right\rangle e_k\) (the series converges in the norm of \(H\));
\(\left\langle f, g \right\rangle = \sum_{k=1}^\infty \left\langle f, e_k \right\rangle \left\langle g, e_k \right\rangle\);
\(\left\| f\right\|^2 = \sum_{k=1}^\infty \left| \left\langle f, e_k \right\rangle \right|^2\).
(Idea: set \(S_n:=\sum_{k=1}^n \left\langle f, e_k \right\rangle e_k\). By Bessel’s inequality the series \(\sum_k\left| \left\langle f, e_k \right\rangle \right|^2\) converges, whence (by the Pythagorean theorem for tails) \((S_n)\) is a Cauchy sequence, and hence converges to some \(s\in H\) (\(H\) is complete); from the orthogonality \(f-S_n\perp e_j\) for \(n\geq j\) and continuity of the inner product we obtain \(f-s\perp e_j\) for all \(j\), i.e., \(f-s\in \left(\overline{ \left\langle e_k:k\right\rangle }\right)^\perp = H^\perp =\left\{ 0\right\}\) — this gives (a); (b),(c) follow immediately from (a) by a limit passage.)
Let \(H=\ell^2\), and let \(\varphi : \ell^2 \to \mathbb {R}\) be defined by the formula \(\varphi \left((a_1,a_2,\ldots )\right) = a_1\) (projection onto the first coordinate). Check that \(\varphi\) is a bounded linear functional, and find its representative \(u\in \ell^2\) from the Riesz theorem (Theorem Theorem 17).
Linearity is clear. Boundedness: \(\left|\varphi (a)\right| = \left|a_1\right| \leq \left\| a\right\|\) (since \(a_1^2 \leq \sum_k a_k^2 = \left\| a\right\|^2\)), so \(\left\| \varphi \right\| \leq 1\); moreover, at \(a=e_1=(1,0,0,\ldots )\) (norm \(1\)) the value is \(\varphi (e_1)=1\), hence \(\left\| \varphi \right\| =1\).
Note that \(\varphi (a) = a_1 = \left\langle a, e_1 \right\rangle\) for the standard basis vector \(e_1=(1,0,0,\ldots )\). Hence \(u=e_1\) is the desired representative (and by the theorem, the unique one), \(\left\| u\right\| = 1 = \left\| \varphi \right\|\), which agrees with the formula \(\left\| \varphi \right\| =\left\| u\right\|\).
Show that the standard basis vectors \(e_k=(0,\ldots ,0,1,0,\ldots )\) (\(k\in \mathbb {N}\)) form an orthonormal basis of \(\ell^2\) (i.e., that \(\overline{ \left\langle e_k\; :\; k\in \mathbb {N}\right\rangle }=\ell^2\)).
For \(a=(a_1,a_2,\ldots )\in \ell^2\) consider \(S_n=\sum_{k=1}^n a_ke_k\) and show that \(\left\| a-S_n\right\| \to 0\), using convergence of the series \(\sum_ka_k^2\).
Duality (a preview)
Let us close the chapter with a brief preview of one of the central facts about \(L^p\)-spaces — it will tie together everything we have discussed in this chapter.
The special case \(p=p'=2\) of this fact is the Riesz representation theorem for the Hilbert space \(L^2(\mu )\) (Theorem Theorem 17 from §3 of this same chapter), which we, unlike the general case, proved in full — using only the geometry of projections, without any Hahn–Banach. This same observation will become a key step in the proof of the Radon–Nikodym theorem in the next chapter.
References
Axler, Sheldon. 2020. Measure, Integration & Real Analysis. Springer.