Mathematics for Machine Learning · Prof. Dr. Nihat Ay

Lecture 4
Norms on function spaces, completeness and completion

Supremum and $L^p$ norms · incomplete spaces · Hilbert and Banach spaces · completion

Overview

From vectors to functions

So far we have mostly worked in finite dimensions. We now turn to infinite-dimensional spaces of functions and sequences.

  1. Norms for functions: the supremum norm and the $L^p$ norms
  2. Completeness: Hilbert and Banach spaces
    2.1 Incomplete spaces of infinite dimension
    2.2 Hilbert and Banach spaces
    2.3 Completion
Section 1 of 2
1

Norms for functions: the supremum norm and the $L^p$ norms

1 · Norms for functions: the supremum norm and the $L^p$ norms

The supremum norm

A vector $\mathbf{x}\in\mathbb{R}^n$ is a function $i\mapsto x_i$ (Lecture 2). For ${f:\Omega\to\mathbb{R}}$, the maximum becomes a supremum:

$${\|f\|}_\infty:=\sup_{t\in\Omega}|f(t)|\in[0,\infty].$$
  • Finite exactly for bounded $f$; a norm on the vector space of bounded functions, the supremum norm.
  • On a compact $\Omega$, e.g. a closed bounded interval, continuous functions are bounded: a norm on $C(\Omega)$.
  • ${\|f-g\|}_\infty\leq\varepsilon$: the graph of $g$ stays in a band of width $\varepsilon$ around the graph of $f$.
$f$ (blue), the band $f\pm\varepsilon$, and $g$ (orange): drag the top of the bump
1 · Norms for functions: the supremum norm and the $L^p$ norms

The $L^p$ norms

For a measure $\mu$ on $\Omega$ (Lecture 2), $1\leq p<\infty$ and $f$ on $\Omega$:

$${\|f\|}_p:=\Bigl(\int_\Omega|f(t)|^p\,\mu(dt)\Bigr)^{1/p}\in[0,\infty].$$
  • $L^p(\mu)$: all $f$ with ${\|f\|}_p<\infty$, modulo $\mu$-null sets; a norm by Minkowski (Lecture 3), the $L^p$ norm; for $p=2$, that of ${\langle f,g\rangle}_2$ (Lecture 2).
  • For $p\to\infty$: ${\|f\|}_p\to{\|f\|}_\infty$ if some ${\|f\|}_q<\infty$; monotone from some $p^*$ on, from below or above.
  • If $\mu(\Omega)\leq1$: increasing in $p$ (Hölder, Lecture 8); on $[0,1]$, from the mean height ${\|f\|}_1$ to the maximum.
  • Example: $f=h\,\mathbf{1}_A$ (indicator function of $A$), $\mu(A)=w$: ${\|f\|}_p=h\,w^{1/p}\to h$, increasing in $p$ if $w<1$, decreasing if $w>1$, so then from above.
$|f|$ on $[0,1]$, $\mu(\Omega)=1$; dashed: ${\|f\|}_1$ (mean height), ${\|f\|}_\infty$ (maximum)
1 · Norms for functions: the supremum norm and the $L^p$ norms

Sequence spaces

With the counting measure on $\{1,\dots,n\}$ we recover the $p$-norms on $\mathbb{R}^n$; on $\mathbb{N}$ we obtain the $p$-norms on the sequence spaces

$$\ell^p:=\left\{\mathbf{x}\in\mathbb{R}^{\mathbb{N}}:\ {\|\mathbf{x}\|}_p=\Bigl(\sum_{i=1}^\infty|x_i|^p\Bigr)^{1/p}<\infty\right\},$$
  • $p=2$: the space $\ell^2$ of Lecture 2.
  • $x_i=i^{-\alpha}$: $\sum_ii^{-\alpha p}$ is finite if and only if $\alpha p>1$.
$x_i=i^{-\alpha}$, $i=1,\dots,40$
1 · Norms for functions: the supremum norm and the $L^p$ norms

Which norms come from an inner product?

A norm ${\|\cdot\|}_V$ comes from an inner product if and only if it satisfies the parallelogram law (Theorem 1.2 of Lecture 3):

$${\|f+g\|}_V^2+{\|f-g\|}_V^2=2{\|f\|}_V^2+2{\|g\|}_V^2.$$

Among the $L^p$ norms, only $p=2$ does: for disjoint $A,B$ with $\mu(A)=\mu(B)=a>0$, the law for $\mathbf{1}_A,\mathbf{1}_B$ reads

$$2\,(2a)^{2/p}=4\,a^{2/p}\quad\Longleftrightarrow\quad p=2 .$$
  • ${\|\cdot\|}_\infty$: all norms in the law are $1$, for any $a$: $1+1\neq2+2$.
  • $p=2$, all $f,g$: ${\langle f,g\rangle}_2=\int_\Omega fg\,d\mu$, ${\|f\|}_2^2={\langle f,f\rangle}_2$.
$f$, $g$ (solid), $f+g$, $f-g$ (dashed) on $[0,1]$
1 · Norms for functions: the supremum norm and the $L^p$ norms

Norms that are not equivalent

Example 1.1. On $[0,1]$ with the Lebesgue measure, $f_k(t)=t^k$ has the norms

$${\|f_k\|}_\infty=1,\qquad{\|f_k\|}_1=\frac{1}{k+1},\qquad{\|f_k\|}_2=\frac{1}{\sqrt{2k+1}}.$$
  • One direction holds for all $f\in C([0,1])$: ${\|f\|}_1\leq{\|f\|}_2\leq{\|f\|}_\infty$, since ${\|f\|}_p$ increases in $p$ when $\mu(\Omega)\leq1$ (Section 1).
  • The other fails: ${\|f_k\|}_\infty/{\|f_k\|}_1=k+1\to\infty$, so ${\|\cdot\|}_\infty$ is not equivalent to ${\|\cdot\|}_1$ or ${\|\cdot\|}_2$, unlike norms on $\mathbb{R}^n$ (Theorem 3.4 of Lecture 3).
  • $f_k\to0$ in $L^1$ and $L^2$, not uniformly: different open sets.
  • Completeness can depend on the norm too (Section 2).
$f_k(t)=t^k$ on $[0,1]$
the three norms as functions of $k$
Section 2 of 2
2

Completeness: Hilbert and Banach spaces

2.1 Incomplete spaces of infinite dimension2.2 Hilbert and Banach spaces2.3 Completion

2.1 · Incomplete spaces of infinite dimension

Example 2.1: finitely many non-zero entries

Unlike $\mathbb{R}^n$ (Lecture 3), infinite-dimensional spaces need not be complete. Three examples:

The subspace of $\ell^2$ of all sequences with finitely many non-zero entries, with the norm of $\ell^2$:

$$\mathbf{x}^{(k)}=\bigl(1,\tfrac12,\tfrac13,\dots,\tfrac1k,0,0,\dots\bigr)^\top.$$
  1. Cauchy: ${\|\mathbf{x}^{(k)}-\mathbf{x}^{(m)}\|}_2^2=\sum_{i=m+1}^k\frac{1}{i^2}\to0$ for $k>m$.
  2. Limit outside the subspace: in $\ell^2$, $\mathbf{x}^{(k)}\to(1,\frac12,\frac13,\dots)^\top$, with infinitely many non-zero entries.
  3. Conclusion: no limit in the subspace, so it is not complete.
$\mathbf{x}^{(k)}$ (bars) and the limit $(1/i)_i$ (outlines), $i\le25$
distance to the limit as a function of $k$ (logarithmic $k$-axis)
2.1 · Incomplete spaces of infinite dimension

Example 2.1, continued: polynomials, supremum norm

$\mathbb{R}[t]$ on $[0,1]$ with the supremum norm; Taylor polynomials of $e^t$:

$$\begin{aligned}p_k(t)&=\textstyle\sum_{j=0}^k\frac{t^j}{j!},\\ {\|p_k-p_m\|}_\infty&\leq\textstyle\sum_{j=m+1}^k\frac{1}{j!}\to0\quad(k>m).\end{aligned}$$
  1. Cauchy: by the estimate above.
  2. Limit outside the space: $p_k\to e^t$ uniformly, not a polynomial.
  3. Conclusion: $\mathbb{R}[t]$ with the supremum norm is not complete.
$e^t$ (blue) and $p_k$ (orange) on $[0,1]$
${\|e^t-p_k\|}_\infty$ on a logarithmic scale
2.1 · Incomplete spaces of infinite dimension

Example 2.1, continued: continuous functions, $L^2$ norm

$C([0,1])$ with ${\|\cdot\|}_2$; continuous “ramps”:

$$f_k(t)=\begin{cases}0, & t\leq\frac12,\\ k\,(t-\frac12), & \frac12\leq t\leq\frac12+\frac1k,\\ 1, & t\geq\frac12+\frac1k.\end{cases}$$
  1. Cauchy: ${\|f_k-f_m\|}_2=\frac{k-m}{k\sqrt{3m}}\leq\frac{1}{\sqrt{3m}}\to0$ for $k>m$.
  2. Limit outside the space: ${\|f_k-s\|}_2=\frac{1}{\sqrt{3k}}\to0$ for the step function $s$ (dashed), which is not continuous, even up to null sets.
  3. Conclusion: $C([0,1])$ with ${\|\cdot\|}_2$ is not complete.

With ${\|\cdot\|}_\infty$, $C([0,1])$ is complete (uniform limits).

$f_k$ (orange), $f_{2k}$ (grey), the step function $s$ (dashed)
2.2 · Hilbert and Banach spaces

Complete spaces

Definition 2.2. A Banach space is a normed space $(V,{\|\cdot\|}_V)$ that is complete with respect to the metric $d(\mathbf{x},\mathbf{y})={\|\mathbf{y}-\mathbf{x}\|}_V$ of its norm. A Hilbert space is an inner product space $(V,{\langle\cdot,\cdot\rangle}_V)$ that is complete with respect to the norm ${\|\mathbf{x}\|}_V=\sqrt{{\langle\mathbf{x},\mathbf{x}\rangle}_V}$ of its inner product.

Every Hilbert space is a Banach space. The examples of Lectures 2, 3 and 4:

  • Hilbert spaces: $\mathbb{R}^n$ with the dot product; $\ell^2$; $L^2(\mu)$.
  • Banach, not Hilbert: $\mathbb{R}^n$ ($n\geq2$) with ${\|\cdot\|}_1$, ${\|\cdot\|}_\infty$; $\ell^p$, $L^p(\mu)$ ($p\neq2$); $C([0,1])$ with ${\|\cdot\|}_\infty$.
  • Neither: $C([0,1])$ with ${\|\cdot\|}_2$; $\mathbb{R}[t]$ with ${\|\cdot\|}_\infty$; finitely many non-zero entries, ${\|\cdot\|}_2$.

The norms in the second item violate the parallelogram law (Theorem 1.2 of Lecture 3; Section 1). The completeness of $\ell^p$ and $L^p(\mu)$ is the Riesz–Fischer theorem (without proof).

2.2 · Hilbert and Banach spaces

Closed subspaces

A linear subspace $U$ of a normed space $X$ is closed if it contains the limits of its convergent sequences:

$$\mathbf{u}_k\in U,\ \ \mathbf{u}_k\to\mathbf{x}\in X\quad\Longrightarrow\quad\mathbf{x}\in U.$$

Theorem 2.3. Let $U\subseteq X$ be a linear subspace, with the norm (and inner product) of $X$.

  1. If $X$ is a Banach space, then $U$ is a Banach space if and only if $U$ is closed.
  2. If $X$ is a Hilbert space, then $U$ is a Hilbert space if and only if $U$ is closed.

Proof. 1. Closed $\Rightarrow$ complete: a Cauchy sequence in $U$ converges in $X$, with limit in $U$. Complete $\Rightarrow$ closed: $\mathbf{u}_k\to\mathbf{x}$ is Cauchy, so $\mathbf{u}_k\to\mathbf{u}\in U$, and $\mathbf{x}=\mathbf{u}$. 2. The restricted inner product induces the restricted norm; apply 1.□

  • Closed: every finite-dimensional subspace of a Banach space, since it is complete (as $\mathbb{R}^n$ is).
  • Not closed: the sequences in $\ell^2$ with finitely many non-zero entries; the continuous functions in $L^2([0,1])$.

Later: projections onto closed subspaces (Lecture 5), Moore–Penrose inverse (Lecture 12).

2.3 · Completion

Completions

The incomplete spaces of Section 2.1 have “holes”, as $\mathbb{Q}$ has a hole at $\sqrt2$. A completion fills them:

Definition 2.4 (Completion). A completion of a normed space $X$ is a Banach space $Y$ with a linear map $\iota:X\to Y$ that preserves norms and has dense image $\iota(X)$. For an inner product space $X$, $Y$ is a Hilbert space and $\iota$ preserves inner products.

  1. Existence: every normed space has a completion, built from classes of Cauchy sequences (Theorem 2.5).
  2. Uniqueness: two completions are identified by exactly one isometric isomorphism compatible with their maps (Proposition 2.6).
  3. Closures: if $X$ lies in a Banach space $Z$, its closure in $Z$, with the inclusion, is a completion (Proposition 2.7).
2.3 · Completion

Filling the holes

Existence. $\mathbb{R}$ fills the hole of $\mathbb{Q}$ at $\sqrt2$ with Cauchy sequences, and the same works for every normed space $X$: a Cauchy sequence without a limit in $X$ defines a new element, its “ideal limit”, and two Cauchy sequences define the same element if they come arbitrarily close:

$$(\mathbf{x}_k)\sim(\mathbf{y}_k)\quad:\Longleftrightarrow\quad{\|\mathbf{x}_k-\mathbf{y}_k\|}_X\to0 .$$
$a_k=\lfloor k\sqrt2\rfloor/k$ (orange) and $b_k=a_k+\frac1k$ (blue); dashed: $\sqrt2\notin\mathbb{Q}$
their distance $|a_k-b_k|=\frac1k$
2.3 · Completion

Construction by Cauchy sequences

Let $X$ be a normed space, possibly an inner product space. The equivalence class of a Cauchy sequence $(\mathbf{x}_k)$ is denoted by $[\mathbf{x}_k]$, the set of all classes by $\widehat{X}$. On $\widehat{X}$ we define

$$[\mathbf{x}_k]+[\mathbf{y}_k]:=[\mathbf{x}_k+\mathbf{y}_k],\qquad\lambda[\mathbf{x}_k]:=[\lambda\mathbf{x}_k],\qquad{\|[\mathbf{x}_k]\|}_{\widehat{X}}:=\lim_{k\to\infty}{\|\mathbf{x}_k\|}_X,$$

and, if $X$ is an inner product space,

$${\langle[\mathbf{x}_k],[\mathbf{y}_k]\rangle}_{\widehat{X}}:=\lim_{k\to\infty}{\langle\mathbf{x}_k,\mathbf{y}_k\rangle}_X.$$
  • The definitions make sense: the sum of two Cauchy sequences is a Cauchy sequence, by the triangle inequality, and equivalent sequences have equivalent sums; similarly for multiples.
  • The norm exists: $\bigl|{\|\mathbf{x}_k\|}_X-{\|\mathbf{x}_m\|}_X\bigr|\leq{\|\mathbf{x}_k-\mathbf{x}_m\|}_X$, so $({\|\mathbf{x}_k\|}_X)$ is a Cauchy sequence of real numbers, which converges since $\mathbb{R}$ is complete; equivalent sequences have the same limit.
  • The inner product exists: by the Cauchy–Schwarz inequality, $({\langle\mathbf{x}_k,\mathbf{y}_k\rangle}_X)$ is a Cauchy sequence of real numbers, and its limit does not depend on the representatives (proof below).
  • $X$ sits inside: every $\mathbf{x}\in X$ defines the class $J(\mathbf{x}):=[\mathbf{x},\mathbf{x},\mathbf{x},\dots]$ of the constant sequence.
2.3 · Completion

Existence of a completion

Theorem 2.5 (Existence of a completion). Let $X$ be a normed space.

  1. $\widehat{X}$, with these operations and this norm, is a Banach space; $J:X\to\widehat{X}$ is linear, preserves norms, and $J(X)$ is dense in $\widehat{X}$. So $(\widehat{X},J)$ is a completion of $X$.
  2. If $X$ is an inner product space, then ${\langle\cdot,\cdot\rangle}_{\widehat{X}}$ is an inner product with norm ${\|\cdot\|}_{\widehat{X}}$: $\widehat{X}$ is a Hilbert space, and $(\widehat{X},J)$ is a completion of the inner product space $X$.

Proof. The vector space axioms and norm properties follow by taking limits; $\lim_k{\|\mathbf{x}_k\|}_X=0$ means $(\mathbf{x}_k)\sim(0,0,\dots)$. Density: for $\xi=[\mathbf{x}_k]$, ${\|\xi-J(\mathbf{x}_m)\|}_{\widehat{X}}=\lim_k{\|\mathbf{x}_k-\mathbf{x}_m\|}_X\to0$ for $m\to\infty$. Completeness: for a Cauchy sequence $(\xi_j)$ in $\widehat{X}$, choose $\mathbf{x}_j\in X$ with ${\|\xi_j-J(\mathbf{x}_j)\|}_{\widehat{X}}<1/j$. Then

$${\|\mathbf{x}_j-\mathbf{x}_l\|}_X={\|J(\mathbf{x}_j)-J(\mathbf{x}_l)\|}_{\widehat{X}}\leq\tfrac1j+{\|\xi_j-\xi_l\|}_{\widehat{X}}+\tfrac1l,$$

so $(\mathbf{x}_j)$ is Cauchy and defines $\xi:=[\mathbf{x}_j]$, with $J(\mathbf{x}_j)\to\xi$ and hence ${\|\xi_j-\xi\|}_{\widehat{X}}\leq1/j+{\|J(\mathbf{x}_j)-\xi\|}_{\widehat{X}}\to0$.

2.3 · Completion

Inner products survive in $\widehat{X}$

Proof, continued. Cauchy sequences are bounded, say ${\|\mathbf{x}_k\|}_X,{\|\mathbf{y}_k\|}_X\leq C$, and by the Cauchy–Schwarz inequality

$$|{\langle\mathbf{x}_k,\mathbf{y}_k\rangle}_X-{\langle\mathbf{x}_m,\mathbf{y}_m\rangle}_X|\leq{\|\mathbf{x}_k-\mathbf{x}_m\|}_X\,{\|\mathbf{y}_k\|}_X+{\|\mathbf{x}_m\|}_X\,{\|\mathbf{y}_k-\mathbf{y}_m\|}_X\leq C\bigl({\|\mathbf{x}_k-\mathbf{x}_m\|}_X+{\|\mathbf{y}_k-\mathbf{y}_m\|}_X\bigr).$$

Hence $({\langle\mathbf{x}_k,\mathbf{y}_k\rangle}_X)$ is a Cauchy sequence of real numbers and converges, and the same estimate shows that the limit does not depend on the representatives. Bilinearity, symmetry and $\lim_k{\langle\mathbf{x}_k,\mathbf{x}_k\rangle}_X=\lim_k{\|\mathbf{x}_k\|}_X^2={\|[\mathbf{x}_k]\|}_{\widehat{X}}^2$ follow by taking limits.□

  • The constant sequences show that $J$ preserves inner products.
  • Density of $J(X)$: every element of $\widehat{X}$ is approximated by elements of $X$, as every real number by rational numbers.
2.3 · Completion

Uniqueness of completions

Proposition 2.6 (Uniqueness of completions). For any two completions $(Y_1,\iota_1)$ and $(Y_2,\iota_2)$ of $X$, there is exactly one isometric isomorphism $\Phi:Y_1\to Y_2$ with $\Phi\circ\iota_1=\iota_2$. If $Y_1$ and $Y_2$ are Hilbert spaces, $\Phi$ preserves inner products.

Proof. For $\mathbf{y}\in Y_1$, choose $\mathbf{x}_k\in X$ with $\iota_1(\mathbf{x}_k)\to\mathbf{y}$; $(\mathbf{x}_k)$ is Cauchy, so $\Phi(\mathbf{y}):=\lim_k\iota_2(\mathbf{x}_k)$ exists in $Y_2$. $\Phi$ is well defined, linear, preserves norms, and $\Phi\circ\iota_1=\iota_2$; its image is complete, hence closed (Theorem 2.3), and contains the dense set $\iota_2(X)$: $\Phi$ is onto. Two such maps agree on the dense set $\iota_1(X)$, hence everywhere. Inner products: polarization, ${\langle\mathbf{y},\mathbf{z}\rangle}_{Y_i}=\frac14({\|\mathbf{y}+\mathbf{z}\|}_{Y_i}^2-{\|\mathbf{y}-\mathbf{z}\|}_{Y_i}^2)$.□

The proposition does not say that two completions are equal; in general this makes no sense: $\widehat{X}$ consists of classes of Cauchy sequences, $\ell^2$ of sequences.

2.3 · Completion

Closures are completions

Proposition 2.7 (Closures are completions). Let $Z$ be a Banach space and $X\subseteq Z$ a linear subspace, with the norm of $Z$. Then the closure $\overline{X}$ of $X$ in $Z$, with the inclusion $X\hookrightarrow\overline{X}$, is a completion of $X$; if $Z$ is a Hilbert space, of the inner product space $X$.

Proof. $\overline{X}$ is a linear subspace, since limits of sums and multiples are sums and multiples of limits. It is closed, hence a Banach space, and a Hilbert space if $Z$ is one (Theorem 2.3). The inclusion preserves norms and inner products, and its image $X$ is dense in $\overline{X}$.□

  • Two natural completions of $X\subseteq Z$: $(\widehat{X},J)$, from classes of Cauchy sequences, and $\overline{X}$ with the inclusion, from the ambient space $Z$. Proposition 2.6 identifies them: $[\mathbf{x}_k]\mapsto\lim_k\mathbf{x}_k$ in $Z$.
  • Two completions by subspaces of the same $Z$ with the inclusions are equal: both are closed in $Z$ with $X$ dense, so both are $\overline{X}$.
2.3 · Completion

Example 2.8: completions as closures

  • Rational numbers $\mathbb{Q}$: dense in $\mathbb{R}$; the construction of $\widehat{\mathbb{Q}}$ is one of the standard ways of defining $\mathbb{R}$, due to Cantor.
  • Sequences with finitely many non-zero entries with ${\|\cdot\|}_2$: dense in $\ell^2$.
  • Polynomials on $[0,1]$ with ${\|\cdot\|}_\infty$: dense in $C([0,1])$. Every continuous function is a uniform limit of polynomials, the Weierstrass approximation theorem (without proof). Right: Bernstein polynomials.
  • $C([0,1])$ with ${\|\cdot\|}_2$: dense in $L^2([0,1])$; this needs measure theory (without proof).
a continuous $f$ (blue) and a polynomial of degree $n$ (orange)
Lecture 4

How the spaces are related

Affine spaces
e.g. the plane $x_1+x_2+x_3=1$ in $\mathbb{R}^3$
Vector spaces
e.g. all real sequences, all functions $[0,1]\to\mathbb{R}$
Normed spaces
e.g. polynomials on $[0,1]$ with ${\|\cdot\|}_\infty$
Inner product
spaces
$C([0,1])$, ${\|\cdot\|}_2$
Hilbert spaces
$\mathbb{R}^n$, $\ell^2$, $L^2(\mu)$
Banach spaces
$C([0,1])$, ${\|\cdot\|}_\infty$
$\ell^p$, $L^p(\mu)$, $p\neq2$
Metric spaces
e.g. the sphere with the great-circle distance, $\mathbb{Z}$ with $|m-n|$
Topological spaces

A norm gives a metric, $d(\mathbf{x},\mathbf{y})={\|\mathbf{x}-\mathbf{y}\|}_V$, and a metric gives a topology. Completion: normed → Banach, inner product → Hilbert.

Lecture 4

Summary

  • Supremum norm on the bounded functions, $L^p$ norm on $L^p(\mu)$, $p$-norms on $\ell^p$.
  • By the parallelogram law, only the $L^2$ norm comes from an inner product, ${\langle f,g\rangle}_2=\int_\Omega fg\,d\mu$.
  • On $C([0,1])$, ${\|\cdot\|}_\infty$ is not equivalent to ${\|\cdot\|}_1$, ${\|\cdot\|}_2$ (unlike on $\mathbb{R}^n$): $t^k\to0$ in $L^1$, $L^2$, not uniformly.
  • Completeness depends on the norm. Banach and Hilbert spaces are the complete normed and inner product spaces; their closed subspaces are again complete.
  • Completion: a dense, norm-preserving linear map $\iota:X\to Y$ into a Banach (Hilbert) space; it exists and is unique up to isometric isomorphism.
  • Closures are completions: ${\mathbb{Q}\subseteq\mathbb{R}}$, polynomials in $C([0,1])$, ${C([0,1])\subseteq L^2}$.

Next: inner products on $\mathbb{R}^n$ and projections onto subspaces (Lecture 5).