Mathematics for Machine Learning · Prof. Dr. Nihat Ay
Lecture 3 Geometric structures of $\mathbb{R}^n$, Part 2
Normed space structure · metric space structure · topological space structure
Overview
Three weaker structures
In Lecture 2 we met the vector, affine and inner product space structures of $\mathbb{R}^n$. We now continue with three further structures, each weaker than the previous one.
Normed space structure
1.1 Norms and normed spaces 1.2 The $p$-norms on $\mathbb{R}^n$ 1.3 What the different norms mean for data 1.4 When does a norm come from an inner product? 1.5 Comparing norms
Metric space structure
2.1 Metrics and metric spaces 2.2 Convergence and Cauchy sequences 2.3 Completeness in $\mathbb{R}^n$ 2.4 Continuous maps
Topological space structure
3.1 Open sets and topologies 3.2 Convergence and continuity, topologically 3.3 Compactness
Convergence, completeness, continuity and compactness will accompany us throughout the course, in particular in infinite dimensions (Lecture 7).
Section 1 of 3
1
Normed space structure
1.1 Norms and normed spaces1.2 The $p$-norms on $\mathbb{R}^n$1.3 What the different norms mean for data1.4 When does a norm come from an inner product?1.5 Comparing norms
1.1 · Norms and normed spaces
Norms
The dot product defines the length, or norm, of a vector; from now on we add the index $2$:
For real $p\ge1$, the unit ball$\{\mathbf{x} : {\|\mathbf{x}\|}_p\le 1\}$ is a diamond for $p=1$, the disc for $p=2$, a square for $p=\infty$.
For $p<1$ the set is not convex and the triangle inequality fails, so ${\|\cdot\|}_p$ is not a norm.
1.2 · The $p$-norms on $\mathbb{R}^n$
Unit balls in $\mathbb{R}^3$
drag to turn
In $\mathbb{R}^3$, the unit ball $\{\mathbf{x} : {\|\mathbf{x}\|}_p\le 1\}$ is an octahedron for $p=1$, the round ball for $p=2$, a cube for $p=\infty$.
The coordinate plane $x_3=0$ cuts it in the two-dimensional unit ball (orange curve).
$0<p<1$: not a norm, ${\|\mathbf{e}_1+\mathbf{e}_2\|}_{1/2}=4>2$; the unit ball is not convex.
$\mathbf{z}$ lies on the chord from $\mathbf{x}/{\|\mathbf{x}\|}_p$ to $\mathbf{y}/{\|\mathbf{y}\|}_p$;${\|\mathbf{z}\|}_p\le1$ is the triangle inequality, so it holds if and only if the unit ball is convex.
1.3 · What the different norms mean for data
Example 1.1: peak, energy and total deviation
${\|\mathbf{x}\|}_\infty$ is the peak amplitude; it must not exceed the admissible range, to avoid clipping.
${\|\mathbf{x}\|}_2^2=\sum_i x_i^2$ is proportional to the energy of the signal.
${\|\mathbf{x}\|}_1$ measures the total absolute deviation from silence.
1.4 · When does a norm come from an inner product?
The parallelogram law
drag $\mathbf{x}$ and $\mathbf{y}$
If a norm comes from an inner product, ${\|\mathbf{x}\|}_V=\sqrt{{\langle\mathbf{x},\mathbf{x}\rangle}_V}$, it satisfies the parallelogram law:
Proof. Expand ${\|\mathbf{x}\pm\mathbf{y}\|}_V^2={\|\mathbf{x}\|}_V^2\pm2{\langle\mathbf{x},\mathbf{y}\rangle}_V+{\|\mathbf{y}\|}_V^2$ and add.□
The law involves only the norm, so it can be tested for any norm.
$\mathbf{x}=\mathbf{e}_1$,$\mathbf{y}=\mathbf{e}_2$: $8\ne4$ for $p=1$, $2\ne4$ for $p=\infty$.
Among the $p$-norms on $\mathbb{R}^n$ ($n\ge2$), only ${\|\cdot\|}_2$ satisfies the law.
1.4 · When does a norm come from an inner product?
The parallelogram law characterises inner products
Theorem 1.2 (Jordan–von Neumann). A norm ${\|\cdot\|}_V$ on a vector space $V$ comes from an inner product if and only if it satisfies the parallelogram law. The inner product is then given by the polarisation identity
A function $d_X : X\times X\to\mathbb{R}$ on a set $X$ with 1–4 is a metric; $(X,d_X)$ is a metric space; no addition or scaling is needed.
2.1 · Metrics and metric spaces
Example 2.1: three metrics on a street grid
drag $\mathbf{x}$ and $\mathbf{y}$ on the street grid; drag elsewhere to turn the city
$d_p(\mathbf{x},\mathbf{y}) := {\|\mathbf{y}-\mathbf{x}\|}_p$; $d_1$ is the Manhattan metric: the walking distance, without cutting through the blocks.
The same two points, three different distances.
2.1 · Metrics and metric spaces
Example 2.2: is a modified 7 closer to the 7 or to a 5?
“7”
“$7_{\mathrm{mod}}$”: the “7”, modified
“5”
Blur ($\sigma\le 3$) and noise ($\eta<1$): “$7_{\mathrm{mod}}$” stays closer to the “7” than the “5”, in all three metrics.
Shift by 5 to 14 pixels: the “5” is closer in $d_1$, $d_2$. Shifted out of the image, even the empty image is closer than the “5”. Pixel distances miss what a digit means.
2.1 · Metrics and metric spaces
Example 2.3: distances on the Earth
drag to turn
Points on the Earth (radius $R\approx 6371$ km) are unit vectors in $S^2:=\{\mathbf{x}\in\mathbb{R}^3 : {\|\mathbf{x}\|}_2=1\}$. Two metrics on the set $S^2$:
Euclidean (dashed, through the Earth): $d_2(\mathbf{x},\mathbf{y}) = R\,{\|\mathbf{y}-\mathbf{x}\|}_2$.
Great-circle (red, on the surface): $d_S(\mathbf{x},\mathbf{y}) = R\arccos\langle\mathbf{x},\mathbf{y}\rangle$, the length of the shortest path on the sphere, a geodesic.
$S^2$ is no vector space, since $\mathbf{x}+\mathbf{y}\notin S^2$, but a metric space.
2.1 · Metrics and metric spaces
Example 2.4: the Hamming distance between genomes
illustrative segment · mutate: one random base of either string changes · or click a base · drag the helix
DNA is a string over the four bases: $X=\{\mathrm{A},\mathrm{C},\mathrm{G},\mathrm{T}\}^n$. The Hamming distance counts the positions where two strings differ:
Triangle inequality: if $x_i\ne y_i$, then $x_i\ne z_i$ or $z_i\ne y_i$.
No addition or scaling of bases: $X$ is only a set.
Human and chimpanzee: about $1.2\%$ of the aligned positions differ: some 35 million of about 3 billion bases.
2.1 · Metrics and metric spaces
Example 2.5: the Hamburg U-Bahn as a metric space
click two stations · schematic map of the lines U1–U4
The 93 stations of the Hamburg U-Bahn form a set $X$. The graph metric counts stops:
$$d(x,y) := \text{fewest stops from } x \text{ to } y.$$
Metric: $d(x,y)=0$ only for $x=y$; trains run both ways; a ride via $z$ is a ride from $x$ to $y$.
Defined on every connected network, e.g. social networks or the internet.
2.2 · Convergence and Cauchy sequences
Converging to a limit, or to each other
$(a_k)$ converges to $a$, $a_k\to a$, if for every $\varepsilon>0$ there is $N$ with
$$d_X(a_k,a)<\varepsilon\quad\text{for all } k\ge N.$$
The limit is unique, but we must know it.
$(a_k)$ is a Cauchy sequence if its elements become close to each other: for every $\varepsilon>0$ there is $N$ with
$$d_X(a_k,a_m)<\varepsilon\quad\text{for all } k,m\ge N.$$
An intrinsic test: no limit is needed.
Convergent $\Rightarrow$ Cauchy, by the triangle inequality; the converse may fail.
2.2 · Convergence and Cauchy sequences
Example 2.6: the rational numbers have a hole
In $\mathbb{Q}$ with $d_{\mathbb{Q}}(a,b)=|b-a|$, the decimal approximations of $\sqrt2$ are a Cauchy sequence: $|a_k-a_m|\le10^{-\min\{k,m\}+1}$.
In $\mathbb{R}$ they converge to $\sqrt2$, but $\sqrt2\notin\mathbb{Q}$:no limit in $\mathbb{Q}$. The same holds for every rational sequence tending to $\sqrt2$.
A metric space $(X,d_X)$ is complete if every Cauchy sequence in $X$ converges to a limit in $X$.
In a complete space, a sequence converges if and only if it is Cauchy. $\mathbb{R}$ is complete, $\mathbb{Q}$ is not.
2.3 · Completeness in $\mathbb{R}^n$
$\mathbb{R}^n$ is complete
For the maximum norm, completeness follows from that of $\mathbb{R}$, coordinate by coordinate:
A Cauchy sequence $(\mathbf{x}^{(k)})$ in $\mathbb{R}^n$ has Cauchy coordinate sequences $(x^{(k)}_i)_k$ (the ticks on the axes).
Each converges to some $x_i\in\mathbb{R}$, and $\mathbf{x}^{(k)}\to\mathbf{x}=(x_1,\dots,x_n)^\top$.
By ${\|\mathbf{x}\|}_\infty\le{\|\mathbf{x}\|}_p\le n^{1/p}{\|\mathbf{x}\|}_\infty$, all $p$-norms have the same Cauchy and convergent sequences. All norms on $\mathbb{R}^n$ do: Theorem 3.4 (end of lecture).
Completeness may fail in infinite dimensions: $\mathbb{R}[t]$ with $\sup_{[0,1]}|f|$ is not complete (Lecture 7).
2.4 · Continuous maps
Continuity via sequences
$f : X\to Y$ between $(X,d_X)$ and $(Y,d_Y)$ is continuous at $x$ if for every sequence
$$x_k\to x \quad\Longrightarrow\quad f(x_k)\to f(x),$$
Continuous: at every point; equivalently, for every $\varepsilon>0$ there is $\delta>0$ with $d_Y(f(x),f(y))<\varepsilon$ whenever $d_X(x,y)<\delta$.
Example 2.7.
The norm is continuous:$\bigl|{\|\mathbf{x}\|}_V-{\|\mathbf{y}\|}_V\bigr|\le{\|\mathbf{x}-\mathbf{y}\|}_V$.
The distance to a set$d_X(x,A):=\inf_{a\in A}d_X(x,a)$is continuous: $|d_X(x,A)-d_X(y,A)|\le d_X(x,y)$.
2.4 · Continuous maps
Example 2.7, continued: 3. a threshold classifier
drag to turn
The McCulloch–Pitts neuron with weight vector $\mathbf{w}\ne\mathbf{0}$ and bias $b$:
with $\operatorname{sgn}(r)=1$ for $r\ge0$ and $-1$ for $r<0$. Its graph consists of two flat pieces, at heights $+1$ and $-1$, over the two half-planes:
continuous at every $\mathbf{x}$ with $\mathbf{w}^\top\mathbf{x}+b\ne0$:$f$ is constant near $\mathbf{x}$;
not continuous on the decision boundary $\mathbf{w}^\top\mathbf{x}+b=0$, a line orthogonal to $\mathbf{w}$: there the graph jumps by $2$.
Arbitrarily small perturbations of the input can change the output.
Section 3 of 3
3
Topological space structure
3.1 Open sets and topologies3.2 Convergence and continuity, topologically3.3 Compactness
3.1 · Open sets and topologies
Open balls and open sets
drag the point $\mathbf{x}$, also onto the left edge
Convergence and continuity only depend on which points are close to each other. The open ball:
$$B_r(x):=\{y : d_X(x,y)<r\}.$$
$U$ is open if every $x\in U$ has a ball $B_r(x)\subseteq U$,$r>0$.
Open balls and open intervals $(a,b)$ are open; $[a,b)$ is not: no ball around $a$ fits.
3.1 · Open sets and topologies
Topologies
The open sets of a metric space satisfy:
$\emptyset$ and $X$ are open;
unions of arbitrarily many open sets are open;
intersections of finitely many open sets are open: $B_r(x)\subseteq\bigcap_i U_i$ for $r=\min_i r_i$.
A topology on a set $X$ is a family $\mathcal{T}$ of subsets, the open sets, with these properties; $(X,\mathcal{T})$ is a topological space.
Every metric space is a topological space; different metrics may define the same topology.
On $\mathbb{R}^n$, all $p$-norms define the same open sets: by ${\|\mathbf{x}\|}_\infty\le{\|\mathbf{x}\|}_p\le n^{1/p}{\|\mathbf{x}\|}_\infty$, every ball of one contains a ball of the other with the same centre. The same holds for all norms (Theorem 3.4, end of lecture).
$C$ is closed if $X\setminus C$ is open; in a metric space, if it contains the limits of its convergent sequences.
3.2 · Convergence and continuity, topologically
Convergence and continuity through open sets
drag the ends of the interval $V$ on the vertical axis
$x_k\to x$ if every open $U\ni x$ contains all but finitely many $x_k$.
Cauchy sequences and completeness are not topological: they need a metric.
$f : X\to Y$ is continuous if preimages of open sets are open:
In metric spaces, this agrees with the sequence definition; compositions are continuous.
3.3 · Compactness
Open covers and finite subcovers
Open sets $U_i$, $i\in I$, form an open cover of $K$ if $K\subseteq\bigcup_i U_i$.
$K$ is compact if every open cover has a finite subcover$K\subseteq U_{i_1}\cup\dots\cup U_{i_m}$.
Example 3.1. Every finite set is compact.
$\mathbb{R}$ is not compact: $(-k,k)$, $k=1,2,\dots$, cover $\mathbb{R}$, but finitely many of them only cover a bounded interval.
3.3 · Compactness
Compact sets in $\mathbb{R}^n$
A subset $K$ of a normed space $V$ is bounded if ${\|\mathbf{x}\|}_V\le C$ for all $\mathbf{x}\in K$ and some constant $C$.
Theorem 3.2 (Heine–Borel). Let $\|\cdot\|$ be any norm on $\mathbb{R}^n$. A subset of $\mathbb{R}^n$ is compact if and only if it is closed and bounded with respect to $\|\cdot\|$. None of these properties depends on the choice of the norm.
We state the theorem without proof. That closedness, boundedness and compactness do not depend on the norm follows from Theorem 3.4 (end of lecture).
In metric spaces: $K$ is compact exactly when every sequence in $K$ has a subsequence converging in $K$.
In $\ell^2$, the space of real sequences $\mathbf{x}=(x_1,x_2,\dots)$ with ${\|\mathbf{x}\|}_2=\bigl(\sum_{i=1}^\infty x_i^2\bigr)^{1/2}<\infty$ (Lecture 7), the unit vectors $\mathbf{e}_i$ lie in the closed unit ball, but no subsequence is Cauchy.
3.3 · Compactness
Continuous functions attain their extremes
Theorem 3.3. If $f : X\to Y$ is continuous and $K\subseteq X$ compact, then $f(K)$ is compact. In particular, every continuous $f : K\to\mathbb{R}$ on $K\ne\emptyset$ attains its maximum and minimum.
Proof. If $V_i$ cover $f(K)$, the open sets $f^{-1}(V_i)$ cover $K$, so finitely many do, and their $V_i$ cover $f(K)$. For $Y=\mathbb{R}$: $f(K)$ is bounded, so $s=\sup f(K)$ is finite and the limit of a sequence in $f(K)$; $f(K)$ is closed, so $s\in f(K)$.□
Machine Learning: a minimiser of a loss function exists whenever the loss is continuous and the parameters range over a compact set.
3.3 · Compactness
All norms on $\mathbb{R}^n$ are equivalent
Theorem 3.4. For any two norms $\|\cdot\|$, $\|\cdot\|'$ on $\mathbb{R}^n$ there are $c,C>0$ with $c\,\|\mathbf{x}\|'\le\|\mathbf{x}\|\le C\,\|\mathbf{x}\|'$ for all $\mathbf{x}$.
Proof. It suffices to show $c{\|\mathbf{x}\|}_\infty\le\|\mathbf{x}\|\le C{\|\mathbf{x}\|}_\infty$ for every norm; applied to $\|\cdot\|$ and $\|\cdot\|'$, this gives the theorem.
Lower bound: $\|\cdot\|$ is continuous (Example 2.7), by the upper bound also with respect to ${\|\cdot\|}_\infty$. On the compact set ${S=\{{\|\mathbf{x}\|}_\infty=1\}}$ it attains a minimum $c$ (Theorem 3.3), and $c>0$ as $\mathbf{0}\notin S$. With $\mathbf{x}/{\|\mathbf{x}\|}_\infty\in S$: $\|\mathbf{x}\|\ge c{\|\mathbf{x}\|}_\infty$.□
Hence all norms on $\mathbb{R}^n$ have the same convergent sequences, open and compact sets, and $\mathbb{R}^n$ is complete for every norm.
Lecture 3
Summary
A norm assigns lengths to vectors; the $p$-norms ${\|\cdot\|}_1$, ${\|\cdot\|}_2$, ${\|\cdot\|}_\infty$ capture different aspects of data, such as peak and energy.
A norm comes from an inner product exactly when the parallelogram law holds (Theorem 1.2).
A metric measures distances on any set (genomes, U-Bahn); $\mathbb{R}^n$ is complete, $\mathbb{Q}$ is not.
Continuous maps preserve limits; a threshold classifier jumps at its decision boundary.
A topology, the family of open sets, describes convergence and continuity without distances.
In $\mathbb{R}^n$, compact means closed and bounded (Theorem 3.2); continuous functions attain their extremes on compact sets (Theorem 3.3); all norms are equivalent (Theorem 3.4).
Each structure is weaker than the previous one: inner product ${\Rightarrow}$ norm ${\Rightarrow}$ metric ${\Rightarrow}$ topology.
Next: inner products on $\mathbb{R}^n$ and projections onto subspaces (Lecture 4); infinite dimensions (Lecture 7).