Mathematics for Machine Learning · Prof. Dr. Nihat Ay

Lecture 3
Geometric structures of $\mathbb{R}^n$, Part 2

Normed space structure · metric space structure · topological space structure

Overview

Three weaker structures

In Lecture 2 we met the vector, affine and inner product space structures of $\mathbb{R}^n$. We now continue with three further structures, each weaker than the previous one.

  1. Normed space structure
    1.1 Norms and normed spaces
    1.2 The $p$-norms on $\mathbb{R}^n$
    1.3 What the different norms mean for data
    1.4 When does a norm come from an inner product?
    1.5 Comparing norms
  2. Metric space structure
    2.1 Metrics and metric spaces
    2.2 Convergence and Cauchy sequences
    2.3 Completeness in $\mathbb{R}^n$
    2.4 Continuous maps
  3. Topological space structure
    3.1 Open sets and topologies
    3.2 Convergence and continuity, topologically
    3.3 Compactness

Convergence, completeness, continuity and compactness will accompany us throughout the course, in particular in infinite dimensions (Lecture 7).

Section 1 of 3
1

Normed space structure

1.1 Norms and normed spaces1.2 The $p$-norms on $\mathbb{R}^n$1.3 What the different norms mean for data1.4 When does a norm come from an inner product?1.5 Comparing norms

1.1 · Norms and normed spaces

Norms

The dot product defines the length, or norm, of a vector; from now on we add the index $2$:

$${\|\mathbf{x}\|}_2 := \sqrt{\langle\mathbf{x},\mathbf{x}\rangle} = \sqrt{\textstyle\sum_{i=1}^n x_i^2}.$$

For all $\mathbf{x},\mathbf{y}\in\mathbb{R}^n$ and $\lambda\in\mathbb{R}$:

  1. Non-negativity: ${\|\mathbf{x}\|}_2\ge 0$
  2. Positive definiteness: ${\|\mathbf{x}\|}_2=0$ if and only if $\mathbf{x}=0$
  3. Absolute homogeneity: ${\|\lambda\mathbf{x}\|}_2=|\lambda|\,{\|\mathbf{x}\|}_2$
  4. Triangle inequality: ${\|\mathbf{x}+\mathbf{y}\|}_2\le{\|\mathbf{x}\|}_2+{\|\mathbf{y}\|}_2$
  • A function ${\|\cdot\|}_V : V\to\mathbb{R}$ with 1–4 is a norm; $(V,{\|\cdot\|}_V)$ is a normed space.
  • Every inner product defines a norm, ${\|\mathbf{x}\|}_V:=\sqrt{{\langle\mathbf{x},\mathbf{x}\rangle}_V}$, e.g. ${\|\cdot\|}_G$ (Lecture 4).
  • Not every norm comes from an inner product.
1.2 · The $p$-norms on $\mathbb{R}^n$

Unit balls of the $p$-norms

$${\|\mathbf{x}\|}_p := \Bigl(\sum_{i=1}^n |x_i|^p\Bigr)^{1/p}, \qquad {\|\mathbf{x}\|}_\infty := \max_i |x_i|$$

For real $p\ge1$, the unit ball $\{\mathbf{x} : {\|\mathbf{x}\|}_p\le 1\}$ is a diamond for $p=1$, the disc for $p=2$, a square for $p=\infty$.

For $p<1$ the set is not convex and the triangle inequality fails, so ${\|\cdot\|}_p$ is not a norm.

1.2 · The $p$-norms on $\mathbb{R}^n$

Unit balls in $\mathbb{R}^3$

drag to turn

In $\mathbb{R}^3$, the unit ball $\{\mathbf{x} : {\|\mathbf{x}\|}_p\le 1\}$ is an octahedron for $p=1$, the round ball for $p=2$, a cube for $p=\infty$.

The coordinate plane $x_3=0$ cuts it in the two-dimensional unit ball (orange curve).

For $p<1$ the set is again not convex.

1.2 · The $p$-norms on $\mathbb{R}^n$

The triangle inequality

drag $\mathbf{x},\mathbf{y}$; $\mathbf{z}:=(\mathbf{x}+\mathbf{y})/({\|\mathbf{x}\|}_p+{\|\mathbf{y}\|}_p)$
  • $p=1,\infty$: sum or maximise $|x_i+y_i|\le|x_i|+|y_i|$ over $i$.
  • Real $p\ge1$: the Minkowski inequality (proof in Lecture 8):
$$\Bigl(\sum_i |x_i+y_i|^p\Bigr)^{\!1/p} \le \Bigl(\sum_i |x_i|^p\Bigr)^{\!1/p} + \Bigl(\sum_i |y_i|^p\Bigr)^{\!1/p}$$
  • $0<p<1$: not a norm, ${\|\mathbf{e}_1+\mathbf{e}_2\|}_{1/2}=4>2$; the unit ball is not convex.
  • $\mathbf{z}$ lies on the chord from $\mathbf{x}/{\|\mathbf{x}\|}_p$ to $\mathbf{y}/{\|\mathbf{y}\|}_p$; ${\|\mathbf{z}\|}_p\le1$ is the triangle inequality, so it holds if and only if the unit ball is convex.
1.3 · What the different norms mean for data

Example 1.1: peak, energy and total deviation

  • ${\|\mathbf{x}\|}_\infty$ is the peak amplitude; it must not exceed the admissible range, to avoid clipping.
  • ${\|\mathbf{x}\|}_2^2=\sum_i x_i^2$ is proportional to the energy of the signal.
  • ${\|\mathbf{x}\|}_1$ measures the total absolute deviation from silence.
1.4 · When does a norm come from an inner product?

The parallelogram law

drag $\mathbf{x}$ and $\mathbf{y}$

If a norm comes from an inner product, ${\|\mathbf{x}\|}_V=\sqrt{{\langle\mathbf{x},\mathbf{x}\rangle}_V}$, it satisfies the parallelogram law:

$${\|\mathbf{x}+\mathbf{y}\|}_V^2+{\|\mathbf{x}-\mathbf{y}\|}_V^2 = 2{\|\mathbf{x}\|}_V^2+2{\|\mathbf{y}\|}_V^2$$

Proof. Expand ${\|\mathbf{x}\pm\mathbf{y}\|}_V^2={\|\mathbf{x}\|}_V^2\pm2{\langle\mathbf{x},\mathbf{y}\rangle}_V+{\|\mathbf{y}\|}_V^2$ and add.□

  • The law involves only the norm, so it can be tested for any norm.
  • $\mathbf{x}=\mathbf{e}_1$, $\mathbf{y}=\mathbf{e}_2$: $8\ne4$ for $p=1$, $2\ne4$ for $p=\infty$.
  • Among the $p$-norms on $\mathbb{R}^n$ ($n\ge2$), only ${\|\cdot\|}_2$ satisfies the law.
1.4 · When does a norm come from an inner product?

The parallelogram law characterises inner products

Theorem 1.2 (Jordan–von Neumann). A norm ${\|\cdot\|}_V$ on a vector space $V$ comes from an inner product if and only if it satisfies the parallelogram law. The inner product is then given by the polarisation identity

$${\langle\mathbf{x},\mathbf{y}\rangle}_V=\tfrac14\bigl({\|\mathbf{x}+\mathbf{y}\|}_V^2-{\|\mathbf{x}-\mathbf{y}\|}_V^2\bigr).$$

Proof of necessity. See the proof on the previous slide.□

  • The converse is stated without proof; it holds in every normed space, also in infinite dimensions (Lecture 7).
  • ${\|\cdot\|}_1$ and ${\|\cdot\|}_\infty$ do not come from an inner product.
  • Normed spaces carry length, but in general no angle or orthogonality.
1.5 · Comparing norms

Norms on $\mathbb{R}^n$ differ at most by a constant factor

$${\|\mathbf{x}\|}_\infty \;\le\; {\|\mathbf{x}\|}_p \;\le\; n^{1/p}\,{\|\mathbf{x}\|}_\infty, \qquad 1\le p<\infty$$

Proof. If $|x_k|={\|\mathbf{x}\|}_\infty$, then $|x_k|^p\le\sum_i|x_i|^p\le n|x_k|^p$.□

unit balls, $n=2$
$\mathbf{x}=(0.8,\,0.45)^\top$: $p\mapsto{\|\mathbf{x}\|}_p$ tends to ${\|\mathbf{x}\|}_\infty$
  • Sharp: ${\|(1,\dots,1)^\top\|}_p=n^{1/p}\,{\|(1,\dots,1)^\top\|}_\infty$, so the factor is unbounded as $n\to\infty$ (Lecture 7).
Section 2 of 3
2

Metric space structure

2.1 Metrics and metric spaces2.2 Convergence and Cauchy sequences2.3 Completeness in $\mathbb{R}^n$2.4 Continuous maps

2.1 · Metrics and metric spaces

Metrics

drag $\mathbf{x}$, $\mathbf{y}$, $\mathbf{z}$ · $d$ from ${\|\cdot\|}_2$ · dashed: detour via $\mathbf{z}$

A norm ${\|\cdot\|}_V$, e.g. ${\|\cdot\|}_p$, defines $d(\mathbf{x},\mathbf{y}) := {\|\mathbf{y}-\mathbf{x}\|}_V$ with:

  1. $d(\mathbf{x},\mathbf{x})=0$,
  2. Positivity: $d(\mathbf{x},\mathbf{y})>0$ for $\mathbf{x}\ne\mathbf{y}$,
  3. Symmetry: $d(\mathbf{x},\mathbf{y})=d(\mathbf{y},\mathbf{x})$,
  4. Triangle inequality: $d(\mathbf{x},\mathbf{y})\le d(\mathbf{x},\mathbf{z})+d(\mathbf{z},\mathbf{y})$.
  • A function $d_X : X\times X\to\mathbb{R}$ on a set $X$ with 1–4 is a metric; $(X,d_X)$ is a metric space; no addition or scaling is needed.
2.1 · Metrics and metric spaces

Example 2.1: three metrics on a street grid

drag $\mathbf{x}$ and $\mathbf{y}$ on the street grid; drag elsewhere to turn the city
  • $d_p(\mathbf{x},\mathbf{y}) := {\|\mathbf{y}-\mathbf{x}\|}_p$; $d_1$ is the Manhattan metric: the walking distance, without cutting through the blocks.
  • The same two points, three different distances.
2.1 · Metrics and metric spaces

Example 2.2: is a modified 7 closer to the 7 or to a 5?

“7”
“$7_{\mathrm{mod}}$”: the “7”, modified
“5”
  • Blur ($\sigma\le 3$) and noise ($\eta<1$): “$7_{\mathrm{mod}}$” stays closer to the “7” than the “5”, in all three metrics.
  • Shift by 5 to 14 pixels: the “5” is closer in $d_1$, $d_2$. Shifted out of the image, even the empty image is closer than the “5”. Pixel distances miss what a digit means.
2.1 · Metrics and metric spaces

Example 2.3: distances on the Earth

drag to turn

Points on the Earth (radius $R\approx 6371$ km) are unit vectors in $S^2:=\{\mathbf{x}\in\mathbb{R}^3 : {\|\mathbf{x}\|}_2=1\}$. Two metrics on the set $S^2$:

  • Euclidean (dashed, through the Earth): $d_2(\mathbf{x},\mathbf{y}) = R\,{\|\mathbf{y}-\mathbf{x}\|}_2$.
  • Great-circle (red, on the surface): $d_S(\mathbf{x},\mathbf{y}) = R\arccos\langle\mathbf{x},\mathbf{y}\rangle$, the length of the shortest path on the sphere, a geodesic.
  • $S^2$ is no vector space, since $\mathbf{x}+\mathbf{y}\notin S^2$, but a metric space.
2.1 · Metrics and metric spaces

Example 2.4: the Hamming distance between genomes

illustrative segment · mutate: one random base of either string changes · or click a base · drag the helix

DNA is a string over the four bases: $X=\{\mathrm{A},\mathrm{C},\mathrm{G},\mathrm{T}\}^n$. The Hamming distance counts the positions where two strings differ:

$$d_H(\mathbf{x},\mathbf{y}) := \#\{i : x_i\ne y_i\}.$$
  • Triangle inequality: if $x_i\ne y_i$, then $x_i\ne z_i$ or $z_i\ne y_i$.
  • No addition or scaling of bases: $X$ is only a set.
  • Human and chimpanzee: about $1.2\%$ of the aligned positions differ: some 35 million of about 3 billion bases.
2.1 · Metrics and metric spaces

Example 2.5: the Hamburg U-Bahn as a metric space

click two stations · schematic map of the lines U1–U4

The 93 stations of the Hamburg U-Bahn form a set $X$. The graph metric counts stops:

$$d(x,y) := \text{fewest stops from } x \text{ to } y.$$
  • Metric: $d(x,y)=0$ only for $x=y$; trains run both ways; a ride via $z$ is a ride from $x$ to $y$.
  • Defined on every connected network, e.g. social networks or the internet.
2.2 · Convergence and Cauchy sequences

Converging to a limit, or to each other

  • $(a_k)$ converges to $a$, $a_k\to a$, if for every $\varepsilon>0$ there is $N$ with
$$d_X(a_k,a)<\varepsilon\quad\text{for all } k\ge N.$$
  • The limit is unique, but we must know it.
  • $(a_k)$ is a Cauchy sequence if its elements become close to each other: for every $\varepsilon>0$ there is $N$ with
$$d_X(a_k,a_m)<\varepsilon\quad\text{for all } k,m\ge N.$$
  • An intrinsic test: no limit is needed.
  • Convergent $\Rightarrow$ Cauchy, by the triangle inequality; the converse may fail.
2.2 · Convergence and Cauchy sequences

Example 2.6: the rational numbers have a hole

  • In $\mathbb{Q}$ with $d_{\mathbb{Q}}(a,b)=|b-a|$, the decimal approximations of $\sqrt2$ are a Cauchy sequence: $|a_k-a_m|\le10^{-\min\{k,m\}+1}$.
  • In $\mathbb{R}$ they converge to $\sqrt2$, but $\sqrt2\notin\mathbb{Q}$: no limit in $\mathbb{Q}$. The same holds for every rational sequence tending to $\sqrt2$.

A metric space $(X,d_X)$ is complete if every Cauchy sequence in $X$ converges to a limit in $X$.

In a complete space, a sequence converges if and only if it is Cauchy. $\mathbb{R}$ is complete, $\mathbb{Q}$ is not.

2.3 · Completeness in $\mathbb{R}^n$

$\mathbb{R}^n$ is complete

For the maximum norm, completeness follows from that of $\mathbb{R}$, coordinate by coordinate:

$$|x^{(k)}_i-x^{(m)}_i| \le {\|\mathbf{x}^{(k)}-\mathbf{x}^{(m)}\|}_\infty .$$
  • A Cauchy sequence $(\mathbf{x}^{(k)})$ in $\mathbb{R}^n$ has Cauchy coordinate sequences $(x^{(k)}_i)_k$ (the ticks on the axes).
  • Each converges to some $x_i\in\mathbb{R}$, and $\mathbf{x}^{(k)}\to\mathbf{x}=(x_1,\dots,x_n)^\top$.
  • By ${\|\mathbf{x}\|}_\infty\le{\|\mathbf{x}\|}_p\le n^{1/p}{\|\mathbf{x}\|}_\infty$, all $p$-norms have the same Cauchy and convergent sequences. All norms on $\mathbb{R}^n$ do: Theorem 3.4 (end of lecture).
  • Completeness may fail in infinite dimensions: $\mathbb{R}[t]$ with $\sup_{[0,1]}|f|$ is not complete (Lecture 7).
2.4 · Continuous maps

Continuity via sequences

$f : X\to Y$ between $(X,d_X)$ and $(Y,d_Y)$ is continuous at $x$ if for every sequence

$$x_k\to x \quad\Longrightarrow\quad f(x_k)\to f(x),$$
  • Continuous: at every point; equivalently, for every $\varepsilon>0$ there is $\delta>0$ with $d_Y(f(x),f(y))<\varepsilon$ whenever $d_X(x,y)<\delta$.

Example 2.7.

  1. The norm is continuous: $\bigl|{\|\mathbf{x}\|}_V-{\|\mathbf{y}\|}_V\bigr|\le{\|\mathbf{x}-\mathbf{y}\|}_V$.
  2. The distance to a set $d_X(x,A):=\inf_{a\in A}d_X(x,a)$ is continuous: $|d_X(x,A)-d_X(y,A)|\le d_X(x,y)$.
2.4 · Continuous maps

Example 2.7, continued: 3. a threshold classifier

drag to turn

The McCulloch–Pitts neuron with weight vector $\mathbf{w}\ne\mathbf{0}$ and bias $b$:

$$f(\mathbf{x};\mathbf{w},b)=\operatorname{sgn}(\mathbf{w}^\top\mathbf{x}+b),$$

with $\operatorname{sgn}(r)=1$ for $r\ge0$ and $-1$ for $r<0$. Its graph consists of two flat pieces, at heights $+1$ and $-1$, over the two half-planes:

  • continuous at every $\mathbf{x}$ with $\mathbf{w}^\top\mathbf{x}+b\ne0$: $f$ is constant near $\mathbf{x}$;
  • not continuous on the decision boundary $\mathbf{w}^\top\mathbf{x}+b=0$, a line orthogonal to $\mathbf{w}$: there the graph jumps by $2$.

Arbitrarily small perturbations of the input can change the output.

Section 3 of 3
3

Topological space structure

3.1 Open sets and topologies3.2 Convergence and continuity, topologically3.3 Compactness

3.1 · Open sets and topologies

Open balls and open sets

drag the point $\mathbf{x}$, also onto the left edge

Convergence and continuity only depend on which points are close to each other. The open ball:

$$B_r(x):=\{y : d_X(x,y)<r\}.$$
  • $U$ is open if every $x\in U$ has a ball $B_r(x)\subseteq U$, $r>0$.
  • Open balls and open intervals $(a,b)$ are open; $[a,b)$ is not: no ball around $a$ fits.
3.1 · Open sets and topologies

Topologies

The open sets of a metric space satisfy:

  1. $\emptyset$ and $X$ are open;
  2. unions of arbitrarily many open sets are open;
  3. intersections of finitely many open sets are open: $B_r(x)\subseteq\bigcap_i U_i$ for $r=\min_i r_i$.

A topology on a set $X$ is a family $\mathcal{T}$ of subsets, the open sets, with these properties; $(X,\mathcal{T})$ is a topological space.

  • Every metric space is a topological space; different metrics may define the same topology.
  • On $\mathbb{R}^n$, all $p$-norms define the same open sets: by ${\|\mathbf{x}\|}_\infty\le{\|\mathbf{x}\|}_p\le n^{1/p}{\|\mathbf{x}\|}_\infty$, every ball of one contains a ball of the other with the same centre. The same holds for all norms (Theorem 3.4, end of lecture).
  • $C$ is closed if $X\setminus C$ is open; in a metric space, if it contains the limits of its convergent sequences.
3.2 · Convergence and continuity, topologically

Convergence and continuity through open sets

drag the ends of the interval $V$ on the vertical axis
  • $x_k\to x$ if every open $U\ni x$ contains all but finitely many $x_k$.
  • Cauchy sequences and completeness are not topological: they need a metric.
  • $f : X\to Y$ is continuous if preimages of open sets are open:
    $$V \text{ open} \;\Longrightarrow\; f^{-1}(V)=\{x : f(x)\in V\} \text{ open}.$$
  • In metric spaces, this agrees with the sequence definition; compositions are continuous.
3.3 · Compactness

Open covers and finite subcovers

  • Open sets $U_i$, $i\in I$, form an open cover of $K$ if $K\subseteq\bigcup_i U_i$.
  • $K$ is compact if every open cover has a finite subcover $K\subseteq U_{i_1}\cup\dots\cup U_{i_m}$.
  • Example 3.1. Every finite set is compact.
  • $\mathbb{R}$ is not compact: $(-k,k)$, $k=1,2,\dots$, cover $\mathbb{R}$, but finitely many of them only cover a bounded interval.
3.3 · Compactness

Compact sets in $\mathbb{R}^n$

A subset $K$ of a normed space $V$ is bounded if ${\|\mathbf{x}\|}_V\le C$ for all $\mathbf{x}\in K$ and some constant $C$.

Theorem 3.2 (Heine–Borel). Let $\|\cdot\|$ be any norm on $\mathbb{R}^n$. A subset of $\mathbb{R}^n$ is compact if and only if it is closed and bounded with respect to $\|\cdot\|$. None of these properties depends on the choice of the norm.

  • We state the theorem without proof. That closedness, boundedness and compactness do not depend on the norm follows from Theorem 3.4 (end of lecture).
  • In metric spaces: $K$ is compact exactly when every sequence in $K$ has a subsequence converging in $K$.
$$\begin{aligned}\mathbf{e}_1&=(1,0,0,0,\dots)\\ \mathbf{e}_2&=(0,1,0,0,\dots)\\ \mathbf{e}_3&=(0,0,1,0,\dots)\\[-4pt] &\ \ \vdots\end{aligned}\quad {\|\mathbf{e}_i-\mathbf{e}_j\|}_2=\sqrt2 \ \ (i\ne j)$$
  • In infinite dimensions the theorem fails.
  • In $\ell^2$, the space of real sequences $\mathbf{x}=(x_1,x_2,\dots)$ with ${\|\mathbf{x}\|}_2=\bigl(\sum_{i=1}^\infty x_i^2\bigr)^{1/2}<\infty$ (Lecture 7), the unit vectors $\mathbf{e}_i$ lie in the closed unit ball, but no subsequence is Cauchy.
3.3 · Compactness

Continuous functions attain their extremes

Theorem 3.3. If $f : X\to Y$ is continuous and $K\subseteq X$ compact, then $f(K)$ is compact. In particular, every continuous $f : K\to\mathbb{R}$ on $K\ne\emptyset$ attains its maximum and minimum.

Proof. If $V_i$ cover $f(K)$, the open sets $f^{-1}(V_i)$ cover $K$, so finitely many do, and their $V_i$ cover $f(K)$. For $Y=\mathbb{R}$: $f(K)$ is bounded, so $s=\sup f(K)$ is finite and the limit of a sequence in $f(K)$; $f(K)$ is closed, so $s\in f(K)$.□

Machine Learning: a minimiser of a loss function exists whenever the loss is continuous and the parameters range over a compact set.

3.3 · Compactness

All norms on $\mathbb{R}^n$ are equivalent

Theorem 3.4. For any two norms $\|\cdot\|$, $\|\cdot\|'$ on $\mathbb{R}^n$ there are $c,C>0$ with $c\,\|\mathbf{x}\|'\le\|\mathbf{x}\|\le C\,\|\mathbf{x}\|'$ for all $\mathbf{x}$.

Proof. It suffices to show $c{\|\mathbf{x}\|}_\infty\le\|\mathbf{x}\|\le C{\|\mathbf{x}\|}_\infty$ for every norm; applied to $\|\cdot\|$ and $\|\cdot\|'$, this gives the theorem.

  • Upper bound: $\|\mathbf{x}\|=\|\sum_i x_i\mathbf{e}_i\|\le\sum_i|x_i|\,\|\mathbf{e}_i\|\le C{\|\mathbf{x}\|}_\infty$, $C:=\sum_i\|\mathbf{e}_i\|$.
  • Lower bound: $\|\cdot\|$ is continuous (Example 2.7), by the upper bound also with respect to ${\|\cdot\|}_\infty$. On the compact set ${S=\{{\|\mathbf{x}\|}_\infty=1\}}$ it attains a minimum $c$ (Theorem 3.3), and $c>0$ as $\mathbf{0}\notin S$. With $\mathbf{x}/{\|\mathbf{x}\|}_\infty\in S$: $\|\mathbf{x}\|\ge c{\|\mathbf{x}\|}_\infty$.□

Hence all norms on $\mathbb{R}^n$ have the same convergent sequences, open and compact sets, and $\mathbb{R}^n$ is complete for every norm.

Lecture 3

Summary

  • A norm assigns lengths to vectors; the $p$-norms ${\|\cdot\|}_1$, ${\|\cdot\|}_2$, ${\|\cdot\|}_\infty$ capture different aspects of data, such as peak and energy.
  • A norm comes from an inner product exactly when the parallelogram law holds (Theorem 1.2).
  • A metric measures distances on any set (genomes, U-Bahn); $\mathbb{R}^n$ is complete, $\mathbb{Q}$ is not.
  • Continuous maps preserve limits; a threshold classifier jumps at its decision boundary.
  • A topology, the family of open sets, describes convergence and continuity without distances.
  • In $\mathbb{R}^n$, compact means closed and bounded (Theorem 3.2); continuous functions attain their extremes on compact sets (Theorem 3.3); all norms are equivalent (Theorem 3.4).
  • Each structure is weaker than the previous one: inner product ${\Rightarrow}$ norm ${\Rightarrow}$ metric ${\Rightarrow}$ topology.

Next: inner products on $\mathbb{R}^n$ and projections onto subspaces (Lecture 4); infinite dimensions (Lecture 7).