Mathematics for Machine Learning · Prof. Dr. Nihat Ay

Lecture 2
Geometric structures of $\mathbb{R}^n$, Part 1

Vector space structure · affine space structure · inner product space structure

Overview

Beyond $\mathbb{R}^n$

The geometric structures of $\mathbb{R}^n$ extend to more general settings that are important in Machine Learning: matrices, sequences, functions.

  1. Vector space structure
    1.1 Vector spaces, subspaces and bases
    1.2 Linear maps
  2. Affine space structure
    2.1 Affine spaces and affine subspaces
    2.2 Affine maps
  3. Inner product space structure
    3.1 Inner products and examples
    3.2 Norms and angles
    3.3 Isometries
Section 1 of 3
1

Vector space structure

1.1 Vector spaces, subspaces and bases1.2 Linear maps

1.1 · Vector spaces, subspaces and bases

Vector spaces

With the componentwise operations of Lecture 1, $\mathbb{R}^n$ is a vector space. In general, a (real) vector space is a set $V$ with addition and scalar multiplication

$$+ : V\times V\to V,\ (\mathbf{x},\mathbf{y})\mapsto\mathbf{x}+\mathbf{y}, \qquad \cdot : \mathbb{R}\times V\to V,\ (\lambda,\mathbf{x})\mapsto\lambda\mathbf{x},$$

such that for all $\mathbf{x},\mathbf{y},\mathbf{z}\in V$ and $\lambda,\mu\in\mathbb{R}$:

  1. Associativity: $(\mathbf{x}+\mathbf{y})+\mathbf{z}=\mathbf{x}+(\mathbf{y}+\mathbf{z})$
  2. Commutativity: $\mathbf{x}+\mathbf{y}=\mathbf{y}+\mathbf{x}$
  3. Zero vector: there is $0\in V$ with $\mathbf{x}+0=\mathbf{x}$
  4. Inverse: there is $-\mathbf{x}$ with $\mathbf{x}+(-\mathbf{x})=0$
  1. Compatibility: $\lambda(\mu\mathbf{x})=(\lambda\mu)\mathbf{x}$
  2. Identity: $1\,\mathbf{x}=\mathbf{x}$
  3. Distributivity I: $\lambda(\mathbf{x}+\mathbf{y})=\lambda\mathbf{x}+\lambda\mathbf{y}$
  4. Distributivity II: $(\lambda+\mu)\mathbf{x}=\lambda\mathbf{x}+\mu\mathbf{x}$

Axioms 1–4: $(V,+)$ is an abelian group; 5–8 link scaling to it. Here all vector spaces are real.

1.1 · Vector spaces, subspaces and bases

Subspaces

$\emptyset\neq U\subseteq V$ is a subspace if $\mathbf{x}+\mathbf{y}\in U$ and $\lambda\mathbf{x}\in U$ for all $\mathbf{x},\mathbf{y}\in U$, $\lambda\in\mathbb{R}$.

$U$ is closed under both operations. Together with $U\neq\emptyset$ (equivalently $0\in U$, as $0=0\cdot\mathbf{x}$), the operations restrict to $U$, and all eight axioms are inherited: $U$ is again a vector space, without checking them again. Trivial subspaces: $\{0\}$ and $V$.

A plane $U$ through $0$ in $\mathbb{R}^3$. Drag $\mathbf{x}$ and $\mathbf{y}$ within $U$, drag elsewhere to turn the axes; change $\lambda$.

1.1 · Vector spaces, subspaces and bases

Examples of vector spaces

  1. $\mathbb{R}^n$, the model space of Lecture 1: the other examples generalise it.
  2. Matrices $\mathbb{R}^{m\times n}$, entrywise: “$\mathbb{R}^{mn}$ in disguise” — weights of a linear layer, a batch of images, a covariance matrix.
  3. All sequences $\mathbb{R}^{\mathbb{N}}$, $\mathbf{x}=(x_1,x_2,\dots)^\top$.
  4. Square-summable sequences $$\ell^2=\left\{\mathbf{x}\in\mathbb{R}^{\mathbb{N}} : \sum\nolimits_i x_i^2<\infty\right\}$$ form a subspace of $\mathbb{R}^{\mathbb{N}}$: closed under scalar multiplication, and under $+$ since ${(a+b)}^2\le 2a^2+2b^2$.
1.1 · Vector spaces, subspaces and bases

More examples: functions are vectors

A vector in $\mathbb{R}^n$ is a function on $\{1,\dots,n\}$, $i\mapsto x_i$. For large $n$, it looks like a function on an interval.

  1. All functions on $\Omega\subseteq\mathbb{R}^d$ form a vector space $\mathbb{R}^\Omega$, with pointwise operations $(\alpha f+\beta g)(t)=\alpha f(t)+\beta g(t)$.
  2. Square-integrable functions $L^2(\mu)=\{f\in\mathbb{R}^\Omega:\int_\Omega f(t)^2\,\mu(dt)<\infty\}$ form a subspace of $\mathbb{R}^\Omega$, as $\ell^2$ does of $\mathbb{R}^{\mathbb{N}}$. The measure $\mu$ weights regions of $\Omega$ (Lebesgue $dt$, or a data distribution); counting measure on $\mathbb{N}$ gives $\ell^2$.
  3. Polynomials $\mathbb{R}[t]\supseteq\mathbb{R}[t]_{\le n}$, a subspace of $\mathbb{R}^{\mathbb{R}}$.
  4. Continuous functions $C(\Omega)$, a subspace of $\mathbb{R}^\Omega$.
1.1 · Vector spaces, subspaces and bases

Span, linear independence and bases

Let $V$ be a vector space and $S\subseteq V$ a set of vectors, finite or infinite. A linear combination uses finitely many of them, $c_1\mathbf{s}_1+\dots+c_m\mathbf{s}_m$.

  • The span of $S$: all linear combinations, the smallest subspace containing $S$.
  • $S$ is linearly independent if only the trivial combination gives $0$; equivalently, every vector of the span has unique coefficients.
  • A basis of $V$ is a linearly independent set with span $V$. All bases have the same number of elements, the dimension $\dim V$. In a fixed order, $(\mathbf{b}_1,\dots,\mathbf{b}_n)$ gives each vector its coordinates: $\mathbf{v}=\sum_i c_i\mathbf{b}_i$.

Examples. $\mathbf{e}_1,\dots,\mathbf{e}_n$ is a basis of $\mathbb{R}^n$. The hyperplane ${\displaystyle \left\{\mathbf{x} : \sum\nolimits_i x_i=0\right\}}$ has the basis $\mathbf{e}_i-\mathbf{e}_n$, $i<n$, and dimension $n-1$. $\mathbb{R}[t]$ has the infinite basis $1,t,t^2,\dots$; like $\ell^2$, $L^2(\mu)$ and $C([0,1])$, it has no finite basis and is infinite-dimensional.

1.2 · Linear maps

Structure-preserving maps

Whenever we introduce a structure, we ask which maps preserve it. For vector spaces these are the linear maps; in this lecture also affine maps for affine spaces and isometries for inner product spaces; continuous maps for topological spaces follow in Lecture 3.

A map $f : V\to W$ between vector spaces is linear if

$$f(\mathbf{x}+\mathbf{y})=f(\mathbf{x})+f(\mathbf{y}) \ \ \text{(additivity)}, \qquad f(\lambda\mathbf{x})=\lambda f(\mathbf{x}) \ \ \text{(homogeneity)}.$$
  • Equivalently, $f$ preserves linear combinations: $f(\lambda\mathbf{x}+\mu\mathbf{y})=\lambda f(\mathbf{x})+\mu f(\mathbf{y})$. With $\lambda=\mu=0$: $f(0)=0$.
  • A bijective linear map is an isomorphism; its inverse is automatically linear. Isomorphic spaces carry the same structure and differ only in the names of their elements.
1.2 · Linear maps

Examples of linear maps

  1. 1.1 Inclusion of a subspace $U\subseteq V$: $U\hookrightarrow V$, $\mathbf{u}\mapsto\mathbf{u}$.
  2. 1.2 Matrix multiplication $\mathbf{x}\mapsto A\mathbf{x}$, $A\in\mathbb{R}^{m\times n}$: defined because $\mathbf{x}$ is a column, an $n\times1$ matrix, and $A\mathbf{x}$ is again a column, an element of $\mathbb{R}^m$. → Example 1.2
  3. 1.3 Evaluation at finitely many points $\mathrm{ev}_{t_1,\dots,t_k}:\mathbb{R}^\Omega\to\mathbb{R}^k$. → Example 1.3
  4. 1.4 Reading out the entries of a matrix, $\mathbb{R}^{m\times n}\to\mathbb{R}^{mn}$: a linear isomorphism, the precise sense in which $\mathbb{R}^{m\times n}$ is “$\mathbb{R}^{mn}$ in disguise”. → Example 1.4
  5. 1.5 Smoothing an image $S:\mathbb{R}^{784}\to\mathbb{R}^{784}$. → Example 1.5
  6. 1.6 Averaging over blocks of $4\times4$ pixels, $P:\mathbb{R}^{784}\to\mathbb{R}^{49}$: a ${28\times28}$ image becomes a ${7\times7}$ image. → Example 1.6

Example 1.1 is complete here; Examples 1.2 to 1.6 follow, one slide each.

1.2 · Linear maps

Example 1.2: matrix multiplication

$$\mathbf{x}=x_1\mathbf{e}_1+x_2\mathbf{e}_2 \quad\mapsto\quad A\mathbf{x}=x_1\,f(\mathbf{e}_1)+x_2\,f(\mathbf{e}_2)$$
domain $\mathbb{R}^2$: drag $\mathbf{x}$
image $\mathbb{R}^2$: drag $f(\mathbf{e}_1)$ and $f(\mathbf{e}_2)$

Same coefficients $x_1, x_2$ on both sides. The columns of $A$ are $f(\mathbf{e}_1), f(\mathbf{e}_2)$, and they determine $f$.

1.2 · Linear maps

Example 1.3: evaluation at finitely many points

$$\mathrm{ev}_{t_1,\dots,t_k} : \mathbb{R}^\Omega\to\mathbb{R}^k, \qquad f\mapsto\bigl(f(t_1),\dots,f(t_k)\bigr)^\top$$

Evaluating a function at $t_1,\dots,t_k$ is a linear map $\mathbb{R}^\Omega\to\mathbb{R}^k$: each coordinate satisfies $(\alpha f+\beta g)(t_i)=\alpha f(t_i)+\beta g(t_i)$. For ${k=1}$, this is the evaluation at a single point. Drag the points $t_i$ along the axis. The same works on $C(\Omega)$ and $\mathbb{R}[t]$.

1.2 · Linear maps

Example 1.4: reading out the entries of a matrix

$$\mathbb{R}^{m\times n}\to\mathbb{R}^{mn}, \qquad A=(a_{ij}) \mapsto (a_{11},\dots,a_{1n},\,a_{21},\dots,a_{2n},\,\dots,\,a_{mn})^\top$$
  • Linear, since $+$ and scalar multiplication act entry by entry; bijective, since putting the entries back is the inverse. So it is an isomorphism, and $\dim\mathbb{R}^{m\times n}=mn$.
  • This is the map of Lecture 1 that turned a greyscale image into a column: a $28\times28$ image is a matrix of grey values, and reading it row by row gives its vector in $\mathbb{R}^{784}$. Column by column gives another isomorphism.
1.2 · Linear maps

Example 1.5: smoothing an image

$$(S\mathbf{x})_{(i,j)}=\tfrac12\,x_{(i,j)}+\tfrac18\bigl(x_{(i-1,j)}+x_{(i+1,j)}+x_{(i,j-1)}+x_{(i,j+1)}\bigr)$$

Every pixel becomes a weighted average of itself and its four neighbours, with fixed weights (values outside the image count as $0$). So $S$ is linear: a $784\times784$ matrix.

⅛i−1,j⅛i,j−1½i,j⅛i,j+1⅛i+1,j
row $i$, column $j$
$\mathbf{x}$

Diffusion models (DALL·E 2 and 3, Stable Diffusion) degrade images step by step with Gaussian noise, some variants by blurring as $S$ does, and train a neural network to undo each step; run backwards from noise, it creates new images.

1.2 · Linear maps

Example 1.6: coarse-graining an image

The coarse-graining map $P$ averages over blocks of $4\times4$ pixels:

$$P:\mathbb{R}^{784}\to\mathbb{R}^{49},\qquad (P\mathbf{x})_{(I,J)}=\frac{1}{16}\sum_{i=4I-3}^{4I}\ \sum_{j=4J-3}^{4J} x_{(i,j)},\qquad I,J=1,\dots,7$$
$\mathbf{x}\in\mathbb{R}^{784}$, $28\times28$ pixels
$P\mathbf{x}\in\mathbb{R}^{49}$, $7\times7$ pixels
  • Each of the $49$ blocks of $4\times4$ pixels becomes one pixel, the average of its $16$ values.
  • Linear: fixed weights $\tfrac1{16}$; not injective: $784$ numbers become $49$.

Applications. The same basic idea, fewer numbers at lower resolution, is behind streaming video when the connection is slow, pixel binning in smartphone cameras, and colour storage in JPEG images.

1.2 · Linear maps

The space of linear maps

Sums and multiples of linear maps are again linear:

$$(f+g)(\mathbf{x}):=f(\mathbf{x})+g(\mathbf{x}), \qquad (\lambda f)(\mathbf{x}):=\lambda f(\mathbf{x}),$$

so the set of all linear maps is a vector space,

$$\mathrm{Lin}(V,W):=\{f:V\to W \;:\; f\ \text{linear}\}.$$

Since $f(\mathbf{x})=\sum_j x_j f(\mathbf{e}_j)=A\mathbf{x}$ with the columns $f(\mathbf{e}_j)$,

$$\mathrm{Lin}(\mathbb{R}^n,\mathbb{R}^m)\cong\mathbb{R}^{m\times n}.$$

The zero element is the zero map; the axioms hold value by value, as for $\mathbb{R}^X$. The map $f\mapsto A$ is an isomorphism: the sum of two linear maps corresponds to the sum of their matrices. Special case $\mathrm{Lin}(V,\mathbb{R})$: Lecture 6.

Section 2 of 3
2

Affine space structure

2.1 Affine spaces and affine subspaces2.2 Affine maps

2.1 · Affine spaces and affine subspaces

Affine spaces

Points have no origin; we translate them by vectors. An affine space $(A,V,+)$ consists of a set $A$ of points, a vector space $V$ and a translation $+ : A\times V\to A$, $(p,v)\mapsto p+v$, with:

  1. Identity: $p+0=p$ for all $p\in A$.
  2. Associativity: $(p+v)+w=p+(v+w)$ for all $p\in A$ and all $v,w\in V$.
  3. Unique connecting vector: for any $p,q\in A$, there is exactly one $v\in V$ with $p+v=q$, the connecting vector $q-p$.
pqq − p
physical space: no origin, only $q-p$ matters

Every vector space $V$ is an affine space over itself: $(V,V,+)$, with the vector addition as translation, $(\mathbf{x},\mathbf{v})\mapsto\mathbf{x}+\mathbf{v}$.

The translation is part of the structure: another translation on the same $A$ and $V$ gives another affine space; when it is clear, we say “$A$ is an affine space over $V$”. We do not add two points but translate a point by a vector: “$+$” has two meanings. Everyday affine spaces: physical space, positions on a map, instants of time (durations as differences).

2.1 · Affine spaces and affine subspaces

Affine subspaces

$$B=p_0+U=\{p_0+u : u\in U\}, \qquad U \text{ a linear subspace, the direction}$$

From the base point $p_0$, $s$ reaches all points $p=p_0+s\,d$ of $B$; $d$ is a unit vector in $U$. Drag $p_0$.

  • $B$ is an affine space over $U$, with the restriction of the translation of $A$:
    $$B\times U\to B,\qquad (p,v)\mapsto p+v.$$
  • Any $p\in B$ is a base point, $B=p+U$; the direction $U$ is determined by $B$.
  • For $A=V$: $p_0+U$ is a linear subspace $\iff p_0\in U$.
2.1 · Affine spaces and affine subspaces

Example 2.1: a hyperplane of probability distributions

$$H=\left\{\mathbf{x}\in\mathbb{R}^n : \sum\nolimits_i x_i=1\right\}$$

An affine subspace: $H=p_0+U$ with $p_0=(\tfrac1n,\dots,\tfrac1n)^\top$ and ${\displaystyle U=\left\{\mathbf{x}\in\mathbb{R}^n : \sum\nolimits_i x_i=0\right\}}$.

drag $\mathbf{x}$ in $H$; drag elsewhere to turn the axes

For $n=3$: $H$ is the plane through $\mathbf{e}_1,\mathbf{e}_2,\mathbf{e}_3$, and $U$ the parallel plane through $0$. Since $0\notin H$, $H$ is not a linear subspace.

The probability distributions, $x_i\ge 0$ and $\sum_i x_i=1$, form the triangle with vertices $\mathbf{e}_1,\mathbf{e}_2,\mathbf{e}_3$ in $H$, the simplex.

2.1 · Affine spaces and affine subspaces

Example 2.2: decision boundary of a neuron

$$r(\mathbf{x})=w_1x_1+w_2x_2+b,\qquad \sigma(r)=\frac{1}{1+e^{-r}},\qquad B=\{\mathbf{x} : r(\mathbf{x})=0\}$$
  • The output is $>\tfrac12$ (blue) exactly where $r>0$, and $<\tfrac12$ (red) where $r<0$.
  • The decision boundary $B$ (black) is an affine subspace, $B=p_0+U$ with $U=\{\mathbf{x}:w_1x_1+w_2x_2=0\}$.
  • The weights fix the direction $U$; the bias $b$ shifts $B$ parallel to itself.
2.2 · Affine maps

Affine maps

An affine map is a linear map plus a translation. Let $A$, $B$ be affine spaces over $V$, $W$.

Definition 2.3. A map $F:A\to B$ is affine if there are a point $p\in A$ and a linear map $L:V\to W$ with

$$F(p+v)=F(p)+L(v)\quad\text{for all } v\in V.$$

Lemma 2.4. For affine $F$, $L$ is unique, the linear part of $F$, and works at every point:

$$F(q+v)=F(q)+L(v)\quad\text{for all } q\in A,\ v\in V.$$

Proof. Uniqueness: $L(v)$ translates $F(p)$ to $F(p+v)$, so it is the connecting vector $F(p+v)-F(p)$ (property 3). Every point is $q=p+u$, and by associativity $F(q+v)=F(p+(u+v))=F(p)+L(u)+L(v)=F(q)+L(v)$.□

2.2 · Affine maps

Example 2.5: affine maps

$$\mathbf{x}\;\longmapsto\;A\mathbf{x}\;\longmapsto\;F(\mathbf{x})=A\mathbf{x}+\mathbf{b}$$
domain $\mathbb{R}^2$: drag $\mathbf{x}$
linear part: drag $A\mathbf{e}_1$, $A\mathbf{e}_2$
shift by $\mathbf{b}$: drag $\mathbf{b}$

The whole grid moves by the same vector $\mathbf{b}=F(0)$: lines stay lines, parallels stay parallel, but $0$ goes to $\mathbf{b}$. Every affine map $\mathbb{R}^n\to\mathbb{R}^m$ has this form, with linear part $L=A$; it is linear only for $\mathbf{b}=0$.

Section 3 of 3
3

Inner product space structure

3.1 Inner products and examples3.2 Norms and angles3.3 Isometries

3.1 · Inner products and examples

Inner products

The dot product of Lecture 1 has three properties, for all $\mathbf{x},\mathbf{y}\in\mathbb{R}^n$:

  1. Bilinearity: it is linear in each argument separately, that is,
    $$\langle\mathbf{x},\cdot\rangle:\ \mathbb{R}^n\to\mathbb{R},\ \ \mathbf{y}\mapsto\langle\mathbf{x},\mathbf{y}\rangle, \qquad\text{and}\qquad \langle\cdot,\mathbf{y}\rangle:\ \mathbb{R}^n\to\mathbb{R},\ \ \mathbf{x}\mapsto\langle\mathbf{x},\mathbf{y}\rangle,$$
    are linear maps, for every $\mathbf{x}$ and every $\mathbf{y}$.
  2. Symmetry: $\langle\mathbf{x},\mathbf{y}\rangle=\langle\mathbf{y},\mathbf{x}\rangle$.
  3. Positive definiteness: $\langle\mathbf{x},\mathbf{x}\rangle\ge0$, with equality if and only if $\mathbf{x}=0$.

A map ${\langle\cdot,\cdot\rangle}_V : V\times V\to\mathbb{R}$ with these three properties is an inner product, and $(V,{\langle\cdot,\cdot\rangle}_V)$ is an inner product space.

These properties, rather than the specific formula $\sum_i x_iy_i$, are what let us speak of norms (lengths) and angles. Whether the dot product extends to the spaces above: next slide.

3.1 · Inner products and examples

Does the dot product extend?

spaceextension of the dot productwhy it converges, or not
$\mathbb{R}^n$$\sum_i x_iy_i$finite sum; many others (Lecture 4)
$\mathbb{R}^{m\times n}$$\sum_{i,j}A_{ij}B_{ij}=\operatorname{tr}(A^\top B)$Frobenius: the dot product of $mn$-tuples
$\mathbb{R}^{\mathbb{N}}$$\sum_i x_iy_i$: not for all pairs$\sum_i x_iy_i$ may diverge, e.g. $x_i=y_i=1$
$\ell^2$$\sum_i x_iy_i$$|x_iy_i|\le\tfrac12(x_i^2+y_i^2)$
$\mathbb{R}^\Omega$$\int_\Omega fg\,d\mu$: not for all pairs$\int_\Omega fg\,d\mu$ may diverge
$L^2(\mu)$$\int_\Omega f(t)g(t)\,\mu(dt)$ → example$|fg|\le\tfrac12(f^2+g^2)$ pointwise
$\mathbb{R}[t]$$\sum_i a_ib_i$ or $\int_{-1}^1 pq\,dt$coefficients, or $L^2$ on $[-1,1]$
$C(\Omega)$$\int_\Omega fg\,d\mu$$\Omega$ compact, $\mu$ finite; fails for $f=1$ on $\mathbb{R}$, $f=1/t$ on $(0,1]$

The question is whether the formula of the dot product converges. On $\mathbb{R}^{\mathbb{N}}$ and $\mathbb{R}^\Omega$ it does not always; this does not mean that these spaces admit no inner product at all.

3.2 · Norms and angles

Cauchy–Schwarz inequality

In every inner product space $V$, the norm (length) of $\mathbf{x}$ is, as in Lecture 1, ${\|\mathbf{x}\|}_V:=\sqrt{{\langle\mathbf{x},\mathbf{x}\rangle}_V}$.

Theorem 3.1 (Cauchy–Schwarz).

  1. For every symmetric bilinear form with ${\langle\mathbf{x},\mathbf{x}\rangle}_V\ge0$ (positive semi-definite): ${\langle\mathbf{x},\mathbf{y}\rangle}_V^2\le{\langle\mathbf{x},\mathbf{x}\rangle}_V\,{\langle\mathbf{y},\mathbf{y}\rangle}_V$.
  2. For an inner product: $\bigl|{\langle\mathbf{x},\mathbf{y}\rangle}_V\bigr|\le{\|\mathbf{x}\|}_V\,{\|\mathbf{y}\|}_V$, with equality if and only if $\mathbf{x},\mathbf{y}$ are linearly dependent.

Proof of 1. $0\le{\langle\mathbf{x}-t\mathbf{y},\mathbf{x}-t\mathbf{y}\rangle}_V={\langle\mathbf{x},\mathbf{x}\rangle}_V-2t\,{\langle\mathbf{x},\mathbf{y}\rangle}_V+t^2{\langle\mathbf{y},\mathbf{y}\rangle}_V$ for all $t$. If ${\langle\mathbf{y},\mathbf{y}\rangle}_V>0$, choose $t={\langle\mathbf{x},\mathbf{y}\rangle}_V/{\langle\mathbf{y},\mathbf{y}\rangle}_V$; if ${\langle\mathbf{y},\mathbf{y}\rangle}_V=0$, the right side is affine in $t$ and $\ge0$, so ${\langle\mathbf{x},\mathbf{y}\rangle}_V=0$.$\square$

Part 1 does not need positive definiteness; this matters for kernels (Lectures 5 and 9), which give positive semi-definite forms.

3.2 · Norms and angles

Cosine similarity and angle

By Theorem 3.1 (part 2), the quotient below lies in $[-1,1]$; it compares directions, not lengths.

Definition 3.2. Let $V$ be an inner product space and $\mathbf{x},\mathbf{y}\in V$, $\mathbf{x},\mathbf{y}\neq0$. The angle $\alpha_V\in[0,\pi]$ between $\mathbf{x}$ and $\mathbf{y}$ is given by

$$\cos(\alpha_V):=\frac{{\langle\mathbf{x},\mathbf{y}\rangle}_V}{{\|\mathbf{x}\|}_V\,{\|\mathbf{y}\|}_V},$$

and $\cos(\alpha_V)$ is called the cosine similarity of $\mathbf{x}$ and $\mathbf{y}$. The vectors are orthogonal, $\mathbf{x}\perp\mathbf{y}$, if ${\langle\mathbf{x},\mathbf{y}\rangle}_V=0$.

  • On $\mathbb{R}^n$ with the dot product, this is the angle of Lecture 1.
  • By the equality case of Theorem 3.1, $\cos(\alpha_V)=\pm1$ exactly when $\mathbf{x}$ and $\mathbf{y}$ are linearly dependent.
  • In machine learning, cosine similarity compares embeddings of words, texts or images, for instance in search and recommendation.
3.2 · Norms and angles

Example 3.3: cosine similarities and angles of functions

$$\langle f,g\rangle=\int_{-1}^{1}f(t)\,g(t)\,dt, \qquad \|f\|=\sqrt{\langle f,f\rangle}, \qquad \cos(\alpha)=\frac{\langle f,g\rangle}{\|f\|\,\|g\|}$$
f
g

The $L^2(\mu)$ inner product for the Lebesgue measure on $[-1,1]$. Shaded: ${\int_{-1}^1 f(t)\,g(t)\,dt}$, positive minus negative parts. Orthogonal, for instance: ${1\perp t^2-\tfrac13}$ and $1\perp t$.

3.3 · Isometries

Isometries

The maps that preserve the structure of an inner product space are the linear maps that preserve the inner product.

Definition 3.4. Let $V$ and $W$ be inner product spaces. A linear map $T:V\to W$ is called an isometry if

$${\langle T\mathbf{x},T\mathbf{y}\rangle}_W={\langle\mathbf{x},\mathbf{y}\rangle}_V \qquad\text{for all } \mathbf{x},\mathbf{y}\in V.$$

A bijective isometry is called an isometric isomorphism.

  • An isometry preserves norms, ${\|T\mathbf{x}\|}_W={\|\mathbf{x}\|}_V$, hence angles and orthogonality.
  • Conversely, a linear map that preserves norms is an isometry, since ${\langle\mathbf{x},\mathbf{y}\rangle}_V=\tfrac14\bigl({\|\mathbf{x}+\mathbf{y}\|}_V^2-{\|\mathbf{x}-\mathbf{y}\|}_V^2\bigr)$.
  • An isometry is injective: $T\mathbf{x}=0$ gives ${\|\mathbf{x}\|}_V={\|T\mathbf{x}\|}_W=0$, so $\mathbf{x}=0$. It need not be surjective, for instance $\mathbb{R}\to\mathbb{R}^2$, $x\mapsto(x,0)^\top$.
  • Isometrically isomorphic spaces differ only in the names of their elements.
3.3 · Isometries

Example 3.5: isometries

drag $\mathbf{x}$, $\mathbf{y}$
  1. Orthogonal matrices on $\mathbb{R}^n$: $\mathbf{x}\mapsto T\mathbf{x}$ with $T^\top T=I$. For $n=2$: the rotations and reflections of the picture.
  2. Reading out the entries of a matrix (Example 1.4): an isometric isomorphism from $\mathbb{R}^{m\times n}$ with the Frobenius inner product onto $\mathbb{R}^{mn}$.
  3. Inclusion of a subspace $U\hookrightarrow V$, with the restricted inner product.
3.3 · Isometries

Isometries of $\mathbb{R}^n$

For $\mathbb{R}^n$ with the dot product, the first example already gives all isometries.

Theorem 3.6. A linear map $\mathbb{R}^n\to\mathbb{R}^n$, $\mathbf{x}\mapsto T\mathbf{x}$, is an isometry for the dot product if and only if $T^\top T=I$, that is, if the columns of $T$ form an orthonormal basis. In this case, $\det(T)=\pm1$. For $n=2$, the isometries are exactly the rotations and the reflections in lines through $0$:

$$\begin{pmatrix}\cos(\varphi)&-\sin(\varphi)\\ \sin(\varphi)&\cos(\varphi)\end{pmatrix} \quad\text{and}\quad \begin{pmatrix}\cos(\varphi)&\sin(\varphi)\\ \sin(\varphi)&-\cos(\varphi)\end{pmatrix},\qquad \varphi\in[0,2\pi).$$

Proof. $\langle T\mathbf{x},T\mathbf{y}\rangle=\mathbf{x}^\top T^\top T\,\mathbf{y}$, which equals $\mathbf{x}^\top\mathbf{y}$ for all $\mathbf{x},\mathbf{y}$ if $T^\top T=I$; conversely, $\mathbf{x}=\mathbf{e}_i$, $\mathbf{y}=\mathbf{e}_j$ give ${(T^\top T)}_{ij}=\langle T\mathbf{e}_i,T\mathbf{e}_j\rangle=\delta_{ij}$. Moreover $\det(T)^2=\det(T^\top T)=1$. For $n=2$, the first column is $(\cos(\varphi),\sin(\varphi))^\top$ and the second a unit vector orthogonal to it, $\pm(-\sin(\varphi),\cos(\varphi))^\top$.$\square$

Lecture 2

Summary

  • Vector spaces: adding and scaling obey the rules of $\mathbb{R}^n$ (matrices, sequences, functions). Span, linear independence and bases use finite linear combinations; bases may be infinite.
  • Linear maps preserve linear combinations; from $\mathbb{R}^n$ to $\mathbb{R}^m$ they are matrices, for instance smoothing and coarse-graining images; $\mathrm{Lin}(V,W)$ is a vector space.
  • An affine space $(A,V,+)$: points translated by vectors, no origin; every vector space is one over itself. Affine subspaces are translated linear subspaces.
  • An affine map is a linear map plus a translation: $F(p+v)=F(p)+L(v)$ with a unique linear part $L$, at every point; on $\mathbb{R}^n$: $F(\mathbf{x})=A\mathbf{x}+\mathbf{b}$.
  • Inner products give norms, cosine similarities and angles beyond $\mathbb{R}^n$, well defined by Cauchy–Schwarz. Isometries preserve them and are injective; on $\mathbb{R}^n$: orthogonal matrices.

Next: norms, metric spaces and topology (Lecture 3).