Mathematics for Machine Learning · Prof. Dr. Nihat Ay
Lecture 2 Geometric structures of $\mathbb{R}^n$, Part 1
Vector space structure · affine space structure · inner product space structure
Overview
Beyond $\mathbb{R}^n$
The geometric structures of $\mathbb{R}^n$ extend to more general settings that are important in Machine Learning: matrices, sequences, functions.
Vector space structure
1.1 Vector spaces, subspaces and bases 1.2 Linear maps
Affine space structure
2.1 Affine spaces and affine subspaces 2.2 Affine maps
Inner product space structure
3.1 Inner products and examples 3.2 Norms and angles 3.3 Isometries
Section 1 of 3
1
Vector space structure
1.1 Vector spaces, subspaces and bases1.2 Linear maps
1.1 · Vector spaces, subspaces and bases
Vector spaces
With the componentwise operations of Lecture 1, $\mathbb{R}^n$ is a vector space. In general, a (real) vector space is a set $V$ with addition and scalar multiplication
Axioms 1–4: $(V,+)$ is an abelian group; 5–8 link scaling to it. Here all vector spaces are real.
1.1 · Vector spaces, subspaces and bases
Subspaces
$\emptyset\neq U\subseteq V$ is a subspace if $\mathbf{x}+\mathbf{y}\in U$ and $\lambda\mathbf{x}\in U$ for all $\mathbf{x},\mathbf{y}\in U$, $\lambda\in\mathbb{R}$.
$U$ is closed under both operations. Together with $U\neq\emptyset$ (equivalently $0\in U$, as $0=0\cdot\mathbf{x}$), the operations restrict to $U$, and all eight axioms are inherited: $U$ is again a vector space, without checking them again. Trivial subspaces: $\{0\}$ and $V$.
A plane $U$ through $0$ in $\mathbb{R}^3$. Drag $\mathbf{x}$ and $\mathbf{y}$ within $U$, drag elsewhere to turn the axes; change $\lambda$.
1.1 · Vector spaces, subspaces and bases
Examples of vector spaces
$\mathbb{R}^n$, the model space of Lecture 1: the other examples generalise it.
Matrices $\mathbb{R}^{m\times n}$, entrywise: “$\mathbb{R}^{mn}$ in disguise” — weights of a linear layer, a batch of images, a covariance matrix.
All sequences $\mathbb{R}^{\mathbb{N}}$, $\mathbf{x}=(x_1,x_2,\dots)^\top$.
Square-summable sequences $$\ell^2=\left\{\mathbf{x}\in\mathbb{R}^{\mathbb{N}} : \sum\nolimits_i x_i^2<\infty\right\}$$ form a subspace of $\mathbb{R}^{\mathbb{N}}$: closed under scalar multiplication, and under $+$ since ${(a+b)}^2\le 2a^2+2b^2$.
1.1 · Vector spaces, subspaces and bases
More examples: functions are vectors
A vector in $\mathbb{R}^n$ is a function on $\{1,\dots,n\}$, $i\mapsto x_i$. For large $n$, it looks like a function on an interval.
All functions on $\Omega\subseteq\mathbb{R}^d$ form a vector space $\mathbb{R}^\Omega$, with pointwise operations $(\alpha f+\beta g)(t)=\alpha f(t)+\beta g(t)$.
Square-integrable functions $L^2(\mu)=\{f\in\mathbb{R}^\Omega:\int_\Omega f(t)^2\,\mu(dt)<\infty\}$ form a subspace of $\mathbb{R}^\Omega$, as $\ell^2$ does of $\mathbb{R}^{\mathbb{N}}$. The measure $\mu$ weights regions of $\Omega$ (Lebesgue $dt$, or a data distribution); counting measure on $\mathbb{N}$ gives $\ell^2$.
Polynomials $\mathbb{R}[t]\supseteq\mathbb{R}[t]_{\le n}$, a subspace of $\mathbb{R}^{\mathbb{R}}$.
Continuous functions $C(\Omega)$, a subspace of $\mathbb{R}^\Omega$.
1.1 · Vector spaces, subspaces and bases
Span, linear independence and bases
Let $V$ be a vector space and $S\subseteq V$ a set of vectors, finite or infinite. A linear combination uses finitely many of them, $c_1\mathbf{s}_1+\dots+c_m\mathbf{s}_m$.
The span of $S$: all linear combinations, the smallest subspace containing $S$.
$S$ is linearly independent if only the trivial combination gives $0$; equivalently, every vector of the span has unique coefficients.
A basis of $V$ is a linearly independent set with span $V$. All bases have the same number of elements, the dimension$\dim V$. In a fixed order, $(\mathbf{b}_1,\dots,\mathbf{b}_n)$ gives each vector its coordinates: $\mathbf{v}=\sum_i c_i\mathbf{b}_i$.
Examples. $\mathbf{e}_1,\dots,\mathbf{e}_n$ is a basis of $\mathbb{R}^n$. The hyperplane ${\displaystyle \left\{\mathbf{x} : \sum\nolimits_i x_i=0\right\}}$ has the basis $\mathbf{e}_i-\mathbf{e}_n$, $i<n$, and dimension $n-1$. $\mathbb{R}[t]$ has the infinite basis $1,t,t^2,\dots$; like $\ell^2$, $L^2(\mu)$ and $C([0,1])$, it has no finite basis and is infinite-dimensional.
1.2 · Linear maps
Structure-preserving maps
Whenever we introduce a structure, we ask which maps preserve it. For vector spaces these are the linear maps; in this lecture also affine maps for affine spaces and isometries for inner product spaces; continuous maps for topological spaces follow in Lecture 3.
A map $f : V\to W$ between vector spaces is linear if
Equivalently, $f$ preserves linear combinations: $f(\lambda\mathbf{x}+\mu\mathbf{y})=\lambda f(\mathbf{x})+\mu f(\mathbf{y})$. With $\lambda=\mu=0$: $f(0)=0$.
A bijective linear map is an isomorphism; its inverse is automatically linear. Isomorphic spaces carry the same structure and differ only in the names of their elements.
1.2 · Linear maps
Examples of linear maps
1.1Inclusion of a subspace $U\subseteq V$: $U\hookrightarrow V$, $\mathbf{u}\mapsto\mathbf{u}$.
1.2Matrix multiplication $\mathbf{x}\mapsto A\mathbf{x}$, $A\in\mathbb{R}^{m\times n}$: defined because $\mathbf{x}$ is a column, an $n\times1$ matrix, and $A\mathbf{x}$ is again a column, an element of $\mathbb{R}^m$.→ Example 1.2
1.3Evaluation at finitely many points$\mathrm{ev}_{t_1,\dots,t_k}:\mathbb{R}^\Omega\to\mathbb{R}^k$.→ Example 1.3
1.4Reading out the entries of a matrix, $\mathbb{R}^{m\times n}\to\mathbb{R}^{mn}$: a linear isomorphism, the precise sense in which $\mathbb{R}^{m\times n}$ is “$\mathbb{R}^{mn}$ in disguise”. → Example 1.4
1.5Smoothing an image$S:\mathbb{R}^{784}\to\mathbb{R}^{784}$.→ Example 1.5
1.6Averaging over blocks of $4\times4$ pixels, $P:\mathbb{R}^{784}\to\mathbb{R}^{49}$: a ${28\times28}$ image becomes a ${7\times7}$ image. → Example 1.6
Example 1.1 is complete here; Examples 1.2 to 1.6 follow, one slide each.
Evaluating a function at $t_1,\dots,t_k$ is a linear map$\mathbb{R}^\Omega\to\mathbb{R}^k$: each coordinate satisfies $(\alpha f+\beta g)(t_i)=\alpha f(t_i)+\beta g(t_i)$. For ${k=1}$, this is the evaluation at a single point. Drag the points $t_i$ along the axis. The same works on $C(\Omega)$ and $\mathbb{R}[t]$.
Linear, since $+$ and scalar multiplication act entry by entry; bijective, since putting the entries back is the inverse. So it is an isomorphism, and $\dim\mathbb{R}^{m\times n}=mn$.
This is the map of Lecture 1 that turned a greyscale image into a column: a $28\times28$ image is a matrix of grey values, and reading it row by row gives its vector in $\mathbb{R}^{784}$. Column by column gives another isomorphism.
Every pixel becomes a weighted average of itself and its four neighbours, with fixed weights (values outside the image count as $0$). So $S$ is linear: a $784\times784$ matrix.
row $i$, column $j$
$\mathbf{x}$
Diffusion models (DALL·E 2 and 3, Stable Diffusion) degrade images step by step with Gaussian noise, some variants by blurring as $S$ does, and train a neural network to undo each step; run backwards from noise, it creates new images.
1.2 · Linear maps
Example 1.6: coarse-graining an image
The coarse-graining map $P$ averages over blocks of $4\times4$ pixels:
Each of the $49$ blocks of $4\times4$ pixels becomes one pixel, the average of its $16$ values.
Linear: fixed weights $\tfrac1{16}$; not injective: $784$ numbers become $49$.
Applications. The same basic idea, fewer numbers at lower resolution, is behind streaming video when the connection is slow, pixel binning in smartphone cameras, and colour storage in JPEG images.
1.2 · Linear maps
The space of linear maps
Sums and multiples of linear maps are again linear:
The zero element is the zero map; the axioms hold value by value, as for $\mathbb{R}^X$. The map $f\mapsto A$ is an isomorphism: the sum of two linear maps corresponds to the sum of their matrices. Special case $\mathrm{Lin}(V,\mathbb{R})$: Lecture 6.
Section 2 of 3
2
Affine space structure
2.1 Affine spaces and affine subspaces2.2 Affine maps
2.1 · Affine spaces and affine subspaces
Affine spaces
Points have no origin; we translate them by vectors. An affine space $(A,V,+)$ consists of a set $A$ of points, a vector space $V$ and a translation $+ : A\times V\to A$,$(p,v)\mapsto p+v$, with:
Identity: $p+0=p$ for all $p\in A$.
Associativity: $(p+v)+w=p+(v+w)$ for all $p\in A$ and all $v,w\in V$.
Unique connecting vector: for any $p,q\in A$, there is exactly one $v\in V$ with $p+v=q$, the connecting vector$q-p$.
physical space: no origin, only $q-p$ matters
Every vector space $V$ is an affine space over itself: $(V,V,+)$, with the vector addition as translation, $(\mathbf{x},\mathbf{v})\mapsto\mathbf{x}+\mathbf{v}$.
The translation is part of the structure: another translation on the same $A$ and $V$ gives another affine space; when it is clear, we say “$A$ is an affine space over $V$”. We do not add two points but translate a point by a vector: “$+$” has two meanings. Everyday affine spaces: physical space, positions on a map, instants of time (durations as differences).
2.1 · Affine spaces and affine subspaces
Affine subspaces
$$B=p_0+U=\{p_0+u : u\in U\}, \qquad U \text{ a linear subspace, the direction}$$
From the base point $p_0$, $s$ reaches all points $p=p_0+s\,d$ of $B$; $d$ is a unit vector in $U$. Drag $p_0$.
$B$ is an affine space over $U$, with the restriction of the translation of $A$:
$$B\times U\to B,\qquad (p,v)\mapsto p+v.$$
Any $p\in B$ is a base point, $B=p+U$; the direction $U$ is determined by $B$.
For $A=V$: $p_0+U$ is a linear subspace $\iff p_0\in U$.
2.1 · Affine spaces and affine subspaces
Example 2.1: a hyperplane of probability distributions
An affine subspace: $H=p_0+U$ with $p_0=(\tfrac1n,\dots,\tfrac1n)^\top$ and ${\displaystyle U=\left\{\mathbf{x}\in\mathbb{R}^n : \sum\nolimits_i x_i=0\right\}}$.
drag $\mathbf{x}$ in $H$; drag elsewhere to turn the axes
For $n=3$: $H$ is the plane through $\mathbf{e}_1,\mathbf{e}_2,\mathbf{e}_3$, and $U$ the parallel plane through $0$. Since $0\notin H$, $H$ is not a linear subspace.
The probability distributions, $x_i\ge 0$ and $\sum_i x_i=1$, form the triangle with vertices $\mathbf{e}_1,\mathbf{e}_2,\mathbf{e}_3$ in $H$, the simplex.
The output is $>\tfrac12$ (blue) exactly where $r>0$, and $<\tfrac12$ (red) where $r<0$.
The decision boundary $B$ (black) is an affine subspace, $B=p_0+U$ with $U=\{\mathbf{x}:w_1x_1+w_2x_2=0\}$.
The weights fix the direction $U$; the bias $b$ shifts $B$ parallel to itself.
2.2 · Affine maps
Affine maps
An affine map is a linear map plus a translation. Let $A$, $B$ be affine spaces over $V$, $W$.
Definition 2.3. A map $F:A\to B$ is affine if there are a point $p\in A$ and a linear map $L:V\to W$ with
$$F(p+v)=F(p)+L(v)\quad\text{for all } v\in V.$$
Lemma 2.4. For affine $F$, $L$ is unique, the linear part of $F$, and works at every point:
$$F(q+v)=F(q)+L(v)\quad\text{for all } q\in A,\ v\in V.$$
Proof. Uniqueness: $L(v)$ translates $F(p)$ to $F(p+v)$, so it is the connecting vector $F(p+v)-F(p)$ (property 3). Every point is $q=p+u$, and by associativity $F(q+v)=F(p+(u+v))=F(p)+L(u)+L(v)=F(q)+L(v)$.□
linear part: drag $A\mathbf{e}_1$, $A\mathbf{e}_2$
shift by $\mathbf{b}$: drag $\mathbf{b}$
The whole grid moves by the same vector $\mathbf{b}=F(0)$: lines stay lines, parallels stay parallel, but $0$ goes to $\mathbf{b}$. Every affine map $\mathbb{R}^n\to\mathbb{R}^m$ has this form, with linear part $L=A$; it is linear only for $\mathbf{b}=0$.
Section 3 of 3
3
Inner product space structure
3.1 Inner products and examples3.2 Norms and angles3.3 Isometries
3.1 · Inner products and examples
Inner products
The dot product of Lecture 1 has three properties, for all $\mathbf{x},\mathbf{y}\in\mathbb{R}^n$:
Bilinearity: it is linear in each argument separately, that is,
Positive definiteness: $\langle\mathbf{x},\mathbf{x}\rangle\ge0$, with equality if and only if $\mathbf{x}=0$.
A map ${\langle\cdot,\cdot\rangle}_V : V\times V\to\mathbb{R}$ with these three properties is an inner product, and $(V,{\langle\cdot,\cdot\rangle}_V)$ is an inner product space.
These properties, rather than the specific formula $\sum_i x_iy_i$, are what let us speak of norms (lengths) and angles. Whether the dot product extends to the spaces above: next slide.
$\Omega$ compact, $\mu$ finite; fails for $f=1$ on $\mathbb{R}$, $f=1/t$ on $(0,1]$
The question is whether the formula of the dot product converges. On $\mathbb{R}^{\mathbb{N}}$ and $\mathbb{R}^\Omega$ it does not always; this does not mean that these spaces admit no inner product at all.
3.2 · Norms and angles
Cauchy–Schwarz inequality
In every inner product space $V$, the norm (length) of $\mathbf{x}$ is, as in Lecture 1, ${\|\mathbf{x}\|}_V:=\sqrt{{\langle\mathbf{x},\mathbf{x}\rangle}_V}$.
Theorem 3.1 (Cauchy–Schwarz).
For every symmetric bilinear form with ${\langle\mathbf{x},\mathbf{x}\rangle}_V\ge0$ (positive semi-definite): ${\langle\mathbf{x},\mathbf{y}\rangle}_V^2\le{\langle\mathbf{x},\mathbf{x}\rangle}_V\,{\langle\mathbf{y},\mathbf{y}\rangle}_V$.
For an inner product: $\bigl|{\langle\mathbf{x},\mathbf{y}\rangle}_V\bigr|\le{\|\mathbf{x}\|}_V\,{\|\mathbf{y}\|}_V$, with equality if and only if $\mathbf{x},\mathbf{y}$ are linearly dependent.
Proof of 1.$0\le{\langle\mathbf{x}-t\mathbf{y},\mathbf{x}-t\mathbf{y}\rangle}_V={\langle\mathbf{x},\mathbf{x}\rangle}_V-2t\,{\langle\mathbf{x},\mathbf{y}\rangle}_V+t^2{\langle\mathbf{y},\mathbf{y}\rangle}_V$ for all $t$. If ${\langle\mathbf{y},\mathbf{y}\rangle}_V>0$, choose $t={\langle\mathbf{x},\mathbf{y}\rangle}_V/{\langle\mathbf{y},\mathbf{y}\rangle}_V$; if ${\langle\mathbf{y},\mathbf{y}\rangle}_V=0$, the right side is affine in $t$ and $\ge0$, so ${\langle\mathbf{x},\mathbf{y}\rangle}_V=0$.$\square$
Part 1 does not need positive definiteness; this matters for kernels (Lectures 5 and 9), which give positive semi-definite forms.
3.2 · Norms and angles
Cosine similarity and angle
By Theorem 3.1 (part 2), the quotient below lies in $[-1,1]$; it compares directions, not lengths.
Definition 3.2. Let $V$ be an inner product space and $\mathbf{x},\mathbf{y}\in V$,$\mathbf{x},\mathbf{y}\neq0$. The angle$\alpha_V\in[0,\pi]$ between $\mathbf{x}$ and $\mathbf{y}$ is given by
and $\cos(\alpha_V)$ is called the cosine similarity of $\mathbf{x}$ and $\mathbf{y}$. The vectors are orthogonal, $\mathbf{x}\perp\mathbf{y}$, if ${\langle\mathbf{x},\mathbf{y}\rangle}_V=0$.
On $\mathbb{R}^n$ with the dot product, this is the angle of Lecture 1.
By the equality case of Theorem 3.1, $\cos(\alpha_V)=\pm1$ exactly when $\mathbf{x}$ and $\mathbf{y}$ are linearly dependent.
In machine learning, cosine similarity compares embeddings of words, texts or images, for instance in search and recommendation.
3.2 · Norms and angles
Example 3.3: cosine similarities and angles of functions
The $L^2(\mu)$ inner product for the Lebesgue measure on $[-1,1]$. Shaded: ${\int_{-1}^1 f(t)\,g(t)\,dt}$,positive minus negative parts. Orthogonal, for instance: ${1\perp t^2-\tfrac13}$ and $1\perp t$.
3.3 · Isometries
Isometries
The maps that preserve the structure of an inner product space are the linear maps that preserve the inner product.
Definition 3.4. Let $V$ and $W$ be inner product spaces. A linear map $T:V\to W$ is called an isometry if
$${\langle T\mathbf{x},T\mathbf{y}\rangle}_W={\langle\mathbf{x},\mathbf{y}\rangle}_V \qquad\text{for all } \mathbf{x},\mathbf{y}\in V.$$
A bijective isometry is called an isometric isomorphism.
An isometry preserves norms, ${\|T\mathbf{x}\|}_W={\|\mathbf{x}\|}_V$, hence angles and orthogonality.
Conversely, a linear map that preserves norms is an isometry, since ${\langle\mathbf{x},\mathbf{y}\rangle}_V=\tfrac14\bigl({\|\mathbf{x}+\mathbf{y}\|}_V^2-{\|\mathbf{x}-\mathbf{y}\|}_V^2\bigr)$.
An isometry is injective: $T\mathbf{x}=0$ gives ${\|\mathbf{x}\|}_V={\|T\mathbf{x}\|}_W=0$, so $\mathbf{x}=0$. It need not be surjective, for instance $\mathbb{R}\to\mathbb{R}^2$, $x\mapsto(x,0)^\top$.
Isometrically isomorphic spaces differ only in the names of their elements.
3.3 · Isometries
Example 3.5: isometries
drag $\mathbf{x}$, $\mathbf{y}$
Orthogonal matrices on $\mathbb{R}^n$: $\mathbf{x}\mapsto T\mathbf{x}$ with $T^\top T=I$. For $n=2$: the rotations and reflections of the picture.
Reading out the entries of a matrix (Example 1.4): an isometric isomorphism from $\mathbb{R}^{m\times n}$ with the Frobenius inner product onto $\mathbb{R}^{mn}$.
Inclusion of a subspace$U\hookrightarrow V$, with the restricted inner product.
3.3 · Isometries
Isometries of $\mathbb{R}^n$
For $\mathbb{R}^n$ with the dot product, the first example already gives all isometries.
Theorem 3.6. A linear map $\mathbb{R}^n\to\mathbb{R}^n$,$\mathbf{x}\mapsto T\mathbf{x}$, is an isometry for the dot product if and only if $T^\top T=I$, that is, if the columns of $T$ form an orthonormal basis. In this case, $\det(T)=\pm1$. For $n=2$, the isometries are exactly the rotations and the reflections in lines through $0$:
Proof.$\langle T\mathbf{x},T\mathbf{y}\rangle=\mathbf{x}^\top T^\top T\,\mathbf{y}$, which equals $\mathbf{x}^\top\mathbf{y}$ for all $\mathbf{x},\mathbf{y}$ if $T^\top T=I$; conversely, $\mathbf{x}=\mathbf{e}_i$, $\mathbf{y}=\mathbf{e}_j$ give ${(T^\top T)}_{ij}=\langle T\mathbf{e}_i,T\mathbf{e}_j\rangle=\delta_{ij}$. Moreover $\det(T)^2=\det(T^\top T)=1$. For $n=2$, the first column is $(\cos(\varphi),\sin(\varphi))^\top$ and the second a unit vector orthogonal to it, $\pm(-\sin(\varphi),\cos(\varphi))^\top$.$\square$
Lecture 2
Summary
Vector spaces: adding and scaling obey the rules of $\mathbb{R}^n$ (matrices, sequences, functions). Span, linear independence and bases use finite linear combinations; bases may be infinite.
Linear maps preserve linear combinations; from $\mathbb{R}^n$ to $\mathbb{R}^m$ they are matrices, for instance smoothing and coarse-graining images; $\mathrm{Lin}(V,W)$ is a vector space.
An affine space $(A,V,+)$: points translated by vectors, no origin; every vector space is one over itself. Affine subspaces are translated linear subspaces.
An affine map is a linear map plus a translation: $F(p+v)=F(p)+L(v)$ with a unique linear part $L$, at every point; on $\mathbb{R}^n$:$F(\mathbf{x})=A\mathbf{x}+\mathbf{b}$.
Inner products give norms, cosine similarities and angles beyond $\mathbb{R}^n$, well defined by Cauchy–Schwarz. Isometries preserve them and are injective; on $\mathbb{R}^n$: orthogonal matrices.
Next: norms, metric spaces and topology (Lecture 3).