Mathematics for Machine Learning · Prof. Dr. Nihat Ay
Lecture 1 $\mathbb{R}^n$ as the natural model space for data
Much of the mathematics behind Machine Learning is geometry.
Introduction
Geometry behind Machine Learning
Much of the mathematical formalism underlying Machine Learning is based on geometry. We begin with a review of the geometric structures of $\mathbb{R}^n$, the most prominent model space.
The set $\mathbb{R}^n$
Examples from Machine Learning: images, audio signals and text
What can we do with vectors in $\mathbb{R}^n$?
3.1 General operations 3.2 Application to images, audio signals and text
Dot product, norm, angle, distance
4.1 General definitions 4.2 Application to images, audio signals and text
The elements of $\mathbb{R}^n$ are called vectors.
In the text, a column is written as a row followed by the transpose symbol: ${\mathbf{x}=(x_1,\dots,x_n)^\top}$; $x_i$ is the $i$-th coordinate of $\mathbf{x}$.
Rows without ${}^\top$ are reserved for a different kind of object, the covectors (Lecture 6).
The distinction pays off later: in probability theory, functions are multiplied from the right and probability measures from the left.
1 · The set ℝⁿ
Point or arrow
Each $\mathbf{x}\in\mathbb{R}^n$ can be pictured as the point with coordinates $x_1,\dots,x_n$, or as the arrow from the origin $0$ to that point, the position vector.
$n = 2$ · drag the point
$n = 3$ · drag the point, or its foot in the floor plane
Pictures only exist for $n = 2, 3$. Data in Machine Learning are high-dimensional.
Data are values on the elements of a finite set $X$: intensities on pixels, counts on words, amplitudes on sampling instants. Such data are a function ${f:X\to\mathbb{R}}$; all of them form ${\mathbb{R}^X}$.
An enumeration of $X$ encodes each $f$ as a column in $\mathbb{R}^n$, one-to-one.
The column depends on the enumeration: the same $3\times3$ image gives different columns.
Section 2 of 4
2
Examples from Machine Learning: images, audio signals and text
2 · Examples from Machine Learning: images, audio signals and text
An image as a vector in $\mathbb{R}^n$
move the mouse over a pixel · click to freeze
A greyscale image is a grid of pixels with intensities $x_i \in [0,1]$: $0$ white, $1$ full ink, grey in between, e.g. at the edges of a stroke.
Reading the pixels row by row gives a vector in $\mathbb{R}^n$, with $n$ the number of pixels; an MNIST digit has ${n = 28 \times 28 = 784}$.
pixel —
the same image as one long column $\mathbf{x} = (x_1,\dots,x_{784})^\top$, written as a strip
2 · Examples from Machine Learning: images, audio signals and text
Signal and samples
zoom
A sound is a signal $s(t)$, given for every instant $t$ of a continuous time interval.
A recording stores $n$ samples $x_i = s(t_i)$, a fixed number per second (the sampling rate): a vector $\mathbf{x}\in\mathbb{R}^n$. One second at $16$ kHz gives $n = 16\,000$.
However large $n$ is, information about $s(t)$ may be lost: at $16$ kHz, $\sin(2\pi\cdot 16\,000\,t)$ has only zero samples, yet is not silence. The signal itself is infinite-dimensional, with one coordinate $s(t)$ for each instant $t$.
2 · Examples from Machine Learning: images, audio signals and text
Words have no built-in geometry
Text has no built-in structure, so we embed words into $\mathbb{R}^n$. The naive way is one-hot encoding: one coordinate per word of the vocabulary.
All words equally (dis)similar: Cat is as far from Tiger as from Car.
This clashes with what we know: Cat and Tiger denote animals, Car does not. Below, we meet a feature-based alternative, and with angles and distances we will see that it captures this semantic proximity.
Section 3 of 4
3
What can we do with vectors in $\mathbb{R}^n$?
3.1 General operations3.2 Application to images, audio signals and text
$\lambda\mathbf{x}$ keeps the direction for $\lambda>0$, reverses it for $\lambda<0$, and collapses to $0$ for $\lambda=0$. All multiples lie on one line through the origin.
The sum is the parallelogram rule: attach a copy of $\mathbf{y}$ to the tip of $\mathbf{x}$. Drag $\mathbf{x}$ and $\mathbf{y}$.
3.1 · General operations
Linear and convex combinations
Combining both operations gives linear combinations $\lambda\mathbf{x}+\mu\mathbf{y}$, $\lambda,\mu\in\mathbb{R}$.
If $\lambda,\mu\ge0$ and $\lambda+\mu=1$, that is, ${(1-t)\,\mathbf{x}+t\,\mathbf{y}}$ with $t\in[0,1]$: a convex combination, a point of the segment from $\mathbf{x}$ to $\mathbf{y}$.
3.1 · General operations
Building vectors from unit vectors
With addition and scalar multiplication, every vector of $\mathbb{R}^n$ is built from the standard unit vectors, where $\mathbf{e}_i$ has entry $1$ in position $i$ and $0$ everywhere else:
scale by $1-t$ and $t$, then add pixel by pixel; $t=\tfrac12$ averages every pixel
Remark 3.1. Why not general linear combinations? Images form the cube ${[0,1]}^n\subset\mathbb{R}^n$. It is not a subspace, since sums and multiples with $\lambda>1$ may leave it, but it is convex: convex combinations of images are images.
3.2 · Application to images
Every image is a sum of pixels
$$\mathbf{x} = \sum_{i=1}^{784} x_i\,\mathbf{e}_i \qquad \text{(an image as a superposition of its pixels)}$$
$\mathbf{e}_i$
partial sum $\sum_{j\le i} x_j\,\mathbf{e}_j$
$\mathbf{e}_i$ is the image consisting of a single fully inked pixel $i$, and every image is the superposition of its pixels, weighted by their intensities $x_i$.
Step through the pixels: each step adds one term $x_i\,\mathbf{e}_i$.
the picture shows the first 5 ms; each button plays 1.6 s of the same tones
Thin curves: the signals, a low hum ($200$ Hz) and a high tone ($1600$ Hz). Dots: their samples at $16$ kHz, $80$ in $5$ ms, the sample vectors $\mathbf{x}$ and $\mathbf{y}$. Adding plays both sounds at the same time.
The signals themselves are added point by point and rescaled the same way: they behave like vectors with infinitely many coordinates, infinite-dimensional vectors (from Lecture 7 on).
Multiplying a sample vector by a scalar $\lambda$ changes the volume.
$\lambda = -1$ flips the sign of every sample: $-\mathbf{x}$ sounds the same as $\mathbf{x}$, but $\mathbf{x} + (-\mathbf{x}) = \mathbf{0}$ is silence. This is the principle of noise cancelling.
Remark 3.2. Recordings are stored in a fixed range, usually $[-1,1]$, set by the recording equipment: for $|\lambda|>1$, samples beyond $\pm1$ are cut off (clipping), heard as distortion.
3.2 · Application to text
A text as a vector
“”
Language models also work with sequences of vectors, but with learned word vectors in place of the one-hot vectors, plus vectors encoding the positions. Inside the model, each vector comes to depend on the whole text, not on its word alone. Their output is a probability distribution over the next token, computed with the softmax map.
columns: the blocks of $\mathbf{x}$ · last column: their sum
One block per position: the order is kept.
Adding the blocks counts the words, a bag of words: the order is lost. Divided by $L$: relative frequencies, a probability distribution on the words.
3.2 · Application to text
A feature-based encoding
One coordinate per semantic feature instead of one per word.
is an animal
is domesticated
has an engine
Sums combine features: $\mathbf{e}_1+\mathbf{e}_2$ is a cat. But $\mathbf{e}_1+\mathbf{e}_3=(1,0,1)^\top$, an animal with an engine, exists nowhere.
Distinct words now share coordinates, and the $\mathbf{e}_i$ stand for features: $\mathbf{x}_{\text{Cat}} = \mathbf{e}_1+\mathbf{e}_2$, an animal that is domesticated.
Pick Cat with probability $1-t$ and Tiger with probability $t$. The coordinates of the mixture are the probabilities of the features:
For $t=\tfrac12$: certainly an animal, domesticated with probability $\tfrac12$, no engine.
Section 4 of 4
4
Dot product, norm, angle, distance
4.1 General definitions4.2 Application to images, audio signals and text
4.1 · General definitions
Length, angle and distance
A third operation assigns a real number to two vectors, the dot product; with it come the norm, the angle $\alpha$, the cosine similarity (for $\mathbf{x},\mathbf{y}\neq\mathbf{0}$) and the distance:
Norm and angle concern vectors (arrows at $0$), the distance concerns points; vector versus affine space: Lecture 2.
4.1 · General definitions
Example 4.1: two vectors in the plane
Drag $\mathbf{x}$ and $\mathbf{y}$.
Orthogonal means $\langle\mathbf{x},\mathbf{y}\rangle = 0$, that is, $\alpha = 90^\circ$: for instance, $\mathbf{y} = (-x_2, x_1)^\top$, the vector $\mathbf{x}$ rotated by $90^\circ$. Try it.
4.2 · Application to images
Comparing two images
$\mathbf{x}$
$\lambda\mathbf{y}$
The same formulas, summed over all $784$ pixels. Far from orthogonal (the $7$ and the $5$ share ink in $152$ pixels), but also far from parallel.
Dim the 5: the cosine stays, the distance changes. Cosine similarity compares shapes, ignoring brightness: standard for images, documents and word embeddings.
4.2 · Application to audio signals
Comparing two sounds
$80$ samples in $5$ ms at $16$ kHz. Different frequencies: orthogonal sample vectors, the products $x_i y_i$ cancel.
$\|\mathbf{x}\|^2$ is the energy of the clip; for orthogonal clips, energies add (Pythagoras).
Same frequency: $\mathbf{y}=\lambda\mathbf{x}$, $\cos = \pm1$, but the distance depends on the volume; $-\mathbf{x}$ sounds like $\mathbf{x}$.
4.2 · Application to text
Features make similarity meaningful
drag to rotate
With features, Cat and Tiger share the feature “animal”, enclose an angle of only $45^\circ$ and are the closest pair; Car is orthogonal to both. With one-hot encoding, all three words are pairwise orthogonal, at the same distance $\sqrt2$.
Real word embeddings (Word2Vec, language models) are built in the same spirit, with latent features learned from huge amounts of text.
Lecture 1
Summary
Images, sound recordings and (embedded) words are vectors in $\mathbb{R}^n$, with $n$ large; vectors are columns.
A sound is a function of time; its sample vector is a finite-dimensional description, and finitely many samples lose information.
Scaling, adding and convex combinations act componentwise: brightness, averages, superposition, volume, word counts.
Every vector is a combination $\sum_i x_i\,\mathbf{e}_i$: pixels, one-hot words, features.
The dot product gives lengths, angles and distances in any dimension; cosine similarity ignores brightness and volume, distance does not.
Feature-based embeddings make semantic proximity of words visible as geometry.
Next: vector spaces and affine spaces (Lecture 2).